Self-Host Whisper: faster-whisper with Whishper, Docker and Caddy on a VPS

Whisper can run on your own VPS. According to the project, faster-whisper is up to 4 times faster than OpenAI's Whisper, with the same accuracy and less memory (faster-whisper README, as of Nov 8, 2026). You pay for the server plan, not the amount of audio.

Afterwards you have a web UI for uploads, export as TXT, JSON, VTT or SRT, and a CLI for batch processing. Recordings and transcripts stay on your server and don't go to a transcription service.

What is Whisper / faster-whisper?

Whisper is OpenAI's model for multilingual speech recognition, speech translation and spoken language identification; code and model weights are under the MIT license (OpenAI Whisper README, as of Nov 8, 2026).

faster-whisper is a reimplementation of OpenAI's Whisper using CTranslate2, an inference engine for Transformer models (faster-whisper README, as of Nov 8, 2026).

Project benchmark: small model, 13 minutes of audio, Intel Core i7-12700K with 8 threads, beam size 5 (faster-whisper README, as of Nov 8, 2026):

These are the project's figures, not a RackDiff measurement; on a VPS they depend on CPU and threads.

Why self-host Whisper?

• Cost: You pay for the server plan, not the audio volume; current prices incl. VAT in the VPS comparison.
• Privacy: Recordings and transcripts sit on your server instead of with a transcription service. If they relate to people, they are personal data (Art. 4(1) GDPR). Unless you process them purely privately, you need a data processing agreement with the host (Art. 28(3) GDPR), and securing the server is up to you (Art. 32(1) GDPR) (GDPR, as of Nov 8, 2026).
• Load: How much audio you process is limited by your server's CPU and RAM.
• Models: Whishper downloads the models on first start (Whishper docs, as of Nov 8, 2026), faster-whisper fetches them from the Hugging Face Hub (faster-whisper README, as of Nov 8, 2026); after that, transcription runs on your server.

Prerequisites

Before we start, you need:

• A VPS with Docker. Whishper gives no minimum RAM or CPU; on small machines transcription takes longer, and without AVX support in the CPU extra steps are needed (Whishper docs, as of Nov 8, 2026). MongoDB and LibreTranslate run alongside transcription (Whishper README, as of Nov 8, 2026). Plans in the VPS comparison.
• An Ubuntu version supported by Docker (22.04, 24.04, 26.04 LTS) (Docker docs, as of Nov 8, 2026)
• SSH access to the server
• A domain pointing to the server, for HTTPS in step 3

Whisper models

OpenAI measured speed on an A100 with English speech; turbo is an optimized version of large-v3 (OpenAI Whisper README, as of Nov 8, 2026). OpenAI gives no RAM figures for CPU; the faster-whisper benchmark lists 1477 MB for small with int8 (faster-whisper README, as of Nov 8, 2026). RackDiff currently does not list GPU servers.

For German, use the multilingual models, not the .en variants; larger models need more memory and time, and large-v3 shows improved results in many languages (Whishper docs, as of Nov 8, 2026).

Step 1: Prepare the server

Connect via SSH and update the system:

ssh root@your-server-ip
apt update && apt upgrade -y

Install Docker Engine with the Compose plugin from Docker's apt repository, as described in the Docker docs; per Docker, the convenience script isn't meant for production systems (Docker docs, as of Nov 8, 2026). The last command is apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin.

Step 2: Set up Whishper as a web UI

This guide uses Whishper as the web UI: upload or link, transcription with faster-whisper, translation with LibreTranslate, subtitle editor, export as TXT, JSON, VTT or SRT. The author is working on a rewrite; per the README, the current version gets no new releases or updates (Whishper README, as of Nov 8, 2026).

2.1 Download files

Download the project files (Whishper docs, as of Nov 8, 2026):

mkdir -p /opt/whishper && cd /opt/whishper
curl -o docker-compose.yml https://raw.githubusercontent.com/pluja/whishper/main/docker-compose.yml
curl -o .env https://raw.githubusercontent.com/pluja/whishper/main/example.env
curl -o nginx.conf https://raw.githubusercontent.com/pluja/whishper/main/nginx.conf
mkdir -p ./whishper_data/libretranslate/{data,cache}
chown -R 1032:1032 whishper_data/libretranslate

2.2 Configure

Set in .env before the first start (Whishper docs, as of Nov 8, 2026):

WHISPER_MODELS=tiny,small
WHISHPER_HOST=whisper.your-domain.com
DB_USER=whishper
DB_PASS=<your-password>
LT_LOAD_ONLY=de,en

WHISPER_MODELS sets the models, WHISHPER_HOST is the browser address without protocol, LT_LOAD_ONLY the languages for LibreTranslate.

Before the first start, replace the port line - 8082:80 of the whishper service in docker-compose.yml so only the reverse proxy reaches the port. Published Docker ports bypass ufw (Docker docs, as of Nov 8, 2026); with 127.0.0.1 the port is reachable only from the server itself (Docker docs, as of Nov 8, 2026):

    ports:
      - "127.0.0.1:8082:80"

2.3 Start

docker compose up -d
docker compose logs -f

The first start downloads transcription and translation models and can take a while; docker compose logs -f shows the progress (Whishper docs, as of Nov 8, 2026). The version: "3.9" in the upstream file only triggers a warning (Docker docs, as of Nov 8, 2026).

Step 3: Reverse proxy with HTTPS and password protection

For HTTPS access, put Caddy in front as a reverse proxy. The Whishper docs describe no user login (Whishper docs, as of Nov 8, 2026); so Caddy protects the site with a username and password.

Install Caddy as root from the project's package repository (Caddy docs, as of Nov 8, 2026):

apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list
chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-stable.list
apt update
apt install caddy

Generate the password hash with caddy hash-password. Since Caddy v2.8.0 the directive is called basic_auth, before that basicauth (Caddy docs, as of Nov 8, 2026). reverse_proxy forwards to the locally bound port (Caddy docs, as of Nov 8, 2026).

/etc/caddy/Caddyfile:

whisper.your-domain.com {
    basic_auth {
        your-name <hash-from-caddy-hash-password>
    }
    reverse_proxy 127.0.0.1:8082
}

systemctl reload caddy

Caddy obtains the certificate automatically from Let's Encrypt or ZeroSSL if the A/AAAA record points to the server and ports 80 and 443 are reachable from outside (Caddy docs, as of Nov 8, 2026).

Check access

Open https://whisper.your-domain.com and log in with the Caddy password. Then:

1. Upload a file or enter a link
2. Choose language and model
3. Start transcription
4. Export as TXT, JSON, VTT or SRT (Whishper README, as of Nov 8, 2026)

Note: Duration depends on model, CPU and threads; the project's figures are in the section "What is Whisper / faster-whisper?".

Alternative: CLI for automation

For batch processing without a web UI, use a command-line client.

whisper-ctranslate2 in a Python environment

whisper-ctranslate2 is a command-line client compatible with OpenAI's client, based on CTranslate2, and can distinguish speakers with pyannote.audio; that requires a Hugging Face access token (whisper-ctranslate2 README, as of Nov 8, 2026).

On Ubuntu 24.04 the venv module comes with the python3-venv package (Ubuntu package archive, as of Nov 8, 2026); python3 -m venv creates a separate Python environment (Python docs, as of Nov 8, 2026). faster-whisper needs Python 3.9 or later and no installed FFmpeg (faster-whisper README, as of Nov 8, 2026).

apt install -y python3-venv
python3 -m venv /opt/whisper-cli
/opt/whisper-cli/bin/pip install whisper-ctranslate2
/opt/whisper-cli/bin/whisper-ctranslate2 podcast.mp3 \
  --model small --language de --compute_type int8 \
  --output_format srt --output_dir .
# output in the current folder

Batch script

For several files:

#!/bin/bash
# transcribe-all.sh
set -euo pipefail

INPUT_DIR="/data/audio"
OUTPUT_DIR="/data/transcripts"
mkdir -p "$OUTPUT_DIR"

for file in "$INPUT_DIR"/*.mp3; do
  [ -e "$file" ] || continue
  /opt/whisper-cli/bin/whisper-ctranslate2 "$file" \
    --model small --language de --compute_type int8 \
    --output_format srt --output_dir "$OUTPUT_DIR"
  echo "Done: $(basename "$file" .mp3)"
done

Performance tuning

Setting compute_type

faster-whisper lists int8 for CPU and float16 or int8_float16 for GPU. In the project benchmark (small, CPU), int8 takes 1 min 42 s and 1477 MB, fp32 2 min 37 s and 2257 MB (faster-whisper README, as of Nov 8, 2026). In the CLI you set it with --compute_type int8 (whisper-ctranslate2 source, as of Nov 8, 2026); Whishper documents no setting for it (Whishper docs, as of Nov 8, 2026).

CPU threads

Whishper uses 4 CPU threads by default (CPU_THREADS, from version 3.1.2) (Whishper docs, as of Nov 8, 2026); in the upstream docker-compose.yml the value sits under environment of the whishper service. The CLI sets threads with --threads (whisper-ctranslate2 source, as of Nov 8, 2026). deploy.resources.limits only caps what a container may use (Docker docs, as of Nov 8, 2026).

VAD (Voice Activity Detection)

faster-whisper ships Silero VAD; by default vad_filter only removes silence longer than 2 seconds (faster-whisper README, as of Nov 8, 2026). In the CLI: --vad_filter True (whisper-ctranslate2 source, as of Nov 8, 2026). Whishper documents no setting for it (Whishper docs, as of Nov 8, 2026).

Automation with n8n

With n8n you integrate transcription into workflows. The following flow is an untested sketch:

Workflow: audio via email → transcript back

1. Email Trigger (IMAP): receive an email with an audio attachment, with the "Download Attachments" option (n8n docs, as of Nov 8, 2026)
2. HTTP Request: send the audio to Whishper with body content type "Form-Data" and parameter type "n8n Binary File" (n8n docs, as of Nov 8, 2026)
3. Send Email: return the result via "Attachments"; check yourself whether Whishper's response already contains the transcript or you fetch it later, because the API does not document this (n8n docs, as of Nov 8, 2026)

In n8n:

[Email Trigger (IMAP)] → [HTTP Request to Whishper] → [Send Email with transcript]

Whishper has a REST interface POST /api/transcriptions with the form fields file, language and modelSize. Per the backend README it connects the ASR service, database and UI; response format and authentication are not documented (Whishper backend README, as of Nov 8, 2026). From n8n you call https://whisper.your-domain.com/api/transcriptions with Basic Auth credentials (HTTP Request, generic credentials) (n8n docs, as of Nov 8, 2026):

POST https://whisper.your-domain.com/api/transcriptions
Content-Type: multipart/form-data

file: [audio-file]
language: de
modelSize: small

More on n8n self-hosting: n8n Self-Hosting Guide →

Combining Whisper with other tools

Whisper + Ollama = Voice-to-LLM

Combine transcription with AI analysis:

1. Transcribe audio with Whisper
2. Send transcript to Ollama
3. Generate summary, analysis or answer

Use case: record meeting → transcribe → extract action items

Whisper + Paperless = audio archive

Transcribe voice memos and archive them in Paperless-ngx:

1. Record voice message
2. Transcribe with Whisper
3. Pass transcript to Paperless-ngx as text file; Paperless keeps text documents unchanged (Paperless-ngx FAQ, as of Nov 8, 2026)

Own server or transcription service

Transcription services bill on their own terms. With your own server you pay the host's plan; current prices incl. VAT in the VPS comparison, and the Whisper calculator estimates your needs.

Common problems

Transcription is very slow

• Check compute_type: use int8 instead of float32
• Use a smaller model: small instead of medium
• Enable VAD (CLI --vad_filter True): filters out parts without speech (whisper-ctranslate2 README, as of Nov 8, 2026)

Out of memory error

The model is too large for your RAM:

Killed (OOM)

Solution: choose a smaller model or a server with more RAM, see VPS comparison.

Poor quality with German speech

• Set language explicitly: --language de instead of auto-detect
• Larger model: small instead of tiny
• Use a prompt: for specialist vocabulary, set a CLI prompt for the first window: --initial_prompt "…" (whisper-ctranslate2 source, as of Nov 8, 2026)

Docker container won't start

# check logs
docker compose logs -f

# restart
docker compose down && docker compose up -d

Conclusion

You now have your own transcription server with faster-whisper. Recordings and transcripts stay on your server; CPU and RAM limit the load.

For podcasts: transcript and subtitles per episode.

For videos: subtitles as SRT or VTT, edited in the Whishper editor.

For conversations involving personal data: recordings stay on your server; what the GDPR requires for this is under "Why self-host Whisper?".

Next step: combine Whisper with Ollama for AI analysis or n8n for automation.

More Self-Hosting Guides

Ollama auf VPS installieren

Lokale KI mit Open WebUI

Lesen

n8n Self-Hosting Guide

Docker Setup Schritt für Schritt

Lesen

Paperless-ngx Guide

Papierloses Büro selbst hosten

Lesen