OpenShortsHow a long video becomes a vertical clip
faster-whisper transcribes the video with word-level timestamps, PySceneDetect finds the scene boundaries, and Google Gemini 3.0 Flash scores the transcript to pick the 3 to 15 strongest moments of 15 to 60 seconds each. Each moment is then cut with FFmpeg and reframed to 9:16 with MediaPipe face tracking.
The two stages that decide whether a clip is usable are moment scoring and reframing. Everything else is mechanical.
OpenShorts self-hosted is free and open source under MIT, so every stage below can be read and changed. OpenShorts Cloud runs the same pipeline on a GPU from $12/month.
What is OpenShorts?
OpenShorts is an open source AI clip generator that turns long videos (podcasts, webinars, livestreams, interviews) into vertical 9:16 clips for TikTok, Instagram Reels and YouTube Shorts.
The pipeline, stage by stage
Transcription with word-level timestamps
faster-whisper transcribes the source audio and returns a timestamp for every individual word, not just every sentence. Word-level timing is what makes both the clip boundaries and the burned-in subtitles land on the right frame.
Scene boundary detection
PySceneDetect scans the video for cuts and hard visual transitions. These boundaries stop a clip from starting mid-shot, which is the most common reason an auto-generated clip reads as machine-made.
Viral moment scoring
Google Gemini 3.0 Flash receives the timestamped transcript and the scene boundaries together, and returns the 3 to 15 strongest segments of 15 to 60 seconds. Each one is scored on hook strength, emotional payload and how well it stands alone without the surrounding context.
Vertical reframing
Each clip is cropped from 16:9 to 9:16 in one of two modes. TRACK mode follows a single subject with MediaPipe face detection and a YOLOv8 fallback, damped by a stabiliser that holds the camera still inside a safe zone instead of chasing every head movement. GENERAL mode handles group shots and landscapes by keeping the full width over a blurred backdrop.
Subtitles, hooks and effects
Subtitles are generated from the word-level transcript and burned in with FFmpeg. An AI-written hook overlay covers the first seconds. Optional Gemini-generated FFmpeg filter chains handle colour and transitions.
Dubbing and publishing
ElevenLabs dubbing translates the audio into 30+ languages while preserving the speaker's voice, and the dubbed track is re-transcribed so the subtitles match the new language. Finished clips post directly to TikTok, Instagram Reels and YouTube Shorts.
Why does moment scoring need the transcript and the scenes together?
A transcript alone finds a good sentence but has no idea whether the shot cuts halfway through it. Scene boundaries alone find clean cuts with nothing worth saying between them. Passing both to the model at once is what lets it pick a segment that is both quotable and visually intact, which is the difference between a clip a person would watch and one that merely starts and stops in the right places.
Why does the camera hold still instead of following the face exactly?
Because a crop that tracks a face frame by frame produces visible swinging, and the swinging reads as amateur even when the framing is technically correct. The reframing keeps a safe zone around the subject and only moves the crop when they leave it, then damps the movement on the way. A speaker tracker sits on top to stop the crop from flipping between people every time someone nods, and to hold position through brief occlusions.
How long does it take?
On a typical CPU, an 8-minute source video takes roughly 5 to 8 minutes end to end. On an NVIDIA GPU the same video takes about 50 seconds. The gap is almost entirely transcription and encoding; the model call is a small fraction of it.
What does it cost to run?
OpenShorts comes in two editions and they are priced very differently, so it is worth being precise. OpenShorts (self-hosted) is free and open source under the MIT licence: Free and open source under MIT. You run it with Docker on your own machine and bring your own API keys. No watermark, no usage cap, no subscription. What it costs you is hardware and time: on a typical CPU an 8-minute video takes 5 to 8 minutes to process. OpenShorts Cloud is the hosted service: The same software with the compute and API keys covered. An NVIDIA GPU clips that same 8-minute video in about 50 seconds. Free plan is 20 minutes a month with a watermark and no credit card. Paid plans start at $12/month for 100 minutes with no watermark, up to $59/month.
Common questions
- What AI model does OpenShorts use to find viral moments?
- Google Gemini 3.0 Flash. It receives the word-level transcript with timestamps together with PySceneDetect scene boundaries, and returns 3 to 15 segments of 15 to 60 seconds scored on hook strength, emotional payload and whether the segment stands alone without surrounding context.
- How does the automatic vertical cropping work?
- Two modes. TRACK mode follows a single subject using MediaPipe face detection with a YOLOv8 fallback, stabilised so the crop holds still inside a safe zone instead of following every movement. GENERAL mode handles group shots and landscapes by preserving the full width over a blurred backdrop.
- Can it dub clips into other languages?
- Yes, into more than 30 languages via ElevenLabs, preserving the original speaker's voice characteristics. The dubbed audio is then re-transcribed so the burned-in subtitles match the new language rather than the original.