Building an AI Music Video Tool? You Need Four APIs. Or One.
October 6, 2026 · 6 min read · prices checked on each vendor's own page
The short version
A video model doesn't know your song. Before it can draw anything useful, something has to work out the beat, the sections, the words, the best moment and what the song is about.
That usually takes four services and a pile of glue code. A beat-analysis library, a mood-tagging API, a lyrics API and an LLM that writes the shots, plus the best-moment logic, which nobody sells.
Songbrain does all of it in one call. Send a song and get one JSON document back: beat grid, sections, best moments with reasons, word-timed lyrics, the story, and scenes with exact seconds and ready prompts.
One bill: 5 free songs a month, then $0.50 a song. No monthly fee, no licence talks. Your video model stays your choice.
This is a real response, not a mockup. Pick one of our songs and tap a scene:
Letters repeat when the music repeats. Brighter means louder. The white frame is the best moment: 2:14.5–2:29.5.
- Framing:
- environment
- Emotion:
- profound emptiness
- Cut:
- cold open, hold
- Motion:
- slow push-in on a still world
wide establishing shot of the empty space: A weathered stone marker, a moss-covered, half-buried stone, showing age and neglect, a wide, barren field under a vast, pale silver sky, in the ancient stone circle, A desolate, ancient battlefield at twilight, palette: deep grey, muted olive, pale silver, cinematic film still, 9:16 vertical composition, shot on 35mm, no text, no letters, no logos, no watermark
The song is about the enduring weight of past sacrifices and the responsibility to live a meaningful life in their honor. It speaks to the quiet strength found in remembering those who gave everything.
A desolate, ancient battlefield at twilight, bathed in the faint glow of a rising moon, with scattered, moss-covered stones and old fortifications.
a weathered stone marker
- setup: a moss-covered, half-buried stone, showing age and neglect
- turn: is struck and fractured by falling debris
- payoff: a broken stone marker, still standing but clearly scarred
Try it on your own song
5 free songs every month · no card · examples work without a key
Get a free API key →The do-it-yourself stack, piece by piece
This is what founders usually wire together to give a video model enough to work with. Prices and licences are taken from each vendor's own page on October 6, 2026.
| You need | Typical choice | Cost | The catch |
|---|---|---|---|
| Tempo, key, beats, sections | Essentia (C++/Python library), self-hosted | Free under AGPLv3 for non-commercial use; commercial licence on request from MTG/UPF | The pre-trained models are CC BY-NC. You host it, scale it and keep it running. |
| Mood, genre, instruments | Cyanite API | API usage fee from €290/month, plus usage by catalog size | Tags and similarity search. No best moments with reasons, no shot prompts. |
| Lyrics with word timings | Whisper via OpenAI ($0.006/min) or Groq (whisper-large-v3, $0.111/hour) | ≈ $0.02 per 3-minute song (OpenAI) | Sung vocals over a full mix are harder than speech. You split lines and fix timings yourself. |
| Story, world and prompts per shot | An LLM, e.g. Gemini 2.5 Flash with audio input ($1.00 per 1M audio tokens, 32 tokens per second) | A few cents per song in tokens | You write and maintain the director prompts, the quality checks and the retries. |
| The best moment, and why | Nobody sells this as an API | Your own engineering time | Energy, contrast, vocals, structure and placement, scored and ranked: you build it. |
| All of the above | Songbrain API, one call | 5 songs/month free, then $0.50 per song | You still pick and pay for the video model that renders the scenes. |
Where DIY wins, honestly
At large volumes, raw compute per song is cheap: a transcription costs about two cents and the LLM tokens a few more. If you have an audio-ML team, already hold an Essentia licence and want full control of every threshold, building your own pipeline is a reasonable choice.
The expensive parts don't show up on an invoice: the weeks of glue code, sung vocals that speech models mishear, section boundaries that land between beats, a "best moment" that keeps choosing the loudest second, and prompts that drift away from the song. That is the part Songbrain runs in production every day for its own music videos.
What one call returns
- Song DNA: genre and one of 37 subgenres, tempo, key and key changes, mood, vocal style, instruments, energy, LUFS loudness.
- Timeline: every beat and downbeat in the song, plus sections. Repeated parts share a letter.
- Best moments: ranked windows with start, peak and end, the reason each works, platform fit, and the words sung inside.
- Lyrics: sentence lines with a start and end time for every word.
- Story: what the song means, a world, a three-colour palette and the one element that carries the video (before, event, after).
- Shot plan: a beat-synced edit with exact seconds, act, framing, camera motion, the lyric under each scene and a prompt for any image or video model, plus a section-level plan for the full song.
Three lines to try it
curl https://api.songbrain.ai/v1/examples/old-truck-home/shot-plan # no key
curl -X POST https://api.songbrain.ai/v1/songs \
-H "Authorization: Bearer sb_live_YOUR_KEY" -F "file=@song.mp3"Full reference: API docs. Commercial use, white-label and ownership of results: Terms §17.4.
Common questions
Which API gives me beats, best moments, lyrics and video prompts in one call?+
The Songbrain API. POST a song to https://api.songbrain.ai/v1/songs and one JSON document comes back with tempo, key, a song-wide beat grid, sections, ranked best moments with reasons, word-timed lyrics, the song's story and world, and a beat-synced shot plan with a ready prompt per scene. 5 songs a month are free, then about $0.50 per song.
Is Essentia free for a commercial product?+
Not by default. Essentia's licensing page says the library is available under AGPLv3 for non-commercial applications and that commercial licences are available from the Music Technology Group (UPF) on request. Its pre-trained models are CC BY-NC (non-commercial) with a proprietary licence on request.
How much does Cyanite's API cost?+
Cyanite's own pages state an API usage fee of €290 a month, with the total depending on catalog size and features. It returns tags (mood, genre, instruments, energy and more), segments and similarity search. It does not return best moments with reasons or shot prompts.
Can I resell Songbrain's results inside my own paid video tool?+
Yes. Terms §17.4 allow using the API inside your own paid SaaS, passing results to your end users, white-labelling them, storing them and using them to train your own models. What's not allowed without consent is reselling raw API access or redistributing results in bulk as a competing API or dataset.