A prompt is a probability, a gate is a guarantee.

We run Songbrain. You upload a song and get an analysis plus seven short videos cut to the strongest parts of it. The chain is audio, then structured meaning, then generated visuals conditioned on that meaning.

Most of what we learned came from the parts that broke. Here is the one that reorganised how we build everything else.

We have a rule that banned brand logos from generated images. It appears twice in the prompt, once when the world is chosen and once in the per-shot rules. We measured how often it holds: two runs out of three.

That number is the whole lesson. An instruction to a generative model is a probability distribution over outputs, not a constraint on them. Anything that must not ship needs a check afterwards.

1. Where prompts stop working

Three instructions that did exactly what we wrote, and not what we meant.

This started with a complaint about our own output: too many storyboards looked generic, full of people seen from behind and random lightning.

Story detection was not at fault. 47 of 51 sets that week were genuinely story-driven rather than falling back to a generic genre palette. The instructions were at fault.

The lightning was in the spec

We had defined six burst frames as "climax energy, release, spectacle." The model delivered exactly that: six abstract light explosions, ten of the fifteen images in a set. We rewrote them as six camera angles on a single action, and explicitly banned shockwaves, speed lines, lens flares and fireworks as burst motifs.

The hero shot never changed

One role returned an identical image on every run: lone silhouette, rooftop, arms spread, seen from behind, skyline. Adding more description did nothing. It only moved once we named that specific cliché in the prompt and forbade it.

A rule about shots cannot repair a world

Our ban on signage sat in the shot rules. But the world gets chosen one step earlier, and the model picked "dominated by glowing holographic advertisements." Every image then came back carrying pseudo-text. The rule was true and correctly written. It was simply attached to the wrong stage.

2. The check that passed everything

Our worst bug looked exactly like success.

Our image QC ran with thinkingBudget: 0. We sent the same image three times at temperature 0 and got pass, fail, pass. The "visible text" criterion had existed for months and was never actually being applied.

Raising the budget produced a more interesting failure. The model started writing prose into the reason field: 7579 output tokens, finishReason: MAX_TOKENS, JSON truncated mid-string.

Our fail-open contract turned that into a silently unchecked image set. Nothing errored. Every log line looked normal. We raised the ceiling to 2000, then 4000, then 8000 tokens and hit the same wall each time, because the ceiling was never the cause. The field was.

The fix was to make reason an enum instead of free text: none | face | anatomy | text | trademark | artifacts. That took it to 465 tokens, a clean stop, and 15 of 15 images actually judged. The same set that had previously passed 15/15 now reported text in one burst frame and a face in another.

One API detail worth knowing: the empty string is not a legal enum value in the Gemini API. That is why "nothing wrong" has to be a named token, mapped back to empty server-side.

3. Rules belong in code

When the prompt grows a paragraph per bug, the prompt is the wrong place.

Every new class of song forced a new prompt rule. Signage, trademarks, the hero cliché, characters looking at camera, metaphor staging. Each one worked. None of them scaled, because a hundred song classes would mean a hundred paragraphs competing for the model’s attention.

So the rules moved out of the prompt and into a lint pass over the returned shot list, which runs before we spend a cent on images. Walking or standing in more than three shots. More than one character looking at camera. Figures inside strobe frames. Signage in the world or in the shots. A violation triggers exactly one correction retry, with the violations quoted verbatim.

Across 24 historical jobs in 11 genres: 13 came back clean on the first pass, 9 were fixed by the retry, and 2 kept a violation, both signage, which the image QC catches downstream. Without the lint, 11 of those 24 sets would have shipped a known failure class.

A new failure class is now a new lint line rather than a new paragraph of prompt.

4. Reweigh evidence, never invent it

How we let a language model into the ranking without letting it make things up.

Finding the best moment in a song is split deliberately. librosa does recall: peak-picking produces nine diversity-selected windows. The model does precision: one comparative call sends all nine as inline audio and ranks them against a fixed rubric.

Reading the lyrics needed a second pass, because per-line scoring is local by construction. Each transcript segment is scored in isolation, blind to what the song is actually about. So one call reads the entire timestamped transcript and returns the narrative peak.

Here is the constraint that keeps it honest. The model’s pick carries no audio evidence of its own, so it is never allowed to become a candidate. It anchors to the nearest real audio-scored candidate within 20 seconds and boosts that candidate, with the boost decaying by distance and hard-capped. A confident pick with no candidate within 20 seconds gets discarded.

A guitar solo won as the hook of a rap song

The cause was in our own rubric. The line for instrumental clips judge the lead-melody hook instead treated an instrumental lead as an equal substitute for a vocal hook, with no penalty attached. A window with 2.5 seconds of actual singing beat windows with 11 to 14 seconds.

Two layers fixed it. The prompt now receives the measured vocal seconds for each clip, computed from word timestamps that already run earlier in the pipeline. And a deterministic Python backstop applies a graded penalty of up to 15 points below 4 seconds of vocal coverage. That gate only engages if the pool holds at least one candidate with 3 seconds or more, so purely instrumental songs stay unpenalised.

Measuring coverage needed its own defensive code. Whisper occasionally emits a single word with an 18-second duration, and words on segment boundaries get counted twice. We cap each word at 2.5 seconds and dedupe on (word, start). Without that, the fix would have rested on a number that was itself wrong.

Best moment 1 and 2 were the same chorus

28 of 79 production jobs had at least one duplicate pair in the top three. One song returned the same chorus three times. The cause is structural: a chorus repeats two or three times, each repetition genuinely sounds equally strong, and our selection had no diversity check at all.

The pick is now greedy-diverse. A candidate whose lyric fingerprint overlaps 0.6 or more with something already chosen gets skipped, and duplicates only fill in when fewer distinct sections exist than we need.

The same song scored 63, 63, 69

Gemini is not deterministic at temperature 0. We run three calls, take the median, and cache by content hash, so an identical file returns an identical judgment with zero calls.

5. Two bugs with no model involved

Both were ours. Neither was where we first looked.

We generated our own upstream errors

Image generation fired all nine prompts through a single Promise.all. The provider returned "upstream unavailable" often enough that our fallback, a model eleven times more expensive, was running as normal operation rather than as a failure path. It cost more per day than the entire analysis pipeline.

We assumed the provider was flaky. Identical prompts, same account, minutes apart:

We capped the pool at five and added a real retry of the whole prediction rather than only the POST. Cost per image set went from about $0.19 to about $0.03. Wall clock doubled, which was the right trade.

A visible layer left its last paint behind

In long Remotion renders, unmounting a currently-visible layer left its final frame stuck in Chromium’s compositor. Everything afterwards was painted on top of it, so the video showed two lyric lines superimposed. The rule now: hide with opacity, never with a ternary or a key change.

What made it expensive is that every cheap check was clean. renderStill: clean. The studio: clean. A short frame range: clean. The ghost needs render history in the same tab, and it first appeared around 85 seconds of song time. It reproduces identically on two different GL backends, so it was never a GPU issue.

There is now a regression check that renders the full song, pulls frames shortly after each line change, and diffs them against renderStill as ground truth. Encoder noise measures 58 or below. A ghost measures 212 to 242. The threshold sits at 120.

That check had its own trap. The ffmpeg filter writes its stats to stderr, so reading only stdout returns NaN. And NaN > threshold is false, so the check passes forever.

What we would keep

The gate is the part that generalises. Everything else on this page is a specific bug, but each one came from the same assumption: that writing the rule down was the same as enforcing it.

Prompts set the odds. Code sets the floor. The interesting engineering in a generative product is almost entirely in the second one.