How do you use Claude Vision to extract text and descriptions from video frames?

Extract one frame per cut from the video, send each frame to Claude Vision with a structured prompt that separates two jobs — transcribe all on-screen text verbatim, then describe the scene (subject, action, setting, framing) — and request JSON output so the results parse reliably. Cut-level sampling gives you one clean frame per scene instead of hundreds of redundant ones.

Start with frame sampling, because it determines both accuracy and token cost. Sample at scene boundaries — one frame per cut — rather than at fixed intervals: every cut is a new composition with potentially new text, while frames inside a shot are near-duplicates. In one documented reference-ad analysis, cut detection surfaced 9 cuts and extracted one representative frame per scene, which was enough to reconstruct the ad's full structure. Visual analysis can also segment on content changes rather than hard cuts — one documented workflow split a video into 2 clips automatically because the model detected an outfit change mid-video.

Next, write the prompt as a schema, not a request. Ask Claude Vision for two distinct fields per frame: on_screen_text (transcribed verbatim, preserving line breaks and original language — this is exactly the job it has been used for in production, extracting on-image text from carousel slides for content analysis) and description (shot type, subject, action, setting, and the function of any text — hook, caption, CTA, UI label). Requesting JSON with named fields keeps outputs consistent across dozens of frames; free-form answers drift in structure and are painful to parse downstream.

For text-dense frames, improve legibility before you prompt. Claude Vision reads structured text — captions, UI screens, title cards — well; small or cluttered text benefits from cropping the text region into its own image and sending it separately. App-screen text extracts cleanly enough that documented localization runs translated and regenerated 5 UI screens per market in the original sequence order, working from the extracted text.

Then decide what the descriptions need to capture, based on what consumes them. If the goal is recreating or adapting the video, extract structural attributes per frame — cut position, camera style, on-screen text, energy, ending pattern — the same headings production workflows use when decoding a reference ad. If the goal is search or content analysis, keep descriptions entity-dense: named objects, readable text, and setting beat adjectives.

One warning if the frames feed a generation pipeline afterward: extraction is faithful, which cuts both ways. Frames with burned-in captions will carry that text forward — in one documented production, attaching a captioned reference during generation caused the model to reproduce the old captions in the new output, so the captioned source was deliberately excluded and only the clean extracted breakdown was used. Strip or crop burned-in captions from any frame you plan to reuse as a visual reference.

If your end goal is video production rather than analysis alone, note that the invideo agent runs this pipeline natively: upload a video and it detects every cut, extracts one frame per scene, reads and translates on-screen and UI text, and saves the extracted structure in project context for generation — no separate vision step to wire up.

Watch some of these to see what works for you:

Go through the reference ad, find where every cut happens, grab a frame from each scene.

— invideo's creative team