Sudip Niroula

← Back to notes

Token-Efficient Extraction of Text from Screenshots: Where the Research Stands

July 2026 · reading notes

Every time a vision-capable LLM reads a screenshot, it converts pixels into hundreds or thousands of tokens before it generates a single word, and those image tokens are billed the same as text tokens. That's easy to ignore for a one-off query, but it adds up fast for anything that sends screenshots repeatedly: GUI agents that click, observe, and repeat, or pipelines that extract code and text from images at scale. So I went looking for how cheap you can make this without losing what actually matters in the image. Here's what I found, and the gap I think is worth filling.

How providers actually charge for images

This isn't one universal number. OpenAI tiles images into 32×32-pixel patches, with a detail: low mode for a cheaper fixed pass. Anthropic scales roughly with (width × height) / 750, so resizing an image before sending it produces a near-linear savings. Gemini uses discrete media_resolution levels, and for Gemini 3-class models, the cost per level is fixed regardless of the image's actual pixel dimensions, which is a genuinely different mechanism from the other two. A same-sized screenshot has been measured at roughly 2,451 tokens on GPT-5.5 versus 6,636 on Claude versus 6,192 on Gemini: a 2.7× spread before generation even starts. Any cost-optimization strategy has to be evaluated against a specific provider's mechanism, because what saves tokens on one may do nothing on another.

Three research threads that don't talk to each other

General visual token compression. A fast-moving line of work (FastV, LLaVA-PruMerge, AIM, SparseVLM) prunes or merges visual tokens inside the model itself, with claimed savings up to ~35× FLOPs. The catch: it's validated almost entirely on VQA and captioning, tasks with soft, semantic correctness. A slightly lossy visual representation rarely changes the final answer there.

GUI-agent-specific efficiency. Because GUI agents send a fresh screenshot every step, cost compounds: one benchmark measured 120,000–320,000 tokens for a single multi-step task. This spawned a cluster of 2025–2026 papers: AQuaUI (adaptive quadtrees: coarse tokens for empty regions, fine tokens for text/icons, ~30% fewer tokens at ~99% retained performance), FocusUI, PixelPrune, and STaR-KV (which exploits similarity between consecutive screenshots). All of these are optimized for action accuracy (did the agent click the right thing), a metric with real tolerance for lossy compression.

OCR-free and hybrid document understanding. Donut and Pix2Struct read text directly from pixels without a separate OCR stage. Older, code-specific work (psc2code, CodeSCAN) extracts source code from programming screencasts via classical OCR plus a correction stage, built on the explicit observation that raw OCR on code is unreliable without that correction step. None of this predates the LLM-token-cost framing, so it was never benchmarked against it.

The gap

Do the token-reduction techniques validated for VQA and GUI-action tasks preserve enough fidelity for exact code or text extraction, or does extraction need a fundamentally different approach?

Nobody has directly tested this. GUI-agent compression papers measure whether a click landed correctly, which tolerates a merged or dropped token. Extracting exact code doesn't: a single lost token can be a dropped character or a wrong indentation level, and there's no existing benchmark measuring that failure mode specifically.

One small, informal data point from my own side experiment: running plain Tesseract OCR (no LLM, no preprocessing) against 24 synthetic code screenshots gave a 0% exact-match rate and a mean character error rate near 90%. Not a rigorous result on its own, but it's directionally consistent with the psc2code-era finding that naive approaches to this problem fail in specific, measurable ways, and that the space between "naive OCR" and "full-detail VLM call on every image" is exactly where the open cost/accuracy trade-off lives.

What I'd want to test next

A benchmark of code and UI-text screenshots with character-exact ground truth, comparing a full-resolution VLM call, provider low-cost modes, an open pruning method like FastV, and a crop-then-OCR-then-LLM-correction hybrid, scored as a Pareto frontier of token cost versus extraction accuracy rather than a single leaderboard number. If you've seen work that already does this, I'd like to know about it.

Sources