Why We Chose Claude for Document Extraction (and What We Tested First)
Why We Chose Claude for Document Extraction (and What We Tested First)
Updated June 2026.
When we started building ExtractBee, the model selection wasn't obvious. Several capable models can read documents. We ran a structured evaluation before committing to our production stack. This post covers what we tested, how we evaluated, and why Claude Sonnet 4.6 became our primary extraction model in early 2026.
Update (May 2026): Since this evaluation, we moved Gemini 2.5 Flash to our primary extraction provider for its substantial cost advantage at scale (roughly 6x cheaper than Claude on our workload), with Claude Sonnet 4.6 as the automatic fallback. The analysis below reflects our reasoning as of April 2026 and explains why Claude remains a core part of our stack. See the note at the end for what changed and why.
What We Were Optimising For
Document extraction has specific requirements that differ from general LLM tasks:
Structured output consistency. The model must return valid JSON every time — not sometimes valid JSON, not JSON with explanatory text around it. Inconsistent output format breaks the parsing pipeline.
Field-level accuracy. We care about individual field accuracy, not holistic document understanding. Getting the invoice number right while getting the due date wrong is a failure, even if the model "understood" the document.
Handling ambiguity correctly. Real invoices have ambiguous data: dates formatted as DD/MM or MM/DD, amounts with comma or period decimal separators, vendor names in two languages on the same document. The model should resolve these correctly or flag uncertainty — not silently pick the wrong interpretation.
Multilingual competence. Our customers process documents in English, German, French, Lithuanian, Latvian, Polish, and other languages. The model needs genuine multilingual capability, not English-first with degraded performance on other languages.
Instruction following under constraint. We give precise instructions: return minified JSON, use ISO 8601 dates, use ISO 4217 currency codes, return null for missing fields rather than empty strings. The model needs to follow these consistently.
Cost at scale. We process potentially millions of pages per month. Cost per page at production volume matters.
Models We Evaluated
We evaluated five models on a test set of 200 real-world documents (50 invoices, 50 receipts, 40 bank statements, 30 purchase orders, 30 mixed/other) across multiple languages.
- Claude Sonnet 4.6 (Anthropic)
- GPT-4o (OpenAI)
- Gemini 2.5 Pro (Google)
- Google Document AI (invoice-specific processor)
- AWS Textract AnalyzeExpense (invoice-specific processor)
A note on the numbers below: these figures reflect our internal test set as of early 2026, on the model versions then current. Vendor pricing and model performance change frequently — and our own production stack has since evolved (see the May 2026 update above) — so treat every exact number here as a point-in-time snapshot rather than a current benchmark. This is also a good place to flag the distinction between raw OCR and AI-based document extraction, which shaped what we were measuring.
Evaluation Results
Structured Output Consistency
This was our first filter. We ran each model 10 times on the same document with the same prompt and checked: how often does the response parse as valid JSON?
| Model | Valid JSON rate |
|---|---|
| Claude Sonnet 4.6 | 99.8% |
| GPT-4o | 98.1% |
| Gemini 2.5 Pro | 94.3% |
| Google Document AI | 100% (native structured output) |
| AWS Textract | 100% (native structured output) |
Claude and GPT-4o both performed well here. Gemini's lower rate (5.7% invalid JSON) was a practical concern — downstream parsing failures require error handling and retries that add latency and cost.
Field-Level Extraction Accuracy
We manually labelled all 200 documents with ground truth values and measured field-level accuracy: for each field, what percentage of extractions returned the correct value?
| Model | Invoice fields | Receipt fields | Bank statement fields | Multi-language docs |
|---|---|---|---|---|
| Claude Sonnet 4.6 | 94.2% | 96.1% | 91.8% | 93.7% |
| GPT-4o | 92.8% | 94.9% | 89.3% | 91.2% |
| Gemini 2.5 Pro | 91.4% | 93.2% | 87.6% | 90.1% |
| Google Document AI | 95.8% | 97.1% | 88.2% | 82.3% |
| AWS Textract | 94.7% | 96.3% | 90.1% | 79.4% |
Key observations:
Google Document AI and AWS Textract are excellent on standard English-language invoices and receipts — their purpose-built processors are highly optimised for these formats. But their multilingual performance dropped significantly, particularly on Lithuanian, Latvian, and Polish documents. For our EU-focused customer base, this was a significant limitation.
Claude's multilingual performance was the most consistent across all languages tested. The gap between English and non-English documents was smaller for Claude than for any other model.
Bank statement extraction is harder than invoice extraction — more rows, more ambiguous data, less standardised formats across banks. Claude and AWS Textract both performed better here than Google Document AI.
Ambiguity Handling
We specifically tested documents with known ambiguities: dates that could be read as DD/MM or MM/DD, amounts in formats common in Germany (1.234,56) vs the UK (1,234.56), vendor names in two languages.
Claude was the most likely to flag these as uncertain (returning lower confidence signals or explicit notes) rather than silently choosing an interpretation. GPT-4o tended to commit to an interpretation without flagging uncertainty. This matters for our Human Review UI — we want uncertain fields surfaced, not silently resolved. (It's also why we put so much weight on field-level confidence scores rather than a single document-level accuracy number.)
Instruction Following
We tested how consistently each model followed our specific output instructions:
- Return minified JSON only (no whitespace, no explanation)
- Use ISO 8601 for dates (YYYY-MM-DD)
- Use ISO 4217 for currency (EUR not €)
- Return null for missing fields (not "", not "N/A", not omitting the key)
| Model | Instruction compliance rate |
|---|---|
| Claude Sonnet 4.6 | 97.3% |
| GPT-4o | 94.1% |
| Gemini 2.5 Pro | 88.7% |
Claude followed our formatting instructions most consistently. GPT-4o occasionally added explanatory text before the JSON. Gemini more frequently deviated from the null convention and returned empty strings or omitted fields entirely.
Cost at Production Volume
At 100,000 pages/month:
| Model | Est. cost/page | Monthly cost |
|---|---|---|
| Claude Sonnet 4.6 | ~$0.010 | ~$1,000 |
| GPT-4o | ~$0.012 | ~$1,200 |
| Gemini 2.5 Pro | ~$0.006 | ~$600 |
| Google Document AI | ~$0.010 | ~$1,000 |
| AWS Textract | ~$0.008 | ~$800 |
Gemini is cheaper — but at the time of this evaluation we tested Gemini 2.5 Pro, whose lower JSON consistency rate (94.3%) meant roughly 5,700 failed parses per 100,000 pages. Each failure requires a retry or fallback, which eroded much of the cost advantage. When we first modelled the real cost including retries, Gemini's effective cost advantage over Claude largely disappeared — which is why we shipped on Claude.
That calculus changed later. Once we moved to the much cheaper Gemini 2.5 Flash and hardened our parsing and fallback pipeline, the JSON-consistency penalty stopped mattering and the cost advantage at our scale became decisive. That's why Flash is now our primary provider, with Claude as the automatic fallback (see the May 2026 update above, and the note at the end).
Why Claude Won
The decision came down to four factors:
1. Multilingual consistency. Our customers are EU-based. Lithuanian, Latvian, German, French, and Polish invoices are not edge cases — they're a significant share of our document volume. Claude's multilingual performance was the most consistent of all models we tested.
2. Instruction following. A parsing pipeline depends on consistent, well-formed output. Claude's 97.3% instruction compliance rate (vs 94.1% for GPT-4o and 88.7% for Gemini) meant significantly fewer downstream parsing failures.
3. Ambiguity handling. Claude was more likely to express uncertainty on genuinely ambiguous fields rather than committing to a potentially wrong answer. This pairs well with our Human Review UI — uncertain fields are surfaced for human review rather than silently propagated.
4. Anthropic's approach to reliability. Anthropic publishes detailed documentation on API reliability, prompt caching, and planned model deprecation timelines. For a production system, knowing when models will be deprecated and having a clear migration path matters.
Where GPT-4o is competitive: GPT-4o is a strong alternative — close on accuracy, slightly worse on instruction following, slightly higher cost. If we were starting fresh, GPT-4o would be a credible choice. The gap is narrower than it was 12 months ago.
Where Google Document AI wins: For high-volume English-language invoice and receipt processing, Google Document AI's purpose-built processor outperforms general-purpose LLMs on standard formats. If we were building a narrow English-only invoice processor, it would be a strong contender.
What We'd Still Like to Test
Smaller models for classification. We currently use a lightweight classifier for document type detection. We haven't rigorously benchmarked smaller models (Haiku, GPT-4o mini) purely for this classification step. There may be cost savings available here. (Gemini Flash, which appeared in our original "to test" list, has since graduated all the way to our primary extraction model — so the open question is now narrowly about the classifier, not the extractor.)
Fine-tuned models for specific document types. For document types with very consistent layouts (specific industry standards, regional invoice formats), a fine-tuned smaller model might outperform a prompted larger model at lower cost. We haven't validated this at scale.
Newer model releases. Model capabilities evolve quickly. Our evaluation reflects the landscape as of early 2026. We re-evaluate periodically — particularly when Anthropic, OpenAI, or Google release significant model updates.
Frequently Asked Questions
Does ExtractBee use only Claude? No. As of May 2026, Gemini 2.5 Flash handles primary extraction and Claude Sonnet 4.6 is our automatic fallback. We use Tesseract OCR as a pre-processing step for scanned documents, and a lightweight classifier for document type detection. Claude remains a core part of the stack — it's what runs whenever the primary provider can't complete a request.
What happens if a provider is unavailable? Our pipeline is provider-abstracted. Gemini 2.5 Flash handles primary extraction; on a recoverable error (5xx, quota, blocked content) we automatically fall back to Claude Sonnet 4.6, with retry and exponential backoff on transient errors. This dual-provider setup gives us resilience that neither model alone provides, and it means a single vendor outage doesn't stop extractions.
Will you switch models if a better option emerges? Yes. Our pipeline is abstracted from the specific model — swapping the extraction model doesn't require changes to the document processing infrastructure. We evaluate new models as they're released.
Does using Claude mean Anthropic can see my documents? Anthropic processes documents as a sub-processor under our Data Processing Agreement. Per Anthropic's policy, inputs to the Claude API are not used to train models. ExtractBee's DPA is available on request and lists all sub-processors including Anthropic.
What Changed in May 2026
The evaluation above stands on its own merits — in April 2026, Claude Sonnet 4.6 was the right primary model for the trade-offs we faced. But two things shifted afterward.
First, the model we'd shortlisted matured. Our original benchmark ran on Gemini 2.5 Pro. Production, however, moved to Gemini 2.5 Flash — dramatically cheaper, and at our scale that mattered. The JSON-consistency concern that hurt Gemini in our early tests was the thing we engineered around: a hardened parsing layer and an automatic fallback to Claude meant occasional malformed output no longer cost us reliability.
Second, our pipeline was already provider-abstracted. Because swapping the extraction model never required touching the document-processing infrastructure (exactly as we noted in the FAQ), promoting Gemini Flash to primary was a configuration change, not a rewrite.
The result: as of 2026-05-18, Gemini 2.5 Flash is our primary extraction provider, with Claude Sonnet 4.6 as the automatic fallback — roughly a 6x cost reduction on our workload, without giving up the resilience and multilingual quality that made Claude our first choice. Claude didn't lose; it became the safety net that lets us run a cheaper primary with confidence.
If you want the broader context, see how we think about the architecture behind invoice extraction and why we built ExtractBee in the first place.
Try ExtractBee free — 20 pages/month →
Last updated: June 2026.
ExtractBee is operated by MB Dokigo, Vilnius, Lithuania.