How We Built Zero-Template Invoice Extraction: Architecture Deep Dive

How We Built Zero-Template Invoice Extraction: Architecture Deep Dive

Updated 2026-06-16: Since this post was first published, Gemini 2.5 Flash became our primary extraction provider (~6x lower cost per page than Claude), with Claude Sonnet 4.6 now serving as the automatic fallback. The pipeline below reflects the current dual-provider setup.

When we started building ExtractBee, the question we kept coming back to was: why do existing document extraction tools require templates? Our answer is a zero-template, AI-first approach to document extraction — understanding rather than pattern-matching.

The answer is that traditional OCR tools don't understand documents — they recognise text and apply rules. "Invoice number is in the top-right quadrant." "Total is the last number in the rightmost column." These rules work until a vendor changes their layout, and then they break silently.

We wanted extraction that works the way a human reads an invoice: by understanding what the document is and what each piece of information means, not by matching text to pre-defined positions.

Here's how we built it.

The Core Problem with Template-Based Extraction

Template-based extraction fails in predictable ways:

Layout changes. A supplier upgrades their accounting software and their invoice format changes. Your template breaks. You don't know until someone notices the wrong data in your accounting system.

Vendor diversity. If you process invoices from 50 vendors, you need 50 templates. Each one takes time to build and maintain. As your vendor base grows, so does your maintenance burden.

New vendors. Every new supplier requires a new template before their first invoice can be processed. There's no "just works" for documents you've never seen before.

Edge cases. Real-world invoices don't always follow the template. A vendor adds an extra line item section, or moves their VAT breakdown to a footnote. The template breaks on that specific invoice but works on all the others.

We needed an extraction approach that handles all of these without configuration.

The Extraction Pipeline

ExtractBee's extraction pipeline has four stages:

Input (PDF / Image)
        ↓
Stage 1: Document Classification
        ↓
Stage 2: OCR Pre-processing (if needed)
        ↓
Stage 3: LLM Extraction (Gemini primary, Claude fallback)
        ↓
Stage 4: Position Resolution + Confidence Scoring
        ↓
Output (JSON + field locations)

Stage 1: Document Classification

Before extraction runs, we classify the document type. The classification uses the first 500 characters of extracted text (or a downscaled image thumbnail for image-only inputs) and returns one of: invoice, receipt, purchase_order, bank_statement, resume, bill_of_lading, or unknown.

Classification determines:

  • Which extraction schema to apply (what fields to extract)
  • Which system prompt variant to use
  • What max_tokens to allocate for the response

We use a lightweight classifier for this — fast and cheap, since classification only needs to make a categorical decision, not extract structured data.

Stage 2: OCR Pre-processing

Not all PDFs contain extractable text. A scanned invoice is an image embedded in a PDF container — there's no text layer to read.

We detect image-based PDFs by attempting text extraction and checking the result length. If the extracted text is below a threshold (typically fewer than 50 characters per page for a document that clearly has content), we treat it as image-based and run OCR.

Our OCR stack:

  • pdf2pic to convert PDF pages to PNG images at 300 DPI
  • Tesseract.js for character recognition, configured with PSM_AUTO (automatic page segmentation mode)
  • A text cleaning step: remove empty lines, collapse multiple spaces, strip non-semantic special characters

After OCR, we have clean text. The confidence score from Tesseract (aggregated across all characters on the page) feeds into our overall confidence calculation.

For digital PDFs: we skip OCR entirely and extract the text layer directly using pdfjs-dist. This is faster, cheaper, and more accurate than OCR on documents that already have a text layer.

Stage 3: LLM Extraction (Gemini primary, Claude fallback)

This is the core of the pipeline. We send the cleaned document text to our extraction model — Google Gemini 2.5 Flash (gemini-2.5-flash) as the primary provider, with Anthropic Claude Sonnet 4.6 (claude-sonnet-4-6) as an automatic fallback when the primary fails with a recoverable error (5xx, content blocked, quota exceeded). The request includes:

A system prompt that:

  • Describes the extraction task and expected output schema
  • Specifies the document type (from Stage 1)
  • Lists all fields to extract with their types and validation rules
  • Instructs the model to return minified JSON only
  • On the Claude fallback path, uses cache_control: ephemeral on the system prompt block — the prompt is identical for all extractions of the same document type, so we benefit from Anthropic's prompt caching

The document text as the user message.

Example extraction schema for invoices:

interface InvoiceExtraction {
  vendor_name: string | null;
  vendor_address: string | null;
  vendor_vat: string | null;
  invoice_number: string | null;
  invoice_date: string | null; // ISO 8601
  due_date: string | null;     // ISO 8601
  purchase_order: string | null;
  currency: string | null;     // ISO 4217
  line_items: LineItem[];
  subtotal: number | null;
  discount: number | null;
  vat_rate: number | null;
  vat_amount: number | null;
  total: number | null;
  iban: string | null;
  bic: string | null;
  payment_terms: string | null;
}

Key decisions in the LLM integration:

Temperature: 0. Extraction is a deterministic task — we want the same document to produce the same output every time. Temperature 0 eliminates randomness.

Minified JSON output. We instruct the model to return minified JSON with no whitespace, no explanation, no markdown. This reduces output token count by 15–20%.

File hash deduplication. Before calling the model, we compute a SHA-256 hash of the file buffer and check Redis. If we've extracted this exact file before (within 7 days), we return the cached result without an API call. This eliminates cost on duplicate submissions entirely.

Prompt caching (Claude fallback path). The system prompt for each document type is hundreds of tokens. When a request falls back to Claude, Anthropic's prompt caching lets us pay full price on the first call for each document type within a cache window, and approximately 10% on subsequent calls — saving roughly 25% on input token costs on that path.

Stage 4: Position Resolution + Confidence Scoring

After the model returns the extracted JSON, we run position resolution: mapping each extracted field value back to its location on the original document.

Why this matters: Our human-in-the-loop review workflow shows the original document with extracted fields highlighted. For this to work, we need to know where on the page each field appears. The model gives us the values — position resolution gives us the coordinates.

How it works:

  1. For each extracted field value, we search the document text for the string (or a fuzzy match within edit distance 2, to handle minor OCR character errors)
  2. We use the character offset in the text stream to calculate the approximate page and bounding box coordinates
  3. These coordinates are stored alongside the extracted values in the database

Confidence scoring: Each field gets a confidence score between 0 and 1, calculated from:

  • The model's implicit confidence (fields returned as null when uncertain vs a specific value)
  • OCR confidence from Tesseract (for scanned documents)
  • Validation rules (e.g. does the IBAN pass checksum validation? Does the date parse correctly in ISO 8601?)
  • Cross-validation (does subtotal + VAT ≈ total within rounding tolerance?)

Fields below the workspace's confidence threshold are flagged for Human Review.

The Tech Stack

Runtime: Node.js 22 (LTS) Language: TypeScript Framework: Hono (lightweight, TypeScript-native, fast cold starts) ORM: Prisma with PostgreSQL Queue: BullMQ with Redis (document processing jobs run asynchronously) File storage: Cloudflare R2 (S3-compatible, zero egress fees) OCR: Tesseract.js (no binary dependencies, runs in Node.js process) AI: Google Gemini 2.5 Flash (primary) via the Gemini API + Anthropic Claude Sonnet 4.6 (fallback) via @anthropic-ai/sdk Cache: Redis (file hash deduplication, rate limiting, BullMQ) Deploy: Self-managed server, deployed via rsync + Docker Compose

Performance and Cost

Processing time per document:

  • Digital PDF: 3–8 seconds end-to-end
  • Scanned PDF (1 page): 8–15 seconds (OCR adds time)
  • Scanned PDF (10 pages): 25–45 seconds

Cost per page (with optimisations):

  • Primary path (Gemini 2.5 Flash): roughly 6x cheaper than the Claude figures below — on the order of ~$0.001–0.002 per page
  • Claude fallback path, prompt caching active: ~$0.006–0.010 per page
  • Claude fallback path, without cache (first call per document type): ~$0.013 per page
  • Duplicate submission: $0 (Redis cache hit)

Optimisations implemented:

  • Gemini 2.5 Flash as the primary extractor (~6x lower per-page cost than the Claude fallback)
  • Prompt caching on system prompts on the Claude fallback path (cache_control: ephemeral)
  • Minified JSON output (15–20% output token reduction)
  • OCR text cleaning before extraction (reduces input tokens by ~22%)
  • File hash deduplication in Redis (eliminates cost on duplicates)
  • Batch API for Pro and Enterprise plans (cost reduction for non-real-time jobs)
  • Temperature: 0 (no impact on cost, improves consistency)

Planned:

  • Per-document-type dynamic max_tokens (reduces output token allocation for simple document types)

Multi-tenancy and Data Isolation

Every tenant's data is encrypted with a unique key derived from their tenant ID using AES-256-GCM. Even at the database level, one tenant's extraction results are not readable by another tenant's key.

File storage on Cloudflare R2 uses per-tenant prefixes with access controlled at the application layer. Direct URL access to R2 objects is not permitted — all file access goes through the ExtractBee API, which enforces tenant isolation.

What We'd Do Differently

Start with BullMQ from day one. We initially processed extractions synchronously in the API handler. When extraction takes 10–40 seconds, this creates timeout issues and poor UX. The queue-based approach (BullMQ for asynchronous job processing) should have been the starting architecture.

Invest in the position resolver earlier. The position resolver — mapping extracted values to document coordinates — was a late addition. It's one of the most valuable features for the Human Review UI, and building it retrofit was harder than building it from the start.

Confidence scoring needs more signals. Our current confidence score is a reasonable proxy but not a calibrated probability. We'd invest more in cross-validation rules (VAT calculation verification, date range plausibility checks, IBAN checksum) as additional signals.

Open Questions

We're still working through several architectural questions:

How should we handle very long bank statements? A 200-transaction bank statement across 20 pages pushes context window limits for some models. We're evaluating chunked extraction with result merging vs summarisation approaches.

When does fine-tuning beat prompting? For some document types with consistent but unusual layouts (specific industry standards, regional formats), fine-tuned models might outperform prompting. We haven't validated this at scale yet.

How do we make confidence scores meaningful? A confidence score of 0.82 should mean something specific — but calibrating what it means in practice requires enough extraction data to measure against ground truth. This becomes more tractable as volume grows.

If you're building something similar and want to compare notes, we're happy to talk. Reach us at hey@extractbee.com.

Try ExtractBee free →

ExtractBee is operated by MB Dokigo, Vilnius, Lithuania.