Menu OCR Pipeline (Bedrock Vision)
A Vision-Model OCR Pipeline for Restaurant Menus
A restaurant technology company had a large stack of menu documents in inconsistent formats and needed the contents as structured data. I built a vision-model OCR pipeline that reads a menu image, sends it to Claude 3 on AWS Bedrock with a prompt that pins the output schema, and turns the reply into clean JSON with correct Turkish characters. Two model variants, Claude 3 Haiku and Claude 3 Sonnet, were built against the same prompt so the client could weigh extraction quality against cost before picking one for production.
The problem
The client sells menu and ordering technology to restaurants and had a large backlog of menu documents to digitise, mostly Turkish-language, in mixed formats and layouts with no shared structure. A generic OCR pass returns flat text with no idea of what is a category, a product name, a description, or a price, so someone still has to read and re-key every menu by hand. What they needed was image in, structured menu data out: category, product, description, price, discounted price, and their special price field, in one pass, without a custom OCR model trained per restaurant.
Constraints
- ·Menus vary in layout per restaurant, so the schema had to come from the prompt, not a per-template parser.
- ·Content is Turkish, so extracted text had to preserve Turkish characters correctly, not garbled ASCII.
- ·No budget or time for training a custom OCR/layout model. The solution had to use an existing vision-capable model.
- ·Cost mattered enough that a cheaper and a stronger model both needed testing before committing to one.
Architecture
A menu image is read from disk, base64-encoded, and sent as an image content block to Claude 3 on AWS Bedrock (Haiku or Sonnet) alongside a text prompt that pins the six output fields and demands a single JSON object. The model's reply is prose-wrapped, so a regex pass carves out the first `{...}` block before `json.loads`. A second pass, dataprocessing.py, walks the parsed JSON and replaces escaped Turkish unicode with the real characters, then re-saves it as a `fixed_` file with `ensure_ascii=False`. The two model variants (Haiku, Sonnet) run the identical prompt independently so their outputs can be compared side by side.
- ›boto3 bedrock-runtime client calling invoke_model
- ›Claude 3 Haiku (anthropic.claude-3-haiku-20240307-v1:0): batch script looping folders and images
- ›Claude 3 Sonnet (anthropic.claude-3-sonnet-20240229-v1:0): single-image script for comparison
- ›Regex JSON carve: `re.search(r'\{.*\}', text, re.DOTALL)` before json.loads
- ›dataprocessing.py: recursive Turkish-unicode fix, writes fixed_<name>.json
- ›Local PNG menu images and a 100-document source corpus (90 .xlsx, 9 .pdf, 1 .docx)
How one request flows
- 01Read the menu PNG from disk and base64-encode it.
- 02Build a Bedrock request: the image as a content block plus a fixed text prompt naming the six output fields (category, product, description, price, discounted price, special price) and demanding a single JSON object.
- 03Send the request to Claude 3 on Bedrock (Haiku in the batch script, Sonnet in the single-image script) with max_tokens 4000.
- 04Pull the text block out of the response content list.
- 05Regex-search the text for the first `{...}` block and parse it as JSON, so surrounding prose from the model does not break the parse.
- 06On a parse failure, the Haiku script preserves the raw response as {error, raw_response} instead of dropping it. The Sonnet script writes an empty object on failure.
- 07Run dataprocessing.py to walk the saved JSON and replace escaped Turkish unicode with real characters, writing a fixed_<name>.json alongside the raw one.
Engineering decisions
Prompt-pinned schema instead of a trained layout model
What. A fixed prompt lists the exact six output fields and forces a single JSON object, rather than training or fine-tuning an OCR/layout model.
Why. Menu layouts differ per restaurant and there was no time or budget for a custom model. A vision-capable LLM with a pinned schema gets structured output on day one.
Tradeoff. Less control than a trained parser and the model can still drift, which is why the reply always needs a regex carve rather than a direct parse.
Regex carve before JSON parse
What. The model's reply is searched for the first `{...}` block with `re.DOTALL` before calling json.loads.
Why. Claude's Bedrock reply is not guaranteed pure JSON, it can wrap the object in explanatory prose. Parsing the raw reply directly would fail on that prose.
Tradeoff. A regex carve is fragile against nested braces in free text, but for a single top-level JSON object in a short reply it was reliable enough in practice.
Two model variants run side by side, not picked upfront
What. The identical prompt and image set were run through both Claude 3 Haiku (cheap, batch) and Claude 3 Sonnet (single image, stronger), rather than committing to one model first.
Why. Haiku is fast and cheap but weaker on dense menu layouts. Sonnet reads denser layouts more reliably but costs more per call. Running both on the same input is the only honest way to compare them before choosing.
Tradeoff. Extra engineering time to keep two variants, worth it to have real comparison data instead of guessing which model tier the client needs.
Turkish character fix as a separate pass, not inline
What. dataprocessing.py runs after the extraction, as its own script, rather than fixing unicode inside API.py.
Why. Keeps the raw model output around for inspection or debugging, and the fix (walk the dict, replace escapes, re-serialise with ensure_ascii=False) is a separate concern from the extraction itself.
Tradeoff. An extra step in the pipeline instead of one combined script, kept because it makes the raw extraction available for comparison after normalisation.
Failure handling differs between the two variants
What. The Haiku script catches a parse failure and saves {error, raw_response} so a failed extraction can still be inspected. The Sonnet script does not; it saves an empty object on the same failure.
Why. The Haiku script was written second, after seeing parse failures in practice, and added the safety net. The Sonnet script stayed as the earlier single-shot version.
Tradeoff. This is a real gap in the Sonnet script, noted here rather than glossed over. It was acceptable because Sonnet was the comparison run, not the production candidate.
What changed along the way
- →Started with a single hardcoded image and model call (what became sonnet3/API.py) to prove the extraction worked at all, before building the batch loop.
- →Added the error/raw_response fallback to the Haiku script after seeing the model occasionally fail to return clean JSON, so failed runs stay inspectable instead of silently empty.
- →Kept the Turkish character fix as its own script once it became clear escaped unicode was showing up in every extraction, not just an occasional edge case.
Security and reliability
- ·Bedrock credentials are never hardcoded. boto3.client("bedrock-runtime") is called with no explicit keys, so credentials come from the default AWS credential chain at runtime.
- ·Parse failures are captured, not silently dropped, in the Haiku script (raw response saved alongside the error).
- ·The raw extracted JSON is kept alongside the Turkish-normalised fixed_ file, so a bad normalisation pass never destroys the original model output.
Metrics
What shipped
- ✓Working image-to-structured-menu extraction on AWS Bedrock with a fixed six-field schema.
- ✓Side-by-side Haiku vs Sonnet 3 comparison on the same prompt and images.
- ✓Turkish character normalisation step that makes the extracted JSON directly usable.
- ✓Parse failures are preserved as raw responses instead of silently lost (Haiku variant).
Lessons learned
- ·A pinned-schema prompt gets structured output from a vision model without training a layout parser, as long as the reply is regex-carved before parsing rather than parsed directly.
- ·Running two model tiers on the identical prompt and images is the only honest way to compare cost against extraction quality, guessing from benchmarks is not enough.
- ·Preserving the raw response on a parse failure turns a silent loss into an inspectable one, worth adding to both variants, not just one.
- ·Turkish (and other non-ASCII) text needs an explicit normalisation pass; ensure_ascii=False plus a manual escape map is a small fix with an outsized effect on usability.