> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluate Receipt Agents with an Image Judge
> Trace receipt-image extraction in Arize AX and use an image-aware LLM judge to evaluate whether structured output is visually grounded.
[Evaluate Receipt Agents with an Image Judge - guide walkthrough](https://www.youtube.com/embed/HB2cXDIVsWk)
> [card] [Runnable Gradio app](https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/receipt_image_evals)
>
> Run the companion app with 14 fictional receipt images and URL-based trace inputs.
## Overview
In this tutorial, you will trace a receipt-extraction agent that takes an image of a receipt, and produces structured JSON with details. You will then create a span-level LLM-as-a-judge evaluator that decides whether the extracted JSON is supported by the image.
This tutorial assumes you have basic knowledge of:
* [tracing](/docs/ax/observe/tracing.md)
- [LLM-as-a-judge evaluators](/docs/ax/evaluate/evaluators/llm-as-a-judge.md)
By the end, you will be able to:
* Trace an image reference, extraction output, and image metadata together.
* Map a receipt image and extracted JSON to a visual-groundedness evaluator.
* Distinguish supported outputs, unsupported assertions, and images that require human review.
## Why visual groundedness needs its own evaluator
A JSON-shaped response from a receipt-extraction agent can be valid while still inventing a total, merchant, or line item. Text-only correctness checks cannot see the evidence. A visual-groundedness evaluator receives both the receipt image and the extraction, so it can ask one focused question: **does the output say only what the image supports?**
The companion app uses `gpt-5.4-mini` to extract receipt data. The tested evaluator configuration uses `gpt-5.6-luna`, a stronger image-aware model, to judge it. If your AX AI integration exposes a different judge model, use that model consistently in the evaluator and app metadata.
## Before you start
You will need:
* An OpenAI API key with access to `gpt-5.4-mini` for extraction and an AX AI provider integration with `gpt-5.6-luna` for judging.
* Python 3.10 or later.
* The example receipt app from [the Arize tutorials repo](https://github.com/Arize-ai/tutorials/tree/main/python/cookbooks/receipt_image_evals)
## Run the receipt intake app
Clone [Arize-ai/tutorials](https://github.com/Arize-ai/tutorials), then install and configure the companion app:
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
cd tutorials/python/cookbooks/receipt_image_evals
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
Copy the `.env.example` file to `.env`, then set the relevant API keys and Arize AX space Id. Then run the app:
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
python receipt_app.py
```
You will see a local URL printed to the terminal. Open this to see the receipt parser.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
❯ python receipt_app.py
* Running on local URL: http://127.0.0.1:7860
```
From the app, select some receipts, then click **Process expense** to process the image. From Arize AX, open the `receipt-image-evals` project, and look at the trace.
## Create the image-aware judge
Create a categorical [LLM-as-a-judge evaluator](/docs/ax/evaluate/evaluators/llm-as-a-judge.md) for visual groundedness.
**1. Create and name the evaluator**
In the left sidebar, select **Evaluators**. Click **New Evaluator**, select **LLM-as-a-Judge**, then choose **Create from blank**. Name it `Receipt Visual Groundedness`.
**2. Set the scope to span**
Set the evaluator scope to **Span**. Each `receipt.extract` span contains one self-contained receipt image input and its extracted JSON output, so the judge does not need the rest of the trace or session.
**3. Add the template**
For **Judge Model**, select your OpenAI provider integration and `gpt-5.6-luna`. Then use the following template. It gives the judge the visual-grounding rubric and separates the image-bearing input from the JSON it must assess. You will map `receipt_image` and `extraction` to span fields in a later step.
```text theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
Assess whether the structured extraction is visually grounded in the receipt
image.
{receipt_image}
{extraction}
Choose grounded only when every asserted merchant, item, amount, currency,
and total is visibly supported. Choose not_grounded for invented or
contradicted values. Choose needs_review when the image is degraded or
ambiguous enough that a reliable decision cannot be made.
Select the classification label using the configured choices. In the AX
evaluator's `explanation` field, return only valid JSON, without Markdown
fences or any other text, with exactly these fields:
{
"evaluated_extraction": ,
"judge_result": {
"label": ,
"reason":
}
}
```
**4. Add the choices**
Add these three classification choices and scores. Set the optimization direction to **maximize**, so the aggregate score represents the share of receipts with safely grounded extractions.
| Label | Score | Meaning |
| - | -: | - |
| `grounded` | `1` | The merchant, line items, values, and review decision are supported by the receipt image. |
| `not_grounded` | `0` | The result asserts content that the image does not support. |
| `needs_review` | `0` | The receipt is too degraded or ambiguous to evaluate with confidence. |
**5. Configure the data source and matching spans**
In **Configuration**, set **Data Source** to the `receipt-image-evals` project. Under **Filter Spans**, filter to `name = "receipt.extract"` so the evaluator receives only the extraction spans that contain the receipt image and structured output. Confirm that **Matching Spans** shows processed `receipt.extract` spans before continuing.
**6. Map variables to data**
Under **Map Variables to Data**, map `receipt_image` to `attributes.input.value` and `extraction` to `attributes.output.value`. Check the preview for each mapping: the first should contain the receipt image URL and the second the extracted JSON.
**7. Run continuously and backfill**
Leave **Run Continuously** enabled so new matching extraction spans are evaluated as they arrive. Turn on **One-Time Backfill** to also evaluate the processed receipt spans that already exist in the project.
**8. Create and run the evaluator**
Click **Create and Run Evaluator** to save the evaluator and start the backfill and continuous evaluation task.
## Review the eval results
Now you have the evaluator running, you can review the evaluator results for the existing receipts that have been processed, and for any new traces.
Select a trace in the project, then select its `receipt.extract` span. In the **Evaluations** section, open **Receipt Visual Groundedness** to see the label, score, and explanation. The explanation contains the JSON extraction evaluated by the judge and its visual-grounding review.
## Summary
In this tutorial, you:
* Traced receipt-image extraction with an image URL and structured JSON output.
* Created a span-level LLM judge to assess whether each extraction is visually grounded.
* Configured continuous evaluation and a backfill, then reviewed the label, score, and JSON explanation on the extraction span.
## Next steps
> [card] [Align with human judgment](/docs/ax/cookbooks/evaluate/align-llm-evals-with-human-judgment.md)
>
> Calibrate the judge against reviewed examples before relying on it for production decisions.
> [card] [Run online evals](/docs/ax/evaluate/run-evals-on-traces.md)
>
> Set sampling, filters, and monitoring for evaluators running on live traces.
> [card] [Human review](/docs/ax/evaluate/human-review.md)
>
> Send ambiguous receipt results to a review queue and use the labels to improve the evaluator.