> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluate your agent
> Turn a Signal issue into an evaluator that scores new traces for that failure, then filter to the runs that still fail.
In [Enable Signal](/docs/ax/get-started/get-started-managed-agent), you turned on issue detection and saw the recurring failures it files as issues. An issue tells you the pattern exists, but it doesn't score every new response for that failure as it arrives.
[**Evaluations**](/docs/ax/evaluate/what-are-evals) solve this. An evaluation is an automated check, either an LLM judging another LLM's output or a deterministic code check, that runs on your production data continuously. By the end of this guide, every response will be scored for the failure you care about, and you'll be able to filter to the ones that need attention.
This guide uses the **hallucination** evaluator template as its example. If Signal surfaced a different failure, pick the template that matches it instead, such as tool selection or task completion.
This is **Part 3** of 4: [Instrument](/docs/ax/get-started/get-started-tracing) → [Signal](/docs/ax/get-started/get-started-managed-agent) → **[Evaluate](/docs/ax/get-started/get-started-evaluations)** → [Improve](/docs/ax/get-started/get-started-improve-your-agent). Use the same project as [Enable Signal](/docs/ax/get-started/get-started-managed-agent). If Signal hasn't filed an issue yet, you can still create the hallucination evaluator below on this project's traces.
## Choose how you want to work
Use [Arize Skills](/docs/ax/skills/overview) to have your coding agent run [evaluations](/docs/ax/evaluate/run-evals) from your editor, [Alyx](/docs/ax/alyx) for a conversational approach inside the Arize platform, the UI for a hands-on step-by-step experience, or **Code** to run them programmatically.
Use [Arize Skills](/docs/ax/skills/overview) with your coding agent to create an [evaluator](/docs/ax/evaluate/create-evaluators), run it on traces as a [task](/docs/ax/evaluate/run-evals-on-traces#create-a-task), and export spans to inspect failures. Install the skills plugin and follow [Set up Arize with AI coding agents](/docs/ax/set-up-with-ai-assistants) for authentication and CLI setup. Then, follow the flow below.
### Step 1: Create eval
[`arize-evaluator`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-evaluator/SKILL.md)
The skill only covers **[LLM-as-a-Judge evaluators](/docs/ax/evaluate/create-evaluators)**. In your prompt, name the evaluator, state which template fits what you want to test (for example tool selection, task completion, or hallucination), and tell it which project the evaluator is for and how your span columns map to the template's inputs. For example, you might say:
> Create a hallucination evaluator for my project using the hallucination template. Map the input, output, and context columns to my span attributes.
Note that templates are a starting point - most teams customize the prompt criteria to match their specific rubric. Once the evaluator is created, you can ask your agent to revise it, such as:
> Update the evaluator's criteria: label the output "hallucinated" if it makes any claim that isn't supported by the provided context, and "factual" only if every claim can be traced back to the context.
### Step 2: Create a task to run your evaluator
[`arize-evaluator`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-evaluator/SKILL.md)
A [task](/docs/ax/evaluate/run-evals-on-traces#create-a-task) connects an [evaluator](/docs/ax/evaluate/create-evaluators) to your project and defines cadence and sampling. See [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces) for the full UI and configuration options.
For example, you might say:
> Set up a task to run my evaluator continuously on incoming traces.
### Step 3: See evaluation results on your traces
[`arize-trace`](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-trace/SKILL.md)
After an [eval task](/docs/ax/evaluate/run-evals-on-traces#create-a-task) has written labels to spans, export failures for triage. See [Viewing results](/docs/ax/evaluate/run-evals-on-traces#viewing-results) for where scores appear in the UI.
For example, you might say:
> Export spans from my project where my evaluator failed this week
Open [Alyx](/docs/ax/alyx) anywhere in the Arize platform and describe what you want in plain text. Alyx will guide you through each step - creating [evaluators](/docs/ax/evaluate/create-evaluators), setting up [tasks](/docs/ax/evaluate/run-evals-on-traces#create-a-task), and analyzing results - conversationally.
### Step 1: Create eval
Describe what you need in plain text for an **LLM-as-a-Judge** [evaluator](/docs/ax/evaluate/create-evaluators), including the eval name, the template that fits what you want to test, the judge model, and which span columns map to input, output, and context. For **code-based evaluators**, use **Evaluators** → **New Evaluator** → **Code** in the UI or see [Code evaluations](/docs/ax/evaluate/evaluators/code-evaluations). For example, you might say:
> Create an evaluator using GPT-4o as the judge and the template that fits what I want to test, and map the input, output, and context columns to my span attributes.
Note that templates are a starting point - most teams customize the prompt criteria to match their specific rubric. Once the evaluator is created, you can ask Alyx to revise it, such as:
> Update the evaluator's criteria: label the output "hallucinated" if it makes any claim that isn't supported by the provided context, and "factual" only if every claim can be traced back to the context.
### Step 2: Create a task to run your evaluator
This mirrors creating a [task](/docs/ax/evaluate/run-evals-on-traces#create-a-task) in the Evaluators UI. See [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces).
For example, you might say:
> Create a task to run my evaluator on my project continuously on new traces at 100% sampling
### Step 3: See evaluation results on your traces
Alyx can filter on the same [evaluation](/docs/ax/evaluate/run-evals-on-traces#viewing-results) fields that [tasks](/docs/ax/evaluate/run-evals-on-traces#create-a-task) write to each trace.
For example, you might say:
> Show me traces that failed evaluation from my hallucination evaluator this week and explain what went wrong
Use the Arize AX UI in **Evaluators** and **Traces** to configure your project's [evals](/docs/ax/evaluate/create-evaluators) end to end.
### Step 1: Understand the two types of evaluations
Arize AX supports two kinds of evaluators. **[LLM-as-a-Judge evaluators](/docs/ax/evaluate/create-evaluators)** use an LLM to assess quality - great for subjective dimensions like helpfulness or groundedness that are hard to check with code. **Code-based** [evaluators](/docs/ax/evaluate/create-evaluators) are deterministic Python checks, ideal for objective conditions like empty responses or keyword presence. We'll focus on LLM-as-a-Judge for this example. For code-based evaluators, see [Create evaluators](/docs/ax/evaluate/create-evaluators).
### Step 2: Create an evaluator
You can start from the Signal issue you found in [Enable Signal](/docs/ax/get-started/get-started-managed-agent), or create an evaluator from scratch.
* **From a Signal issue:** On the **Signal** tab, open the issue and click **Create Evaluator** in the Next Steps section. Select the evaluator template that matches the failure. The form opens already scoped to your project's traces and set to run continuously on new data, so this path also creates the Step 3 task for you.
* **From scratch:** In the left sidebar, click **Evaluators**, then **New Evaluator**. Select the evaluator template that best aligns with what you want to test. You might check whether the agent chose the right tool, completed the task, or returned a hallucinated response.
Then, configure the evaluator:
1. Give your evaluator a name.
2. Select your LLM provider and model (e.g., OpenAI GPT-4o)
3. Review the template and customize the criteria to match your rubric, or leave it as-is to get started quickly
4. Save and run the evaluator if you started from a Signal issue, or click **Create Evaluator** if you started from scratch.
### Step 3: Create a task to run your evaluator
If you created and ran your evaluator from a Signal issue, this step is already done: the same flow created the task. Skip to Step 4.
An evaluator on its own is just a template. To run it on your data, create a [**task**](/docs/ax/evaluate/run-evals-on-traces#what-is-a-task), an automation that applies your evaluator to incoming traces.
1. Click **New Task** and select the evaluator type you created.
2. Click **Add Evaluator** and select the evaluator you created in the previous step.
3. Set the data source as your project, the cadence to **Run continuously on new incoming data**, and sampling to **100%**
4. Map your span attributes to the template variables
5. Click **Create Task**
### Step 4: See evaluation results on your traces
Wait a couple of minutes, then go back to your project. You'll see [evaluation](/docs/ax/evaluate/run-evals-on-traces#viewing-results) scores on each trace. Filter by score to find failures and click into any trace to see the evaluator's label, score, and explanation.
Run this workflow from the [Python SDK](/docs/api-clients/python/overview), [TypeScript SDK](/docs/api-clients/typescript/version-1/overview), or [`ax` CLI](/docs/api-clients/cli/overview). Some features are in alpha or beta - please check individual reference pages for details.
| Step | Python SDK | TypeScript SDK | CLI |
| ------------------------------------- | ------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| Create an evaluator | [Link](/docs/api-clients/python/version-8/client-resources/evaluators#create-an-evaluator) | [Link](/docs/api-clients/typescript/version-1/client-resources/evaluators#create-an-evaluator) | [Link](/docs/api-clients/cli/evaluators#ax-evaluators-create) |
| Create a task to run your evaluator | [Link](/docs/api-clients/python/version-8/client-resources/tasks#create-a-task) | [Link](/docs/api-clients/typescript/version-1/client-resources/tasks#create-a-task) | [Link](/docs/api-clients/cli/tasks#ax-tasks-create) |
| See evaluation results on your traces | [Link](/docs/api-clients/python/version-8/client-resources/spans#filter-spans) | [Link](/docs/api-clients/typescript/version-1/client-resources/spans#list-spans) | [Link](/docs/api-clients/cli/spans#ax-spans-export) |
## Congratulations!
Every response your agent generates is now automatically scored for the failure you care about. Signal told you the issue exists; your evaluator now measures it on each new run, so you can filter straight to the responses that failed and see what went wrong. Learn more about how scoring works in the [evaluators reference](/docs/ax/evaluate/evaluators).
Your scores have probably confirmed a pattern: some responses fail because your agent was never told how to avoid the failure. A common case: the system prompt says "be helpful" but never tells the agent to stick to retrieved context or to say "I don't know." That's a prompt problem, and Signal may even have proposed a change already. Don't ship it on faith: the next guide is how you prove it works first.
**Next up:** build a dataset from the failing traces, fix the prompt in the Prompt Playground, and run experiments that compare this evaluator's scores before and after the change.