Evaluators

The evaluators client methods are currently in BETA. The API may change without notice. A one-time warning is emitted on first use.

Create and manage LLM-as-judge evaluators and their versions programmatically. Evaluators use prompt templates with {variable} placeholders that reference span or trace attributes to automatically score your LLM application’s outputs.

Key Capabilities

Create template-based LLM-as-judge evaluators within a space
Version evaluators with commit messages (versions are immutable once created)
Retrieve evaluators with their latest or a specific version
List, update, and delete evaluators
List and retrieve individual evaluator versions

List Evaluators

List all evaluators you have access to, with optional filtering by space.

resp = client.evaluators.list(
    space="your-space-name-or-id",  # optional
    name="Relevance",               # optional substring filter
    limit=50,
)

for evaluator in resp.evaluators:
    print(evaluator.id, evaluator.name)

For details on pagination, field introspection, and data conversion (to dict/JSON/DataFrame), see Response Objects.

Create a Template (LLM-as-Judge) Evaluator

Create a new template evaluator with an initial version. Evaluator names must be unique within the target space.

from arize.evaluators.types import TemplateConfigInput, EvaluatorLlmConfigRequest, OptimizationDirection

evaluator = client.evaluators.create_template_evaluator(
    name="Relevance",
    space="your-space-name-or-id",
    commit_message="Initial version",
    description="Scores whether the response is relevant to the query",
    template_config=TemplateConfigInput(
        name="Relevance",
        template="Is the following response relevant to the query?\nQuery: {input.value}\nResponse: {output.value}",
        include_explanations=True,
        use_function_calling=True,
        classification_choices={"relevant": 1, "irrelevant": 0},
        direction=OptimizationDirection.MAXIMIZE,
        llm_config=EvaluatorLlmConfigRequest(
            ai_integration_id="your-ai-integration-id",
            model_name="gpt-4o",
            invocation_parameters={"temperature": 0},
            provider_parameters={},
        ),
    ),
)

print(evaluator.id, evaluator.name)

Create a Code Evaluator

Create a new code evaluator with an initial version. Use ManagedCodeConfigRequest for built-in checks (JSONParseable, Regex, KeywordMatch, ExactMatch) or CustomCodeConfigRequest for user-supplied Python.

from arize.evaluators.types import ManagedCodeConfigRequest

evaluator = client.evaluators.create_code_evaluator(
    name="JSON Parseable",
    space="your-space-name-or-id",
    commit_message="Initial version",
    code_config=ManagedCodeConfigRequest(
        type="managed",
        name="json_parseable",
        managed_evaluator="JSONParseable",
        variables=["output"],
    ),
)

print(evaluator.id, evaluator.name)

Evaluator name must match the regex ^[a-zA-Z0-9_\s\-&()]+$.

Template Variables

Template strings use {variable} placeholders (f-string format) that reference span or trace attributes (e.g., {input.value}, {output.value}, {attributes.my_custom_attr}).

Classification vs. Freeform Output

Classification — Provide classification_choices as a dict[str, float] mapping label → numeric score (e.g., {"relevant": 1, "irrelevant": 0}). The evaluator outputs one of these labels along with its score.
Freeform — Omit classification_choices. The evaluator produces a numeric score without predefined labels.

Create a Remote Evaluator

Create a new remote evaluator with an initial version. Remote evaluators call a customer-hosted HTTP endpoint on each evaluation run. The endpoint must already exist as an EVALUATOR integration, and the enableRemoteEvalTasks feature flag must be enabled on the account.

evaluator = client.evaluators.create_remote_evaluator(
    name="External Judge",
    space="your-space-name-or-id",
    integration_id="your-evaluator-integration-id",
    commit_message="Initial version",
    description="Delegates scoring to a customer-hosted endpoint",  # optional
)

print(evaluator.id, evaluator.name)

Create a Decision Judge Evaluator

Create a new Decision Judge evaluator with an initial version. A Decision Judge evaluator asks a set of typed questions about each span and writes one result per question (eval.<evaluator name> <question name>.label / .score). It runs on spans only, and Decision Judge must be enabled for the account (otherwise the API returns 403). Each question is a BOOLEAN, CHOICE, or SCORE question wrapped in a DecisionJudgeQuestionInput. Provide between 1 and 64 questions; question names must be unique within the evaluator (case-insensitive).

from arize.evaluators.types import (
    DecisionJudgeConfigInput,
    DecisionJudgeQuestionInput,
    DecisionJudgeBooleanQuestionInput,
    DecisionJudgeBooleanCriteriaInput,
    DecisionJudgeScoreQuestionInput,
    DecisionJudgeScoreCriteriaInput,
    OptimizationDirection,
)

evaluator = client.evaluators.create_decision_judge_evaluator(
    name="Response Quality",
    space="your-space-name-or-id",
    commit_message="Initial version",
    description="Scores spans on relevance and overall quality",  # optional
    decision_judge_config=DecisionJudgeConfigInput(
        ai_integration_id="your-ai-integration-id",
        model_name="gpt-6-luna",  # a Decision Judge model for the integration's provider
        state_template="Query: {input.value}\nResponse: {output.value}",
        questions=[
            DecisionJudgeQuestionInput(
                DecisionJudgeBooleanQuestionInput(
                    name="is_relevant",
                    type="BOOLEAN",
                    instructions="Is the response relevant to the query?",
                    criteria=DecisionJudgeBooleanCriteriaInput(
                        true_means="The response addresses the query",
                        false_means="The response is off-topic",
                    ),
                )
            ),
            DecisionJudgeQuestionInput(
                DecisionJudgeScoreQuestionInput(
                    name="quality",
                    type="SCORE",
                    instructions="Rate the overall quality of the response.",
                    criteria=DecisionJudgeScoreCriteriaInput(
                        levels=["poor", "acceptable", "excellent"],  # 2–10, lowest to highest
                    ),
                    direction=OptimizationDirection.MAXIMIZE,  # optional; defaults to MAXIMIZE
                )
            ),
        ],
    ),
)

print(evaluator.id, evaluator.name)

Question shapes: BOOLEAN takes optional true_means / false_means; CHOICE takes options (2–255 labels mapped to when each applies) via DecisionJudgeChoiceCriteriaInput; SCORE takes levels (2–10 descriptions from lowest to highest) via DecisionJudgeScoreCriteriaInput. You may also pass decision_judge_config as a plain dict with ai_integration_id, model_name, state_template, and questions keys.

Get an Evaluator

Retrieve an evaluator by name or ID. By default the latest version is returned. When using a name, provide space to disambiguate.

evaluator = client.evaluators.get(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
)

print(evaluator.id, evaluator.name)
print(evaluator.version)

Get a Specific Version

evaluator = client.evaluators.get(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    version_id="specific-version-id",
)

Update an Evaluator

Update an evaluator’s metadata (name and/or description). To change the template configuration, create a new version instead.

evaluator = client.evaluators.update(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    name="Relevance v2",
    description="Updated description",
)

print(evaluator)

Delete an Evaluator

Delete an evaluator and all its versions. This operation is irreversible. There is no response from this call.

client.evaluators.delete(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
)

print("Evaluator deleted successfully")

Manage Versions

Evaluator versions are immutable once created. To change the template configuration, create a new version — it becomes the latest version immediately.

List Versions

List all versions for an evaluator.

resp = client.evaluators.list_versions(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    limit=50,
)

for version in resp.evaluator_versions:
    print(version.id, version.commit_message)

For details on pagination, field introspection, and data conversion (to dict/JSON/DataFrame), see Response Objects.

Get a Version

Retrieve a specific evaluator version by its ID.

version = client.evaluators.get_version(version_id="your-version-id")

print(version.id, version.commit_message)

Create a New Template Version

Add a new template version to an existing template evaluator. The new version becomes the latest immediately.

from arize.evaluators.types import TemplateConfigInput, EvaluatorLlmConfigRequest, OptimizationDirection

version = client.evaluators.create_template_version(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    commit_message="Improved prompt for edge cases",
    template_config=TemplateConfigInput(
        name="Relevance",
        template="Rate the relevance of the response on a scale of 0 to 1.\nQuery: {input.value}\nResponse: {output.value}",
        include_explanations=True,
        use_function_calling=True,
        classification_choices={"relevant": 1, "irrelevant": 0},
        direction=OptimizationDirection.MAXIMIZE,
        llm_config=EvaluatorLlmConfigRequest(
            ai_integration_id="your-ai-integration-id",
            model_name="gpt-4o",
            invocation_parameters={"temperature": 0},
            provider_parameters={},
        ),
    ),
)

print(version.id)

Create a New Code Version

Add a new code version to an existing code evaluator.

from arize.evaluators.types import ManagedCodeConfigRequest

version = client.evaluators.create_code_version(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    commit_message="Updated managed evaluator",
    code_config=ManagedCodeConfigRequest(
        type="managed",
        name="json_parseable",
        managed_evaluator="JSONParseable",
        variables=["output"],
    ),
)

print(version.id)

Create a New Remote Version

Add a new remote version to an existing evaluator. The new version becomes the latest immediately. The referenced integration must be an EVALUATOR integration, and the enableRemoteEvalTasks feature flag must be enabled on the account.

version = client.evaluators.create_remote_version(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    integration_id="your-evaluator-integration-id",
    commit_message="Point at updated endpoint integration",
)

print(version.id)

Create a New Decision Judge Version

Add a new Decision Judge version to an existing Decision Judge evaluator. The new version becomes the latest immediately and replaces the whole configuration, including every question — pass all the questions the version should ask.

from arize.evaluators.types import (
    DecisionJudgeConfigInput,
    DecisionJudgeQuestionInput,
    DecisionJudgeBooleanQuestionInput,
    DecisionJudgeBooleanCriteriaInput,
)

version = client.evaluators.create_decision_judge_version(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    commit_message="Add a relevance question",
    decision_judge_config=DecisionJudgeConfigInput(
        ai_integration_id="your-ai-integration-id",
        model_name="gpt-6-luna",
        state_template="Query: {input.value}\nResponse: {output.value}",
        questions=[
            DecisionJudgeQuestionInput(
                DecisionJudgeBooleanQuestionInput(
                    name="is_relevant",
                    type="BOOLEAN",
                    instructions="Is the response relevant to the query?",
                    criteria=DecisionJudgeBooleanCriteriaInput(
                        true_means="The response addresses the query",
                        false_means="The response is off-topic",
                    ),
                )
            ),
        ],
    ),
)

print(version.id)

Delete Versions

Delete a batch of evaluator versions (1-100 IDs per request). This operation is irreversible. The delete is partial-tolerant: IDs that exist and belong to the evaluator are deleted, and requested IDs that were not deleted are returned in not_deleted_version_ids. Deleting a version pinned to a running online task un-pins that task (falls back to latest).

resp = client.evaluators.delete_versions(
    evaluator="your-evaluator-name-or-id",
    space="your-space-name-or-id",  # required when resolving by evaluator name
    version_ids=["your-version-id-1", "your-version-id-2"],
)

print(resp.completed)
print(resp.deleted_version_ids)
print(resp.not_deleted_version_ids)

Learn more: Online Evaluations Documentation

Skills

MCP server

API reference

API clients

Key Capabilities

List Evaluators

Create a Template (LLM-as-Judge) Evaluator

Create a Code Evaluator

Template Variables

Classification vs. Freeform Output

Create a Remote Evaluator

Create a Decision Judge Evaluator

Get an Evaluator

Get a Specific Version

Update an Evaluator

Delete an Evaluator

Manage Versions

List Versions

Get a Version

Create a New Template Version

Create a New Code Version

Create a New Remote Version

Create a New Decision Judge Version

Delete Versions

​Key Capabilities

​List Evaluators

​Create a Template (LLM-as-Judge) Evaluator

​Create a Code Evaluator

​Template Variables

​Classification vs. Freeform Output

​Create a Remote Evaluator

​Create a Decision Judge Evaluator

​Get an Evaluator

​Get a Specific Version

​Update an Evaluator

​Delete an Evaluator

​Manage Versions

​List Versions

​Get a Version

​Create a New Template Version

​Create a New Code Version

​Create a New Remote Version

​Create a New Decision Judge Version

​Delete Versions

Key Capabilities

List Evaluators

Create a Template (LLM-as-Judge) Evaluator

Create a Code Evaluator

Template Variables

Classification vs. Freeform Output

Create a Remote Evaluator

Create a Decision Judge Evaluator

Get an Evaluator

Get a Specific Version

Update an Evaluator

Delete an Evaluator

Manage Versions

List Versions

Get a Version

Create a New Template Version

Create a New Code Version

Create a New Remote Version

Create a New Decision Judge Version

Delete Versions