> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluator and task mutations

> Create evaluators and run online eval, experiment and prompt optimization tasks. Arguments, return types and a validated example for each of the 14 mutations.

Create evaluators and run online eval, experiment and prompt optimization tasks.

For task-oriented walkthroughs of these operations, see the [Evaluator and task guide](/docs/ax/graphql-reference/guides/evaluators-and-tasks.md). Every mutation below is sent as a `POST` to `https://app.arize.com/graphql` with an `x-api-key` header; see [Forming calls](/docs/ax/graphql-reference/overview/how-to-use-graphql/forming-calls.md).

## Mutations in this page

* [`createEvaluator`](#createevaluator): Create a new evaluator
* [`createEvaluatorVersion`](#createevaluatorversion): Create a new evaluator version
* [`editEvaluator`](#editevaluator): Edit an existing evaluator
* [`deleteEvaluator`](#deleteevaluator): Delete an existing evaluator
* [`createEvalTask`](#createevaltask): Create a new online eval task
* [`patchEvalTask`](#patchevaltask): Patch eval task.
* [`patchEvalTaskWithEvalHub`](#patchevaltaskwithevalhub): Patch eval task with eval hub.
* [`migrateTaskToEvalHub`](#migratetasktoevalhub): migrates a task's evaluators to eval hub
* [`runOnlineTask`](#runonlinetask): Run an online task
* [`cancelOnlineTaskRun`](#cancelonlinetaskrun): Cancel an online task run
* [`deleteOnlineTask`](#deleteonlinetask): Delete online task.
* [`runExperimentTask`](#runexperimenttask): Run an experiment task with configurable LLM interactions
* [`createPromptOptimizationTask`](#createpromptoptimizationtask): Create a new prompt optimization task
* [`patchPromptOptimizationTask`](#patchpromptoptimizationtask): Patch prompt optimization task.

## Reference

### createEvaluator

Create a new evaluator

`createEvaluator(input: CreateEvaluatorMutationInput!): CreateEvaluatorMutationPayload`

#### Arguments

**input** `CreateEvaluatorMutationInput!` *required*

**CreateEvaluatorMutationInput fields**

**spaceId** `ID!` *required*

The space id for the evaluator

**name** `String!` *required*

The name of the evaluator

**description** `String`

The description of the saved evaluator

**commitMessage** `String!` *required*

The commit message describing the changes in this version

**templateEvaluator** `TemplateEvaluationConfigInput`

**TemplateEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the template evaluation config

**rails** `[String!]`

The rails associated with the config

**template** `String!` *required*

The template associated with the config

**turnDefinition** `String`

Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same \{variable} syntax as the main template.

**position** `Int!` *required*

The position of the config

**includeExplanations** `Boolean!` *required*

Whether the config includes explanations

**useFunctionCallingIfAvailable** `Boolean!` *required*

Whether the config uses function calling if available

**useStructuredOutputIfAvailable** `Boolean`

Whether the config uses structured output if available

**classificationChoices** `JSONObject`

The classification choices for the config (Record\<string, number>)

**direction** `TemplateEvaluationConfigDirection`

The direction for the config One of: `maximize`, `minimize`, `none`.

**dataGranularityType** `EvalDataGranularityType`

Data granularity (span, trace, or session). Defaults to span when omitted. One of: `span`, `trace`, `session`.

**llmConfig** `TemplateEvaluationLlmConfigInput`

Llm Config for this evaluator

**TemplateEvaluationLlmConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**codeEvaluator** `CodeEvaluationConfigInput`

**CodeEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the evaluation config

**evaluationClass** `String`

The evaluation type associated with the config

**evalClassCodeBlock** `String`

The code block for the evaluation class

**evaluationInputParams** `JSON`

The evaluation input params

**packageImports** `String`

The package imports for the evaluation class

**position** `Int!` *required*

The position of the config

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**spanAttributes** `[String!]`

The data column associated with the config

**dataGranularityType** `EvalDataGranularityType`

The data granularity associated with the config (span, trace, or session) One of: `span`, `trace`, `session`.

**harnessEvaluator** `HarnessEvaluatorInput`

**HarnessEvaluatorInput fields**

**llmIntegrationId** `ID!` *required*

Relay GID of the llm\_integrations row. Provider must be 'anthropic'.

**sandboxIntegrationIds** `[ID!]`

Optional relay GIDs of sandbox\_integrations to link to the new sandbox config at creation time.

**name** `EvalColumnName!` *required*

Eval column prefix name. Harness output is written under eval.\<name>.\*.

**template** `String!` *required*

Natural-language eval template. \{column\_name} markers are substituted at run time.

**classificationChoices** `JSON`

Optional fixed label set. Null means the agent discovers labels freely.

**strictChoices** `Boolean`

When true the agent must only emit labels from classificationChoices.

**llmModelName** `String`

Optional model name hint forwarded to the pod as ANTHROPIC\_MODEL.

**direction** `HarnessEvaluationConfigDirection`

Optimization direction for scored harness eval output. One of: `maximize`, `minimize`, `none`.

**remoteEvaluator** `RemoteEvaluatorInput`

**RemoteEvaluatorInput fields**

**endpoint** `String!` *required*

HTTP URL of the remote evaluation endpoint.

**headers** `JSON`

Optional custom request headers sent to the endpoint (encrypted at rest).

**inputSchema** `JSON`

Optional JSON schema describing the expected request body.

**dataGranularityType** `EvalDataGranularityType`

Data granularity (span, trace, or session). Defaults to span when omitted. Fixed at creation; later versions inherit it. One of: `span`, `trace`, `session`.

**systemOneEvaluator** `SystemOneEvaluatorInput`

**SystemOneEvaluatorInput fields**

**llmIntegrationId** `String!` *required*

Relay global ID of the TypeSafe AI integration this evaluator calls.

**modelName** `String!` *required*

Provider model route, for example `jev-latest`. Required: the column has no default, so which route was asked for is always recorded.

**stateTemplate** `String!` *required*

The author's JSON-shaped state template, stored verbatim, carrying \{variables} resolved per record.

**questions** `[SystemOneQuestionInput!]!` *required*

At least one. Every question sees the same state and is evaluated independently in one request.

**SystemOneQuestionInput fields**

**questionName** `String!` *required*

Becomes the provider's question key and the published eval column suffix, so it is validated against the eval-name rules.

**questionType** `SystemOneQuestionType!` *required*

One of: `boolean`, `choice`, `score`.

**instructions** `String!` *required*

The complete judgment. Question names are not sent to the provider as context.

**criteria** `JSON`

Required for choice and score; optional for boolean. Shape is validated per type.

**direction** `OptimizationDirection`

Read for score questions only, where it defaults to maximize. Normalized to `none` for every other type. One of: `minimize`, `maximize`, `none`.

**dataGranularityType** `EvalDataGranularityType`

Data granularity (span, trace, or session). Defaults to span when omitted. Fixed at creation; later versions inherit it. One of: `span`, `trace`, `session`.

**clientMutationId** `String`

#### Returns

**evaluator** `Evaluator!`

**Evaluator fields**

Scalar fields: `id`, `name`, `description`, `taskType`, `commitHash`, `commitMessage`, `createdAt`, `updatedAt`.

Object fields (select subfields): `versionHistory`, `webhookSubscriptions`, `config`, `harnessEvaluationConfig`, `remoteEvaluationConfig`, `systemOneEvaluationConfig`, `createdBy`, `tags`, `onlineTasks`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation CreateEvaluator($input: CreateEvaluatorMutationInput!) {
  createEvaluator(input: $input) {
    evaluator { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "spaceId": "<ID>",
    "name": "<string>",
    "commitMessage": "<string>"
  }
}
```

### createEvaluatorVersion

Create a new evaluator version

`createEvaluatorVersion(input: CreateEvaluatorVersionMutationInput!): CreateEvaluatorVersionMutationPayload`

#### Arguments

**input** `CreateEvaluatorVersionMutationInput!` *required*

**CreateEvaluatorVersionMutationInput fields**

**evaluatorId** `ID!` *required*

The evaluator ID in which the version will be created

**commitMessage** `String!` *required*

The commit message describing the changes in this version

**name** `String`

Optional evaluator name to update

**description** `String`

Optional evaluator description to update

**templateEvaluator** `TemplateEvaluationConfigInput`

**TemplateEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the template evaluation config

**rails** `[String!]`

The rails associated with the config

**template** `String!` *required*

The template associated with the config

**turnDefinition** `String`

Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same \{variable} syntax as the main template.

**position** `Int!` *required*

The position of the config

**includeExplanations** `Boolean!` *required*

Whether the config includes explanations

**useFunctionCallingIfAvailable** `Boolean!` *required*

Whether the config uses function calling if available

**useStructuredOutputIfAvailable** `Boolean`

Whether the config uses structured output if available

**classificationChoices** `JSONObject`

The classification choices for the config (Record\<string, number>)

**direction** `TemplateEvaluationConfigDirection`

The direction for the config One of: `maximize`, `minimize`, `none`.

**dataGranularityType** `EvalDataGranularityType`

Data granularity (span, trace, or session). Defaults to span when omitted. One of: `span`, `trace`, `session`.

**llmConfig** `TemplateEvaluationLlmConfigInput`

Llm Config for this evaluator

**TemplateEvaluationLlmConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**codeEvaluator** `CodeEvaluationConfigInput`

**CodeEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the evaluation config

**evaluationClass** `String`

The evaluation type associated with the config

**evalClassCodeBlock** `String`

The code block for the evaluation class

**evaluationInputParams** `JSON`

The evaluation input params

**packageImports** `String`

The package imports for the evaluation class

**position** `Int!` *required*

The position of the config

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**spanAttributes** `[String!]`

The data column associated with the config

**dataGranularityType** `EvalDataGranularityType`

The data granularity associated with the config (span, trace, or session) One of: `span`, `trace`, `session`.

**harnessEvaluator** `HarnessEvaluatorVersionInput`

**HarnessEvaluatorVersionInput fields**

**llmIntegrationId** `ID`

Relay GID of the llm\_integrations row. Provide to create a new sandbox; omit to inherit.

**name** `EvalColumnName!` *required*

Eval column prefix name. Harness output is written under eval.\<name>.\*.

**template** `String!` *required*

Natural-language eval template. \{column\_name} markers are substituted at run time.

**classificationChoices** `JSON`

Optional fixed label set. Null means the agent discovers labels freely.

**strictChoices** `Boolean`

When true the agent must only emit labels from classificationChoices.

**llmModelName** `String`

Optional model name hint forwarded to the pod as ANTHROPIC\_MODEL.

**direction** `HarnessEvaluationConfigDirection`

Optimization direction for scored harness eval output. One of: `maximize`, `minimize`, `none`.

**systemOneEvaluator** `SystemOneEvaluatorVersionInput`

**SystemOneEvaluatorVersionInput fields**

**llmIntegrationId** `String!` *required*

Relay global ID of the TypeSafe AI integration this evaluator calls.

**modelName** `String!` *required*

Provider model route, for example `jev-latest`. Required: the column has no default, so which route was asked for is always recorded.

**stateTemplate** `String!` *required*

The author's JSON-shaped state template, stored verbatim, carrying \{variables} resolved per record.

**questions** `[SystemOneQuestionInput!]!` *required*

At least one. Every question sees the same state and is evaluated independently in one request.

**SystemOneQuestionInput fields**

**questionName** `String!` *required*

Becomes the provider's question key and the published eval column suffix, so it is validated against the eval-name rules.

**questionType** `SystemOneQuestionType!` *required*

One of: `boolean`, `choice`, `score`.

**instructions** `String!` *required*

The complete judgment. Question names are not sent to the provider as context.

**criteria** `JSON`

Required for choice and score; optional for boolean. Shape is validated per type.

**direction** `OptimizationDirection`

Read for score questions only, where it defaults to maximize. Normalized to `none` for every other type. One of: `minimize`, `maximize`, `none`.

**remoteEvaluator** `RemoteEvaluatorVersionInput`

**RemoteEvaluatorVersionInput fields**

**endpoint** `String`

HTTP URL of the remote evaluation endpoint.

**headers** `JSON`

Optional custom request headers sent to the endpoint (encrypted at rest).

**inputSchema** `JSON`

Optional JSON schema describing the expected request body.

**clientMutationId** `String`

#### Returns

**evaluator** `Evaluator!`

**Evaluator fields**

Scalar fields: `id`, `name`, `description`, `taskType`, `commitHash`, `commitMessage`, `createdAt`, `updatedAt`.

Object fields (select subfields): `versionHistory`, `webhookSubscriptions`, `config`, `harnessEvaluationConfig`, `remoteEvaluationConfig`, `systemOneEvaluationConfig`, `createdBy`, `tags`, `onlineTasks`.

**evaluatorVersion** `EvaluatorVersion!`

**EvaluatorVersion fields**

Scalar fields: `id`, `evaluatorId`, `commitHash`, `commitMessage`, `createdAt`, `updatedAt`, `versionNumber`.

Object fields (select subfields): `evaluator`, `harnessEvaluationConfig`, `remoteEvaluationConfig`, `systemOneEvaluationConfig`, `createdBy`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation CreateEvaluatorVersion($input: CreateEvaluatorVersionMutationInput!) {
  createEvaluatorVersion(input: $input) {
    evaluator { id name }
    evaluatorVersion { id }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "evaluatorId": "<ID>",
    "commitMessage": "<string>"
  }
}
```

### editEvaluator

Edit an existing evaluator

`editEvaluator(input: EditEvaluatorMutationInput!): EditEvaluatorMutationPayload`

#### Arguments

**input** `EditEvaluatorMutationInput!` *required*

**EditEvaluatorMutationInput fields**

**evaluatorId** `ID!` *required*

The evaluator id to edit

**name** `String`

The new name to use for the evaluator

**description** `String`

The new description for the evaluator

**clientMutationId** `String`

#### Returns

**evaluator** `Evaluator!`

**Evaluator fields**

Scalar fields: `id`, `name`, `description`, `taskType`, `commitHash`, `commitMessage`, `createdAt`, `updatedAt`.

Object fields (select subfields): `versionHistory`, `webhookSubscriptions`, `config`, `harnessEvaluationConfig`, `remoteEvaluationConfig`, `systemOneEvaluationConfig`, `createdBy`, `tags`, `onlineTasks`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation EditEvaluator($input: EditEvaluatorMutationInput!) {
  editEvaluator(input: $input) {
    evaluator { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "evaluatorId": "<ID>"
  }
}
```

### deleteEvaluator

Delete an existing evaluator

`deleteEvaluator(input: DeleteEvaluatorMutationInput!): DeleteEvaluatorMutationPayload`

#### Arguments

**input** `DeleteEvaluatorMutationInput!` *required*

**DeleteEvaluatorMutationInput fields**

**evaluatorId** `ID!` *required*

The evaluator id to delete

**clientMutationId** `String`

#### Returns

**success** `Boolean!`

Indicates whether the evaluator was deleted

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation DeleteEvaluator($input: DeleteEvaluatorMutationInput!) {
  deleteEvaluator(input: $input) {
    success
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "evaluatorId": "<ID>"
  }
}
```

### createEvalTask

Create a new online eval task

`createEvalTask(input: CreateEvalTaskMutationInput!): CreateEvalTaskMutationPayload`

#### Arguments

**input** `CreateEvalTaskMutationInput!` *required*

**CreateEvalTaskMutationInput fields**

**modelId** `ID`

The model id for the model to be evaluated

**datasetId** `ID`

The dataset id for the dataset to be evaluated

**name** `String!` *required*

The name of the online task

**filters** `[FilterItemInputType!]`

The filters associated with the online task

**FilterItemInputType fields**

**filterType** `FilterRowType!` *required*

The type of filter. One of: `featureLabel`, `tagLabel`, `predictionValue`, `actuals`, `modelVersion`, `batchId`, `predictionClass`, `predictionScore`, `actualClass`, `actualScore`, `topKPercentile`, `spanProperty`, `llmEval`, `annotation`, `userAnnotation`.

**operator** `ComparisonOperator!` *required*

The operator of the filter. One of: `greaterThan`, `lessThan`, `equals`, `notEquals`, `greaterThanOrEqual`, `lessThanOrEqual`, `topN`, `contains`, `containsString`, `similarTo`.

**dimension** `DimensionInput`

**DimensionInput fields**

**id** `ID!` *required*

**name** `String!` *required*

The label of this dimension, e.g. 'Bank Name' or 'age'

**dataType** `DimensionDataType!` *required*

The data type of the values in dimensionName, ex. 'FLOAT', 'STRING', etc. One of: `STRING`, `LONG`, `FLOAT`, `DOUBLE`, `EMBEDDING`, `STRING_LIST`, `DICTIONARY`.

**category** `DimensionCategory`

The category of the dimension, e.g. 'tag' or 'featureLabel', etc. One of: `featureLabel`, `prediction`, `actuals`, `actualScore`, `actualClass`, `predictionClass`, `predictionScore`, `tag`, `spanProperty`, `llmEval`, `annotation`, `userAnnotation`, `modelVersion`, `batchId`.

**dimensionValues** `[DimensionValueInput!]`

The dimension values of the filter.

**DimensionValueInput fields**

**id** `ID!` *required*

**value** `String!` *required*

The value of this dimension value, e.g. 'Wells Fargo' or '60'

**similarityReference** `DimensionValueSimilarityReference`

The similarity references for this dimension value Nested `DimensionValueSimilarityReference` (same shape as above).

**queryFilter** `String`

Filters to apply to the dataset. Compatible with existing filters, joined with AND.

**multiSpanQuery** `MultiSpanQueryDefinitionInput`

Multi-span query definition for admission. Mutually exclusive with queryFilter.

**MultiSpanQueryDefinitionInput fields**

**subqueries** `[MultiSpanSubqueryDefinitionInput!]!` *required*

**MultiSpanSubqueryDefinitionInput fields**

**name** `String!` *required*

Subquery name (A-E)

**query** `String!` *required*

Span-selection filter for this subquery

**expression** `String!` *required*

Boolean expression over the subquery names

**samplingRate** `Float!` *required*

The sampling rate of the online task

**runContinuously** `Boolean!` *required*

Whether the online task runs continuously

**llmConfig** `OnlineTaskLLMConfigInput`

The LLM config for the online task

**OnlineTaskLLMConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**temperature** `Float`

The temperature of the LLM config

**provider** `LLMIntegrationProvider`

The LLM provider One of: `openAI`, `anthropic`, `awsBedrock`, `azureOpenAI`, `vertexAI`, `custom`, `nvidiaNim`, `gemini`, `litellm`, `fireworks`, `togetherAi`, `cursor`, `typeSafeAi`.

**deploymentName** `String`

The deployment name of the LLM config. Required for azure openai provider

**apiVersion** `String`

The api version of the LLM config. Required for azure openai provider

**apiEndpoint** `String`

The api endpoint of the LLM config. Required for azure openai provider

**region** `String`

The region of the LLM config

**maxTokens** `Float`

The max tokens of the LLM config

**maxCompletionTokens** `Float`

The max completion tokens of the LLM config

**topP** `Float`

The top P of the LLM config

**topK** `Float`

The top K of the LLM config

**stop** `[String!]`

The stop associated with the config

**customModelEndpointId** `String`

The id of the Custom Model Endpoint. Required if provider is custom.

**templateEvaluators** `[TemplateEvaluationConfigInput!]`

**TemplateEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the template evaluation config

**rails** `[String!]`

The rails associated with the config

**template** `String!` *required*

The template associated with the config

**turnDefinition** `String`

Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same \{variable} syntax as the main template.

**position** `Int!` *required*

The position of the config

**includeExplanations** `Boolean!` *required*

Whether the config includes explanations

**useFunctionCallingIfAvailable** `Boolean!` *required*

Whether the config uses function calling if available

**useStructuredOutputIfAvailable** `Boolean`

Whether the config uses structured output if available

**classificationChoices** `JSONObject`

The classification choices for the config (Record\<string, number>)

**direction** `TemplateEvaluationConfigDirection`

The direction for the config One of: `maximize`, `minimize`, `none`.

**dataGranularityType** `EvalDataGranularityType`

Data granularity (span, trace, or session). Defaults to span when omitted. One of: `span`, `trace`, `session`.

**llmConfig** `TemplateEvaluationLlmConfigInput`

Llm Config for this evaluator

**TemplateEvaluationLlmConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**codeEvaluators** `[CodeEvaluationConfigInput!]`

**CodeEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the evaluation config

**evaluationClass** `String`

The evaluation type associated with the config

**evalClassCodeBlock** `String`

The code block for the evaluation class

**evaluationInputParams** `JSON`

The evaluation input params

**packageImports** `String`

The package imports for the evaluation class

**position** `Int!` *required*

The position of the config

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**spanAttributes** `[String!]`

The data column associated with the config

**dataGranularityType** `EvalDataGranularityType`

The data granularity associated with the config (span, trace, or session) One of: `span`, `trace`, `session`.

**experiments** `[ID!]`

**postProcessingSteps** `[OnlineTaskPostProcessingStepInput!]`

**OnlineTaskPostProcessingStepInput fields**

**targetType** `String!` *required*

The type of the target (e.g. dataset, annotation queue)

**targetId** `ID!` *required*

The ID of the specific target (e.g. dataset ID)

**position** `Int!` *required*

Ordering position of the post-processing step

**queryFilter** `String`

Optional filter expression for selecting which data is added to target

**evaluators** `[OnlineTaskEvaluatorInput!]`

**OnlineTaskEvaluatorInput fields**

**evaluatorId** `ID!` *required*

id of evaluator to attach to online task

**evaluatorVersionId** `ID`

Optional id of the evaluator version to pin. Omit (or null) to always run the latest version.

**position** `Int!` *required*

position of evaluator in ui

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**columnMappings** `JSON`

Mapping of evaluator variable names to dataset column names

**subqueryColumnMappings** `[SubqueryColumnMappingInput!]`

Subquery-aware variable mappings. Populated instead of columnMappings on tasks that have a multiSpanQuery.

**SubqueryColumnMappingInput fields**

**variableName** `String!` *required*

Evaluator variable name

**subqueries** `[String!]!` *required*

Source subquery names (A-E) from the task's multi\_span\_query that may populate this variable. When multiple listed subqueries match for a unit, their resolved values are concatenated. The list must contain at least one subquery; an empty list matches nothing and is rejected.

**attributePath** `String!` *required*

Span attribute path to resolve within each subquery

**tracingEnabled** `Boolean`

Whether tracing is enabled for the online task

**arizeIntegrationId** `ID`

Arize sandbox integration ID (BYO path for harness tasks). When provided for a harness task the integration is linked to this task so the sandbox pod receives the Arize API key. Omit to use auto-provisioning (phase 2).

**clientMutationId** `String`

#### Returns

**evalTask** `OnlineTask!`

**OnlineTask fields**

Scalar fields: `id`, `taskType`, `name`, `samplingRate`, `lastRunAt`, `queryFilter`, `createdAt`, `updatedAt`, `deletedAt`, `runContinuously`, `tracingEnabled`, `experimentIds`, `hasUnavailableEvaluatorIntegration`.

Object fields (select subfields): `model`, `multiSpanQuery`, `createdBy`, `llmConfig`, `taskConfig`, `evaluators`, `onlineTaskEvaluators`, `allEvaluatorConfigs`, `taskRuns`, `experiments`, `dataset`, `postProcessingSteps`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation CreateEvalTask($input: CreateEvalTaskMutationInput!) {
  createEvalTask(input: $input) {
    evalTask { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "name": "<string>",
    "samplingRate": 1,
    "runContinuously": true
  }
}
```

### patchEvalTask

`patchEvalTask(input: PatchEvalTaskMutationInput!): PatchEvalTaskMutationPayload`

#### Arguments

**input** `PatchEvalTaskMutationInput!` *required*

**PatchEvalTaskMutationInput fields**

**onlineTaskId** `ID!` *required*

The online task id for the eval task to be patched

**modelId** `ID`

The model id for the model to be evaluated

**datasetId** `ID`

The dataset id for the dataset to be evaluated

**name** `String`

The name of the online task

**filters** `[FilterItemInputType!]`

The filters associated with the online task

**FilterItemInputType fields**

**filterType** `FilterRowType!` *required*

The type of filter. One of: `featureLabel`, `tagLabel`, `predictionValue`, `actuals`, `modelVersion`, `batchId`, `predictionClass`, `predictionScore`, `actualClass`, `actualScore`, `topKPercentile`, `spanProperty`, `llmEval`, `annotation`, `userAnnotation`.

**operator** `ComparisonOperator!` *required*

The operator of the filter. One of: `greaterThan`, `lessThan`, `equals`, `notEquals`, `greaterThanOrEqual`, `lessThanOrEqual`, `topN`, `contains`, `containsString`, `similarTo`.

**dimension** `DimensionInput`

**DimensionInput fields**

**id** `ID!` *required*

**name** `String!` *required*

The label of this dimension, e.g. 'Bank Name' or 'age'

**dataType** `DimensionDataType!` *required*

The data type of the values in dimensionName, ex. 'FLOAT', 'STRING', etc. One of: `STRING`, `LONG`, `FLOAT`, `DOUBLE`, `EMBEDDING`, `STRING_LIST`, `DICTIONARY`.

**category** `DimensionCategory`

The category of the dimension, e.g. 'tag' or 'featureLabel', etc. One of: `featureLabel`, `prediction`, `actuals`, `actualScore`, `actualClass`, `predictionClass`, `predictionScore`, `tag`, `spanProperty`, `llmEval`, `annotation`, `userAnnotation`, `modelVersion`, `batchId`.

**dimensionValues** `[DimensionValueInput!]`

The dimension values of the filter.

**DimensionValueInput fields**

**id** `ID!` *required*

**value** `String!` *required*

The value of this dimension value, e.g. 'Wells Fargo' or '60'

**similarityReference** `DimensionValueSimilarityReference`

The similarity references for this dimension value Nested `DimensionValueSimilarityReference` (same shape as above).

**queryFilter** `String`

Filters to apply to the dataset. Compatible with existing filters, joined with AND.

**multiSpanQuery** `MultiSpanQueryDefinitionInput`

Multi-span query definition for admission. Mutually exclusive with queryFilter.

**MultiSpanQueryDefinitionInput fields**

**subqueries** `[MultiSpanSubqueryDefinitionInput!]!` *required*

**MultiSpanSubqueryDefinitionInput fields**

**name** `String!` *required*

Subquery name (A-E)

**query** `String!` *required*

Span-selection filter for this subquery

**expression** `String!` *required*

Boolean expression over the subquery names

**samplingRate** `Float`

The sampling rate of the online task

**runContinuously** `Boolean`

Whether the online task runs continuously

**llmConfig** `OnlineTaskLLMConfigInput`

The LLM config for the online task

**OnlineTaskLLMConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**temperature** `Float`

The temperature of the LLM config

**provider** `LLMIntegrationProvider`

The LLM provider One of: `openAI`, `anthropic`, `awsBedrock`, `azureOpenAI`, `vertexAI`, `custom`, `nvidiaNim`, `gemini`, `litellm`, `fireworks`, `togetherAi`, `cursor`, `typeSafeAi`.

**deploymentName** `String`

The deployment name of the LLM config. Required for azure openai provider

**apiVersion** `String`

The api version of the LLM config. Required for azure openai provider

**apiEndpoint** `String`

The api endpoint of the LLM config. Required for azure openai provider

**region** `String`

The region of the LLM config

**maxTokens** `Float`

The max tokens of the LLM config

**maxCompletionTokens** `Float`

The max completion tokens of the LLM config

**topP** `Float`

The top P of the LLM config

**topK** `Float`

The top K of the LLM config

**stop** `[String!]`

The stop associated with the config

**customModelEndpointId** `String`

The id of the Custom Model Endpoint. Required if provider is custom.

**templateEvaluators** `[TemplateEvaluationConfigInput!]`

**TemplateEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the template evaluation config

**rails** `[String!]`

The rails associated with the config

**template** `String!` *required*

The template associated with the config

**turnDefinition** `String`

Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same \{variable} syntax as the main template.

**position** `Int!` *required*

The position of the config

**includeExplanations** `Boolean!` *required*

Whether the config includes explanations

**useFunctionCallingIfAvailable** `Boolean!` *required*

Whether the config uses function calling if available

**useStructuredOutputIfAvailable** `Boolean`

Whether the config uses structured output if available

**classificationChoices** `JSONObject`

The classification choices for the config (Record\<string, number>)

**direction** `TemplateEvaluationConfigDirection`

The direction for the config One of: `maximize`, `minimize`, `none`.

**dataGranularityType** `EvalDataGranularityType`

Data granularity (span, trace, or session). Defaults to span when omitted. One of: `span`, `trace`, `session`.

**llmConfig** `TemplateEvaluationLlmConfigInput`

Llm Config for this evaluator

**TemplateEvaluationLlmConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**codeEvaluators** `[CodeEvaluationConfigInput!]`

**CodeEvaluationConfigInput fields**

**name** `EvalColumnName!` *required*

The name of the evaluation config

**evaluationClass** `String`

The evaluation type associated with the config

**evalClassCodeBlock** `String`

The code block for the evaluation class

**evaluationInputParams** `JSON`

The evaluation input params

**packageImports** `String`

The package imports for the evaluation class

**position** `Int!` *required*

The position of the config

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**spanAttributes** `[String!]`

The data column associated with the config

**dataGranularityType** `EvalDataGranularityType`

The data granularity associated with the config (span, trace, or session) One of: `span`, `trace`, `session`.

**experiments** `[ID!]`

**postProcessingSteps** `[OnlineTaskPostProcessingStepInput!]`

**OnlineTaskPostProcessingStepInput fields**

**targetType** `String!` *required*

The type of the target (e.g. dataset, annotation queue)

**targetId** `ID!` *required*

The ID of the specific target (e.g. dataset ID)

**position** `Int!` *required*

Ordering position of the post-processing step

**queryFilter** `String`

Optional filter expression for selecting which data is added to target

**evaluators** `[OnlineTaskEvaluatorInput!]`

**OnlineTaskEvaluatorInput fields**

**evaluatorId** `ID!` *required*

id of evaluator to attach to online task

**evaluatorVersionId** `ID`

Optional id of the evaluator version to pin. Omit (or null) to always run the latest version.

**position** `Int!` *required*

position of evaluator in ui

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**columnMappings** `JSON`

Mapping of evaluator variable names to dataset column names

**subqueryColumnMappings** `[SubqueryColumnMappingInput!]`

Subquery-aware variable mappings. Populated instead of columnMappings on tasks that have a multiSpanQuery.

**SubqueryColumnMappingInput fields**

**variableName** `String!` *required*

Evaluator variable name

**subqueries** `[String!]!` *required*

Source subquery names (A-E) from the task's multi\_span\_query that may populate this variable. When multiple listed subqueries match for a unit, their resolved values are concatenated. The list must contain at least one subquery; an empty list matches nothing and is rejected.

**attributePath** `String!` *required*

Span attribute path to resolve within each subquery

**tracingEnabled** `Boolean`

Whether tracing is enabled for the online task

**clientMutationId** `String`

#### Returns

**evalTask** `OnlineTask!`

**OnlineTask fields**

Scalar fields: `id`, `taskType`, `name`, `samplingRate`, `lastRunAt`, `queryFilter`, `createdAt`, `updatedAt`, `deletedAt`, `runContinuously`, `tracingEnabled`, `experimentIds`, `hasUnavailableEvaluatorIntegration`.

Object fields (select subfields): `model`, `multiSpanQuery`, `createdBy`, `llmConfig`, `taskConfig`, `evaluators`, `onlineTaskEvaluators`, `allEvaluatorConfigs`, `taskRuns`, `experiments`, `dataset`, `postProcessingSteps`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation PatchEvalTask($input: PatchEvalTaskMutationInput!) {
  patchEvalTask(input: $input) {
    evalTask { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "onlineTaskId": "<ID>"
  }
}
```

### patchEvalTaskWithEvalHub

`patchEvalTaskWithEvalHub(input: PatchEvalTaskWithEvalHubMutationInput!): PatchEvalTaskWithEvalHubMutationPayload`

#### Arguments

**input** `PatchEvalTaskWithEvalHubMutationInput!` *required*

**PatchEvalTaskWithEvalHubMutationInput fields**

**onlineTaskId** `ID!` *required*

The online task id for the eval task to be patched

**modelId** `ID`

The model id for the model to be evaluated

**datasetId** `ID`

The dataset id for the dataset to be evaluated

**name** `String`

The name of the online task

**filters** `[FilterItemInputType!]`

The filters associated with the online task

**FilterItemInputType fields**

**filterType** `FilterRowType!` *required*

The type of filter. One of: `featureLabel`, `tagLabel`, `predictionValue`, `actuals`, `modelVersion`, `batchId`, `predictionClass`, `predictionScore`, `actualClass`, `actualScore`, `topKPercentile`, `spanProperty`, `llmEval`, `annotation`, `userAnnotation`.

**operator** `ComparisonOperator!` *required*

The operator of the filter. One of: `greaterThan`, `lessThan`, `equals`, `notEquals`, `greaterThanOrEqual`, `lessThanOrEqual`, `topN`, `contains`, `containsString`, `similarTo`.

**dimension** `DimensionInput`

**DimensionInput fields**

**id** `ID!` *required*

**name** `String!` *required*

The label of this dimension, e.g. 'Bank Name' or 'age'

**dataType** `DimensionDataType!` *required*

The data type of the values in dimensionName, ex. 'FLOAT', 'STRING', etc. One of: `STRING`, `LONG`, `FLOAT`, `DOUBLE`, `EMBEDDING`, `STRING_LIST`, `DICTIONARY`.

**category** `DimensionCategory`

The category of the dimension, e.g. 'tag' or 'featureLabel', etc. One of: `featureLabel`, `prediction`, `actuals`, `actualScore`, `actualClass`, `predictionClass`, `predictionScore`, `tag`, `spanProperty`, `llmEval`, `annotation`, `userAnnotation`, `modelVersion`, `batchId`.

**dimensionValues** `[DimensionValueInput!]`

The dimension values of the filter.

**DimensionValueInput fields**

**id** `ID!` *required*

**value** `String!` *required*

The value of this dimension value, e.g. 'Wells Fargo' or '60'

**similarityReference** `DimensionValueSimilarityReference`

The similarity references for this dimension value Nested `DimensionValueSimilarityReference` (same shape as above).

**queryFilter** `String`

Filters to apply to the dataset. Compatible with existing filters, joined with AND.

**multiSpanQuery** `MultiSpanQueryDefinitionInput`

Multi-span query definition for admission. Mutually exclusive with queryFilter.

**MultiSpanQueryDefinitionInput fields**

**subqueries** `[MultiSpanSubqueryDefinitionInput!]!` *required*

**MultiSpanSubqueryDefinitionInput fields**

**name** `String!` *required*

Subquery name (A-E)

**query** `String!` *required*

Span-selection filter for this subquery

**expression** `String!` *required*

Boolean expression over the subquery names

**samplingRate** `Float`

The sampling rate of the online task

**runContinuously** `Boolean`

Whether the online task runs continuously

**llmConfig** `OnlineTaskLLMConfigInput`

The LLM config for the online task

**OnlineTaskLLMConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**temperature** `Float`

The temperature of the LLM config

**provider** `LLMIntegrationProvider`

The LLM provider One of: `openAI`, `anthropic`, `awsBedrock`, `azureOpenAI`, `vertexAI`, `custom`, `nvidiaNim`, `gemini`, `litellm`, `fireworks`, `togetherAi`, `cursor`, `typeSafeAi`.

**deploymentName** `String`

The deployment name of the LLM config. Required for azure openai provider

**apiVersion** `String`

The api version of the LLM config. Required for azure openai provider

**apiEndpoint** `String`

The api endpoint of the LLM config. Required for azure openai provider

**region** `String`

The region of the LLM config

**maxTokens** `Float`

The max tokens of the LLM config

**maxCompletionTokens** `Float`

The max completion tokens of the LLM config

**topP** `Float`

The top P of the LLM config

**topK** `Float`

The top K of the LLM config

**stop** `[String!]`

The stop associated with the config

**customModelEndpointId** `String`

The id of the Custom Model Endpoint. Required if provider is custom.

**evaluators** `[OnlineTaskEvaluatorInput!]`

**OnlineTaskEvaluatorInput fields**

**evaluatorId** `ID!` *required*

id of evaluator to attach to online task

**evaluatorVersionId** `ID`

Optional id of the evaluator version to pin. Omit (or null) to always run the latest version.

**position** `Int!` *required*

position of evaluator in ui

**queryFilter** `String`

Optional query used to filter over a given data granularity (ex. session, trace)

**columnMappings** `JSON`

Mapping of evaluator variable names to dataset column names

**subqueryColumnMappings** `[SubqueryColumnMappingInput!]`

Subquery-aware variable mappings. Populated instead of columnMappings on tasks that have a multiSpanQuery.

**SubqueryColumnMappingInput fields**

**variableName** `String!` *required*

Evaluator variable name

**subqueries** `[String!]!` *required*

Source subquery names (A-E) from the task's multi\_span\_query that may populate this variable. When multiple listed subqueries match for a unit, their resolved values are concatenated. The list must contain at least one subquery; an empty list matches nothing and is rejected.

**attributePath** `String!` *required*

Span attribute path to resolve within each subquery

**experiments** `[ID!]`

**postProcessingSteps** `[OnlineTaskPostProcessingStepInput!]`

**OnlineTaskPostProcessingStepInput fields**

**targetType** `String!` *required*

The type of the target (e.g. dataset, annotation queue)

**targetId** `ID!` *required*

The ID of the specific target (e.g. dataset ID)

**position** `Int!` *required*

Ordering position of the post-processing step

**queryFilter** `String`

Optional filter expression for selecting which data is added to target

**clientMutationId** `String`

#### Returns

**evalTask** `OnlineTask!`

**OnlineTask fields**

Scalar fields: `id`, `taskType`, `name`, `samplingRate`, `lastRunAt`, `queryFilter`, `createdAt`, `updatedAt`, `deletedAt`, `runContinuously`, `tracingEnabled`, `experimentIds`, `hasUnavailableEvaluatorIntegration`.

Object fields (select subfields): `model`, `multiSpanQuery`, `createdBy`, `llmConfig`, `taskConfig`, `evaluators`, `onlineTaskEvaluators`, `allEvaluatorConfigs`, `taskRuns`, `experiments`, `dataset`, `postProcessingSteps`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation PatchEvalTaskWithEvalHub($input: PatchEvalTaskWithEvalHubMutationInput!) {
  patchEvalTaskWithEvalHub(input: $input) {
    evalTask { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "onlineTaskId": "<ID>"
  }
}
```

### migrateTaskToEvalHub

migrates a task's evaluators to eval hub

`migrateTaskToEvalHub(input: MigrateTaskToEvalHubMutationInput!): MigrateTaskToEvalHubMutationPayload`

#### Arguments

**input** `MigrateTaskToEvalHubMutationInput!` *required*

**MigrateTaskToEvalHubMutationInput fields**

**onlineTaskId** `ID!` *required*

The online task id

**clientMutationId** `String`

#### Returns

**onlineTask** `OnlineTask!`

**OnlineTask fields**

Scalar fields: `id`, `taskType`, `name`, `samplingRate`, `lastRunAt`, `queryFilter`, `createdAt`, `updatedAt`, `deletedAt`, `runContinuously`, `tracingEnabled`, `experimentIds`, `hasUnavailableEvaluatorIntegration`.

Object fields (select subfields): `model`, `multiSpanQuery`, `createdBy`, `llmConfig`, `taskConfig`, `evaluators`, `onlineTaskEvaluators`, `allEvaluatorConfigs`, `taskRuns`, `experiments`, `dataset`, `postProcessingSteps`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation MigrateTaskToEvalHub($input: MigrateTaskToEvalHubMutationInput!) {
  migrateTaskToEvalHub(input: $input) {
    onlineTask { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "onlineTaskId": "<ID>"
  }
}
```

### runOnlineTask

Run an online task

`runOnlineTask(input: RunOnlineTaskMutationInput!): RunOnlineTaskMutationPayload`

#### Arguments

**input** `RunOnlineTaskMutationInput!` *required*

**RunOnlineTaskMutationInput fields**

**onlineTaskId** `ID`

The task id for the task to be processed

**dataStartTime** `DateTime`

Start time of the data processed by the online task

**dataEndTime** `DateTime`

Start time of the data processed by the online task

**maxSpans** `Int`

Maximum number of spans processed by the online task. If not specified, default to 10,000.

**overrideEvaluations** `Boolean`

Whether to override the existing evaluation labels for the task.

**isErrorRetry** `Boolean`

When true, the worker reproduces the source run's capped row set by omitting eval-null scan filters. Used by Rerun on Errors.

**experimentIds** `[ID!]`

The experiment ids to run the task on

**clientMutationId** `String`

#### Returns

**result** `RunOnlineTaskResponse!`

Result for queuing online task run on date range Union of `TaskError`, `CreateTaskRunResponse`. Use inline fragments (`... on TypeName`).

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation RunOnlineTask($input: RunOnlineTaskMutationInput!) {
  runOnlineTask(input: $input) {
    result { __typename }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {}
}
```

### cancelOnlineTaskRun

Cancel an online task run

`cancelOnlineTaskRun(input: CancelOnlineTaskRunMutationInput!): CancelOnlineTaskRunMutationPayload`

#### Arguments

**input** `CancelOnlineTaskRunMutationInput!` *required*

**CancelOnlineTaskRunMutationInput fields**

**onlineTaskRunId** `ID!` *required*

The run id for the task to be processed

**clientMutationId** `String`

#### Returns

**result** `CancelOnlineTaskResponse!`

Result for cancelling online task run Union of `TaskError`, `CancelTaskRunResponse`. Use inline fragments (`... on TypeName`).

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation CancelOnlineTaskRun($input: CancelOnlineTaskRunMutationInput!) {
  cancelOnlineTaskRun(input: $input) {
    result { __typename }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "onlineTaskRunId": "<ID>"
  }
}
```

### deleteOnlineTask

`deleteOnlineTask(input: DeleteOnlineTaskMutationInput!): DeleteOnlineTaskMutationPayload`

#### Arguments

**input** `DeleteOnlineTaskMutationInput!` *required*

**DeleteOnlineTaskMutationInput fields**

**onlineTaskIds** `[ID!]!` *required*

The IDs of the online tasks to delete. Maximum 50 per request; callers must implement their own batching for larger sets.

**clientMutationId** `String`

#### Returns

**success** `Boolean!`

True when all requested tasks were deleted (failedTasks is empty). Kept for backward compatibility.

**deletedTaskIds** `[ID!]!`

IDs of the tasks that were successfully deleted.

**failedTasks** `[DeleteOnlineTaskFailure!]!`

Tasks that could not be deleted, each with the reason for failure. The caller should inspect and retry or investigate individually.

**DeleteOnlineTaskFailure fields**

Scalar fields: `taskId`, `message`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation DeleteOnlineTask($input: DeleteOnlineTaskMutationInput!) {
  deleteOnlineTask(input: $input) {
    success
    deletedTaskIds
    failedTasks { message }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "onlineTaskIds": [
      "<ID>"
    ]
  }
}
```

### runExperimentTask

Run an experiment task with configurable LLM interactions

`runExperimentTask(input: RunExperimentTaskMutationInput!): RunExperimentTaskMutationPayload`

#### Arguments

**input** `RunExperimentTaskMutationInput!` *required*

**RunExperimentTaskMutationInput fields**

**datasetId** `ID!` *required*

The dataset id for the experiment

**datasetName** `String!` *required*

The dataset name for the experiment

**datasetVersionId** `ID!` *required*

The dataset version id for the experiment

**spaceId** `ID!` *required*

The space id for the experiment

**exampleOverrides** `[ExampleOverridesInput!]!` *required*

The example overrides for the experiment

**ExampleOverridesInput fields**

**exampleId** `String!` *required*

**overridesJSON** `JSON!` *required*

**exampleIds** `[String!]`

The example ids to run the experiment on

**examplesToRunCount** `Int`

The number of examples to run the experiment on

**queryFilter** `String`

Optional dataset query filter. When set, the experiment (and associated evaluations) run only on examples matching the filter.

**runConfigurations** `[RunExperimentConfigInput!]!` *required*

The run configurations for the experiment

**RunExperimentConfigInput fields**

**experimentType** `ExperimentTypeEnum!` *required*

Discriminant for what kind of work is performed per example. One of: 'llm\_generation', 'template\_evaluation', 'code\_evaluation', 'agent\_call'. One of: `llm_generation`, `template_evaluation`, `code_evaluation`, `agent_call`.

**input** `LLMInput`

LLM input (messages, model, integration). Required for llm\_generation; provider/model fields also used for template\_evaluation. Not used for agent\_call.

**LLMInput fields**

**provider** `ExternalLLMProvider!` *required*

The provider to use for the call. One of: `openAI`, `azureOpenAI`, `anthropic`, `vertexAI`, `awsBedrock`, `custom`, `nvidiaNim`, `gemini`, `litellm`, `fireworks`, `togetherAi`.

**model** `String`

The model to use for the call.

**messages** `[LLMMessageInput!]!` *required*

The full message to send to the LLM (e.g., a prompt template with all variables filled in). Nested `LLMMessageInput` (same shape as above).

**invocationParams** `InvocationParamsInput!` *required*

The params to call the LLM with. Nested `InvocationParamsInput` (same shape as above).

**providerParams** `ProviderParamsInput!` *required*

The provider params to use for the call. Nested `ProviderParamsInput` (same shape as above).

**integrationId** `ID`

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**id** `String`

Unique identifier for this input

**inputVariableFormat** `PromptVersionInputVariableFormatEnum`

The input variable format for determining prompt variables in the messages. Only applicable for llm\_generation. One of: `F_STRING`, `MUSTACHE`, `NONE`.

**experimentName** `String!` *required*

Name for the experiment

**instanceId** `Int!` *required*

The instance id for the experiment

**promptVersionId** `ID`

The version ID of the prompt from Prompt Hub (if loaded from hub)

**templateConfig** `TemplateEvalConfigInput`

Required when experimentType = 'template\_evaluation'.

**TemplateEvalConfigInput fields**

**template** `String!` *required*

The evaluation prompt template.

**provideExplanation** `Boolean!` *required*

Whether to include LLM explanations in output.

**classificationChoices** `JSON`

Map of choice label to numeric score (e.g. \{relevant: 1, irrelevant: 0}). Required when templateConfig is provided.

**columnMapping** `JSON`

Maps named eval param keys to dataset column paths.

**evaluatorVersionId** `ID`

The evaluator version ID from Eval Hub (if the template eval was loaded from hub)

**codeEvalConfig** `CodeEvalConfigInput`

Required when experimentType = 'code\_evaluation'.

**CodeEvalConfigInput fields**

**codeEvalClass** `String!` *required*

Managed evaluator class (e.g. JSONParsable, MatchesRegex) or 'Custom'.

**evalParamsJson** `String`

JSON-encoded params for managed evaluator classes.

**evaluationClass** `String`

Class name for managed evaluators.

**customCodeEvalInfo** `CustomCodeEvalInfoInput`

Only set when codeEvalClass is 'Custom'. Nested `CustomCodeEvalInfoInput` (same shape as above).

**columnMapping** `JSON!` *required*

Maps named eval param keys to dataset column paths.

**agentConfig** `AgentConfigInput`

Required when experimentType = 'agent\_call'.

**AgentConfigInput fields**

**remoteEndpointIntegrationId** `ID!` *required*

Global ID of the remote endpoint integration to invoke.

**inputTemplate** `JSON!` *required*

Fully-merged JSON request body. \{\{column}} mustache refs are substituted at execution time using each dataset row.

**requestTimeoutSeconds** `Int`

Per-call LLM timeout in seconds (whole number, 30-1800) for llm\_generation and template\_evaluation runs; also applies to any LLM evaluator judge calls attached to the run. Rejected for other experiment types. Null uses the system default.

**tracingMetadata** `JSON`

The tracing metadata for the experiment

**evaluationTaskIds** `[String!]`

List of evaluation task IDs to run after the experiment completes

**clientMutationId** `String`

#### Returns

**result** `RunExperimentTaskResponse!`

Result for running experiment task Union of `RunExperimentTaskSuccess`, `RunExperimentTaskError`. Use inline fragments (`... on TypeName`).

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation RunExperimentTask($input: RunExperimentTaskMutationInput!) {
  runExperimentTask(input: $input) {
    result { __typename }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "datasetId": "<ID>",
    "datasetName": "<string>",
    "datasetVersionId": "<ID>",
    "spaceId": "<ID>",
    "exampleOverrides": [
      {
        "exampleId": "<string>",
        "overridesJSON": {}
      }
    ],
    "runConfigurations": [
      {
        "experimentType": "llm_generation",
        "experimentName": "<string>",
        "instanceId": 1
      }
    ]
  }
}
```

### createPromptOptimizationTask

Create a new prompt optimization task

`createPromptOptimizationTask(input: CreatePromptOptimizationTaskMutationInput!): CreatePromptOptimizationTaskMutationPayload`

#### Arguments

**input** `CreatePromptOptimizationTaskMutationInput!` *required*

**CreatePromptOptimizationTaskMutationInput fields**

**datasetId** `ID!` *required*

The dataset id to use for prompt optimization

**experimentId** `ID`

Optional experiment ID to use as datasource. Only one experiment is supported for prompt optimization tasks. If not provided, the dataset will be used.

**name** `String!` *required*

The name of the prompt optimization task

**runContinuously** `Boolean!` *required*

Whether the online task runs continuously

**llmConfig** `OnlineTaskLLMConfigInput!` *required*

The LLM config for the online task

**OnlineTaskLLMConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**temperature** `Float`

The temperature of the LLM config

**provider** `LLMIntegrationProvider`

The LLM provider One of: `openAI`, `anthropic`, `awsBedrock`, `azureOpenAI`, `vertexAI`, `custom`, `nvidiaNim`, `gemini`, `litellm`, `fireworks`, `togetherAi`, `cursor`, `typeSafeAi`.

**deploymentName** `String`

The deployment name of the LLM config. Required for azure openai provider

**apiVersion** `String`

The api version of the LLM config. Required for azure openai provider

**apiEndpoint** `String`

The api endpoint of the LLM config. Required for azure openai provider

**region** `String`

The region of the LLM config

**maxTokens** `Float`

The max tokens of the LLM config

**maxCompletionTokens** `Float`

The max completion tokens of the LLM config

**topP** `Float`

The top P of the LLM config

**topK** `Float`

The top K of the LLM config

**stop** `[String!]`

The stop associated with the config

**customModelEndpointId** `String`

The id of the Custom Model Endpoint. Required if provider is custom.

**promptOptimizationConfig** `PromptOptimizationConfigInput!` *required*

The prompt optimization config

**PromptOptimizationConfigInput fields**

**userInstructions** `String!` *required*

The user instructions for the config

**outputColumn** `String!` *required*

The output column for the config

**feedbackColumns** `[String!]!` *required*

The feedback columns for the config

**batchSize** `Int!` *required*

The batch size for the config

**promptId** `ID!` *required*

The prompt id for the config

**tracingEnabled** `Boolean`

Whether tracing is enabled for the online task

**clientMutationId** `String`

#### Returns

**promptOptimizationTask** `OnlineTask!`

**OnlineTask fields**

Scalar fields: `id`, `taskType`, `name`, `samplingRate`, `lastRunAt`, `queryFilter`, `createdAt`, `updatedAt`, `deletedAt`, `runContinuously`, `tracingEnabled`, `experimentIds`, `hasUnavailableEvaluatorIntegration`.

Object fields (select subfields): `model`, `multiSpanQuery`, `createdBy`, `llmConfig`, `taskConfig`, `evaluators`, `onlineTaskEvaluators`, `allEvaluatorConfigs`, `taskRuns`, `experiments`, `dataset`, `postProcessingSteps`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation CreatePromptOptimizationTask($input: CreatePromptOptimizationTaskMutationInput!) {
  createPromptOptimizationTask(input: $input) {
    promptOptimizationTask { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "datasetId": "<ID>",
    "name": "<string>",
    "runContinuously": true,
    "llmConfig": {
      "integrationId": "<ID>",
      "modelName": "<string>",
      "invocationParameters": "<JSONObject>",
      "providerParameters": "<JSONObject>"
    },
    "promptOptimizationConfig": {
      "userInstructions": "<string>",
      "outputColumn": "<string>",
      "feedbackColumns": [
        "<string>"
      ],
      "batchSize": 1,
      "promptId": "<ID>"
    }
  }
}
```

### patchPromptOptimizationTask

`patchPromptOptimizationTask(input: PatchPromptOptimizationTaskMutationInput!): PatchPromptOptimizationTaskMutationPayload`

#### Arguments

**input** `PatchPromptOptimizationTaskMutationInput!` *required*

**PatchPromptOptimizationTaskMutationInput fields**

**onlineTaskId** `ID!` *required*

The online task id for the eval task to be patched

**datasetId** `ID`

The dataset id for the dataset to be evaluated

**name** `String`

The name of the online task

**runContinuously** `Boolean`

Whether the online task runs continuously

**llmConfig** `OnlineTaskLLMConfigInput`

The LLM config for the online task

**OnlineTaskLLMConfigInput fields**

**integrationId** `ID!` *required*

The selected named LLM integration id (relay global id). If provided, server will use its configuration.

**modelName** `String!` *required*

The LLM model

**invocationParameters** `JSONObject!` *required*

parameters used when running the Llm

**providerParameters** `JSONObject!` *required*

parameters used to initialize the Llm

**temperature** `Float`

The temperature of the LLM config

**provider** `LLMIntegrationProvider`

The LLM provider One of: `openAI`, `anthropic`, `awsBedrock`, `azureOpenAI`, `vertexAI`, `custom`, `nvidiaNim`, `gemini`, `litellm`, `fireworks`, `togetherAi`, `cursor`, `typeSafeAi`.

**deploymentName** `String`

The deployment name of the LLM config. Required for azure openai provider

**apiVersion** `String`

The api version of the LLM config. Required for azure openai provider

**apiEndpoint** `String`

The api endpoint of the LLM config. Required for azure openai provider

**region** `String`

The region of the LLM config

**maxTokens** `Float`

The max tokens of the LLM config

**maxCompletionTokens** `Float`

The max completion tokens of the LLM config

**topP** `Float`

The top P of the LLM config

**topK** `Float`

The top K of the LLM config

**stop** `[String!]`

The stop associated with the config

**customModelEndpointId** `String`

The id of the Custom Model Endpoint. Required if provider is custom.

**promptOptimizationConfig** `PromptOptimizationConfigInput`

The prompt optimization config

**PromptOptimizationConfigInput fields**

**userInstructions** `String!` *required*

The user instructions for the config

**outputColumn** `String!` *required*

The output column for the config

**feedbackColumns** `[String!]!` *required*

The feedback columns for the config

**batchSize** `Int!` *required*

The batch size for the config

**promptId** `ID!` *required*

The prompt id for the config

**tracingEnabled** `Boolean`

Whether tracing is enabled for the online task

**clientMutationId** `String`

#### Returns

**promptOptimizationTask** `OnlineTask!`

**OnlineTask fields**

Scalar fields: `id`, `taskType`, `name`, `samplingRate`, `lastRunAt`, `queryFilter`, `createdAt`, `updatedAt`, `deletedAt`, `runContinuously`, `tracingEnabled`, `experimentIds`, `hasUnavailableEvaluatorIntegration`.

Object fields (select subfields): `model`, `multiSpanQuery`, `createdBy`, `llmConfig`, `taskConfig`, `evaluators`, `onlineTaskEvaluators`, `allEvaluatorConfigs`, `taskRuns`, `experiments`, `dataset`, `postProcessingSteps`.

#### Example

Only required input fields are shown. Replace `<ID>` and `<string>` placeholders with real values; the [object graph](/docs/ax/graphql-reference/queries/object-graph.md) page shows how to look IDs up.

```graphql Mutation theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
mutation PatchPromptOptimizationTask($input: PatchPromptOptimizationTaskMutationInput!) {
  patchPromptOptimizationTask(input: $input) {
    promptOptimizationTask { id name }
  }
}
```

```json Variables theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
{
  "input": {
    "onlineTaskId": "<ID>"
  }
}
```