POST to https://app.arize.com/graphql with an x-api-key header; see Forming calls.
Mutations in this page
createEvaluator: Create a new evaluatorcreateEvaluatorVersion: Create a new evaluator versioneditEvaluator: Edit an existing evaluatordeleteEvaluator: Delete an existing evaluatorcreateEvalTask: Create a new online eval taskpatchEvalTask: Patch eval task.patchEvalTaskWithEvalHub: Patch eval task with eval hub.migrateTaskToEvalHub: migrates a taskās evaluators to eval hubrunOnlineTask: Run an online taskcancelOnlineTaskRun: Cancel an online task rundeleteOnlineTask: Delete online task.runExperimentTask: Run an experiment task with configurable LLM interactionscreatePromptOptimizationTask: Create a new prompt optimization taskpatchPromptOptimizationTask: Patch prompt optimization task.
Reference
createEvaluator
Create a new evaluatorcreateEvaluator(input: CreateEvaluatorMutationInput!): CreateEvaluatorMutationPayload
Arguments
CreateEvaluatorMutationInput!
required
Show CreateEvaluatorMutationInput fields
Show CreateEvaluatorMutationInput fields
ID!
required
The space id for the evaluator
String!
required
The name of the evaluator
String
The description of the saved evaluator
String!
required
The commit message describing the changes in this version
TemplateEvaluationConfigInput
Show TemplateEvaluationConfigInput fields
Show TemplateEvaluationConfigInput fields
EvalColumnName!
required
The name of the template evaluation config
[String!]
The rails associated with the config
String!
required
The template associated with the config
String
Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same {variable} syntax as the main template.
Int!
required
The position of the config
Boolean!
required
Whether the config includes explanations
Boolean!
required
Whether the config uses function calling if available
Boolean
Whether the config uses structured output if available
JSONObject
The classification choices for the config (Record<string, number>)
TemplateEvaluationConfigDirection
The direction for the config One of:
maximize, minimize, none.EvalDataGranularityType
Data granularity (span, trace, or session). Defaults to span when omitted. One of:
span, trace, session.TemplateEvaluationLlmConfigInput
Llm Config for this evaluator
Show TemplateEvaluationLlmConfigInput fields
Show TemplateEvaluationLlmConfigInput fields
CodeEvaluationConfigInput
Show CodeEvaluationConfigInput fields
Show CodeEvaluationConfigInput fields
EvalColumnName!
required
The name of the evaluation config
String
The evaluation type associated with the config
String
The code block for the evaluation class
JSON
The evaluation input params
String
The package imports for the evaluation class
Int!
required
The position of the config
String
Optional query used to filter over a given data granularity (ex. session, trace)
[String!]
The data column associated with the config
EvalDataGranularityType
The data granularity associated with the config (span, trace, or session) One of:
span, trace, session.HarnessEvaluatorInput
Show HarnessEvaluatorInput fields
Show HarnessEvaluatorInput fields
ID!
required
Relay GID of the llm_integrations row. Provider must be āanthropicā.
[ID!]
Optional relay GIDs of sandbox_integrations to link to the new sandbox config at creation time.
EvalColumnName!
required
Eval column prefix name. Harness output is written under eval.<name>.*.
String!
required
Natural-language eval template. {column_name} markers are substituted at run time.
JSON
Optional fixed label set. Null means the agent discovers labels freely.
Boolean
When true the agent must only emit labels from classificationChoices.
String
Optional model name hint forwarded to the pod as ANTHROPIC_MODEL.
HarnessEvaluationConfigDirection
Optimization direction for scored harness eval output. One of:
maximize, minimize, none.RemoteEvaluatorInput
Show RemoteEvaluatorInput fields
Show RemoteEvaluatorInput fields
String!
required
HTTP URL of the remote evaluation endpoint.
JSON
Optional custom request headers sent to the endpoint (encrypted at rest).
JSON
Optional JSON schema describing the expected request body.
EvalDataGranularityType
Data granularity (span, trace, or session). Defaults to span when omitted. Fixed at creation; later versions inherit it. One of:
span, trace, session.SystemOneEvaluatorInput
Show SystemOneEvaluatorInput fields
Show SystemOneEvaluatorInput fields
String!
required
Relay global ID of the TypeSafe AI integration this evaluator calls.
String!
required
Provider model route, for example
jev-latest. Required: the column has no default, so which route was asked for is always recorded.String!
required
The authorās JSON-shaped state template, stored verbatim, carrying {variables} resolved per record.
[SystemOneQuestionInput!]!
required
At least one. Every question sees the same state and is evaluated independently in one request.
Show SystemOneQuestionInput fields
Show SystemOneQuestionInput fields
String!
required
Becomes the providerās question key and the published eval column suffix, so it is validated against the eval-name rules.
SystemOneQuestionType!
required
One of:
boolean, choice, score.String!
required
The complete judgment. Question names are not sent to the provider as context.
JSON
Required for choice and score; optional for boolean. Shape is validated per type.
OptimizationDirection
Read for score questions only, where it defaults to maximize. Normalized to
none for every other type. One of: minimize, maximize, none.EvalDataGranularityType
Data granularity (span, trace, or session). Defaults to span when omitted. Fixed at creation; later versions inherit it. One of:
span, trace, session.String
Returns
Evaluator!
Show Evaluator fields
Show Evaluator fields
Scalar fields:
id, name, description, taskType, commitHash, commitMessage, createdAt, updatedAt.Object fields (select subfields): versionHistory, webhookSubscriptions, config, harnessEvaluationConfig, remoteEvaluationConfig, systemOneEvaluationConfig, createdBy, tags, onlineTasks.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation CreateEvaluator($input: CreateEvaluatorMutationInput!) {
createEvaluator(input: $input) {
evaluator { id name }
}
}
{
"input": {
"spaceId": "<ID>",
"name": "<string>",
"commitMessage": "<string>"
}
}
createEvaluatorVersion
Create a new evaluator versioncreateEvaluatorVersion(input: CreateEvaluatorVersionMutationInput!): CreateEvaluatorVersionMutationPayload
Arguments
CreateEvaluatorVersionMutationInput!
required
Show CreateEvaluatorVersionMutationInput fields
Show CreateEvaluatorVersionMutationInput fields
ID!
required
The evaluator ID in which the version will be created
String!
required
The commit message describing the changes in this version
String
Optional evaluator name to update
String
Optional evaluator description to update
TemplateEvaluationConfigInput
Show TemplateEvaluationConfigInput fields
Show TemplateEvaluationConfigInput fields
EvalColumnName!
required
The name of the template evaluation config
[String!]
The rails associated with the config
String!
required
The template associated with the config
String
Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same {variable} syntax as the main template.
Int!
required
The position of the config
Boolean!
required
Whether the config includes explanations
Boolean!
required
Whether the config uses function calling if available
Boolean
Whether the config uses structured output if available
JSONObject
The classification choices for the config (Record<string, number>)
TemplateEvaluationConfigDirection
The direction for the config One of:
maximize, minimize, none.EvalDataGranularityType
Data granularity (span, trace, or session). Defaults to span when omitted. One of:
span, trace, session.TemplateEvaluationLlmConfigInput
Llm Config for this evaluator
Show TemplateEvaluationLlmConfigInput fields
Show TemplateEvaluationLlmConfigInput fields
CodeEvaluationConfigInput
Show CodeEvaluationConfigInput fields
Show CodeEvaluationConfigInput fields
EvalColumnName!
required
The name of the evaluation config
String
The evaluation type associated with the config
String
The code block for the evaluation class
JSON
The evaluation input params
String
The package imports for the evaluation class
Int!
required
The position of the config
String
Optional query used to filter over a given data granularity (ex. session, trace)
[String!]
The data column associated with the config
EvalDataGranularityType
The data granularity associated with the config (span, trace, or session) One of:
span, trace, session.HarnessEvaluatorVersionInput
Show HarnessEvaluatorVersionInput fields
Show HarnessEvaluatorVersionInput fields
ID
Relay GID of the llm_integrations row. Provide to create a new sandbox; omit to inherit.
EvalColumnName!
required
Eval column prefix name. Harness output is written under eval.<name>.*.
String!
required
Natural-language eval template. {column_name} markers are substituted at run time.
JSON
Optional fixed label set. Null means the agent discovers labels freely.
Boolean
When true the agent must only emit labels from classificationChoices.
String
Optional model name hint forwarded to the pod as ANTHROPIC_MODEL.
HarnessEvaluationConfigDirection
Optimization direction for scored harness eval output. One of:
maximize, minimize, none.SystemOneEvaluatorVersionInput
Show SystemOneEvaluatorVersionInput fields
Show SystemOneEvaluatorVersionInput fields
String!
required
Relay global ID of the TypeSafe AI integration this evaluator calls.
String!
required
Provider model route, for example
jev-latest. Required: the column has no default, so which route was asked for is always recorded.String!
required
The authorās JSON-shaped state template, stored verbatim, carrying {variables} resolved per record.
[SystemOneQuestionInput!]!
required
At least one. Every question sees the same state and is evaluated independently in one request.
Show SystemOneQuestionInput fields
Show SystemOneQuestionInput fields
String!
required
Becomes the providerās question key and the published eval column suffix, so it is validated against the eval-name rules.
SystemOneQuestionType!
required
One of:
boolean, choice, score.String!
required
The complete judgment. Question names are not sent to the provider as context.
JSON
Required for choice and score; optional for boolean. Shape is validated per type.
OptimizationDirection
Read for score questions only, where it defaults to maximize. Normalized to
none for every other type. One of: minimize, maximize, none.RemoteEvaluatorVersionInput
String
Returns
Evaluator!
Show Evaluator fields
Show Evaluator fields
Scalar fields:
id, name, description, taskType, commitHash, commitMessage, createdAt, updatedAt.Object fields (select subfields): versionHistory, webhookSubscriptions, config, harnessEvaluationConfig, remoteEvaluationConfig, systemOneEvaluationConfig, createdBy, tags, onlineTasks.EvaluatorVersion!
Show EvaluatorVersion fields
Show EvaluatorVersion fields
Scalar fields:
id, evaluatorId, commitHash, commitMessage, createdAt, updatedAt, versionNumber.Object fields (select subfields): evaluator, harnessEvaluationConfig, remoteEvaluationConfig, systemOneEvaluationConfig, createdBy.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation CreateEvaluatorVersion($input: CreateEvaluatorVersionMutationInput!) {
createEvaluatorVersion(input: $input) {
evaluator { id name }
evaluatorVersion { id }
}
}
{
"input": {
"evaluatorId": "<ID>",
"commitMessage": "<string>"
}
}
editEvaluator
Edit an existing evaluatoreditEvaluator(input: EditEvaluatorMutationInput!): EditEvaluatorMutationPayload
Arguments
EditEvaluatorMutationInput!
required
Returns
Evaluator!
Show Evaluator fields
Show Evaluator fields
Scalar fields:
id, name, description, taskType, commitHash, commitMessage, createdAt, updatedAt.Object fields (select subfields): versionHistory, webhookSubscriptions, config, harnessEvaluationConfig, remoteEvaluationConfig, systemOneEvaluationConfig, createdBy, tags, onlineTasks.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation EditEvaluator($input: EditEvaluatorMutationInput!) {
editEvaluator(input: $input) {
evaluator { id name }
}
}
{
"input": {
"evaluatorId": "<ID>"
}
}
deleteEvaluator
Delete an existing evaluatordeleteEvaluator(input: DeleteEvaluatorMutationInput!): DeleteEvaluatorMutationPayload
Arguments
DeleteEvaluatorMutationInput!
required
Returns
Boolean!
Indicates whether the evaluator was deleted
Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation DeleteEvaluator($input: DeleteEvaluatorMutationInput!) {
deleteEvaluator(input: $input) {
success
}
}
{
"input": {
"evaluatorId": "<ID>"
}
}
createEvalTask
Create a new online eval taskcreateEvalTask(input: CreateEvalTaskMutationInput!): CreateEvalTaskMutationPayload
Arguments
CreateEvalTaskMutationInput!
required
Show CreateEvalTaskMutationInput fields
Show CreateEvalTaskMutationInput fields
ID
The model id for the model to be evaluated
ID
The dataset id for the dataset to be evaluated
String!
required
The name of the online task
[FilterItemInputType!]
The filters associated with the online task
Show FilterItemInputType fields
Show FilterItemInputType fields
FilterRowType!
required
The type of filter. One of:
featureLabel, tagLabel, predictionValue, actuals, modelVersion, batchId, predictionClass, predictionScore, actualClass, actualScore, topKPercentile, spanProperty, llmEval, annotation, userAnnotation.ComparisonOperator!
required
The operator of the filter. One of:
greaterThan, lessThan, equals, notEquals, greaterThanOrEqual, lessThanOrEqual, topN, contains, containsString, similarTo.DimensionInput
Show DimensionInput fields
Show DimensionInput fields
ID!
required
String!
required
The label of this dimension, e.g. āBank Nameā or āageā
DimensionDataType!
required
The data type of the values in dimensionName, ex. āFLOATā, āSTRINGā, etc. One of:
STRING, LONG, FLOAT, DOUBLE, EMBEDDING, STRING_LIST, DICTIONARY.DimensionCategory
The category of the dimension, e.g. ātagā or āfeatureLabelā, etc. One of:
featureLabel, prediction, actuals, actualScore, actualClass, predictionClass, predictionScore, tag, spanProperty, llmEval, annotation, userAnnotation, modelVersion, batchId.[DimensionValueInput!]
The dimension values of the filter.
String
Filters to apply to the dataset. Compatible with existing filters, joined with AND.
MultiSpanQueryDefinitionInput
Multi-span query definition for admission. Mutually exclusive with queryFilter.
Float!
required
The sampling rate of the online task
Boolean!
required
Whether the online task runs continuously
OnlineTaskLLMConfigInput
The LLM config for the online task
Show OnlineTaskLLMConfigInput fields
Show OnlineTaskLLMConfigInput fields
ID!
required
The selected named LLM integration id (relay global id). If provided, server will use its configuration.
String!
required
The LLM model
JSONObject!
required
parameters used when running the Llm
JSONObject!
required
parameters used to initialize the Llm
Float
The temperature of the LLM config
LLMIntegrationProvider
The LLM provider One of:
openAI, anthropic, awsBedrock, azureOpenAI, vertexAI, custom, nvidiaNim, gemini, litellm, fireworks, togetherAi, cursor, typeSafeAi.String
The deployment name of the LLM config. Required for azure openai provider
String
The api version of the LLM config. Required for azure openai provider
String
The api endpoint of the LLM config. Required for azure openai provider
String
The region of the LLM config
Float
The max tokens of the LLM config
Float
The max completion tokens of the LLM config
Float
The top P of the LLM config
Float
The top K of the LLM config
[String!]
The stop associated with the config
String
The id of the Custom Model Endpoint. Required if provider is custom.
[TemplateEvaluationConfigInput!]
Show TemplateEvaluationConfigInput fields
Show TemplateEvaluationConfigInput fields
EvalColumnName!
required
The name of the template evaluation config
[String!]
The rails associated with the config
String!
required
The template associated with the config
String
Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same {variable} syntax as the main template.
Int!
required
The position of the config
Boolean!
required
Whether the config includes explanations
Boolean!
required
Whether the config uses function calling if available
Boolean
Whether the config uses structured output if available
JSONObject
The classification choices for the config (Record<string, number>)
TemplateEvaluationConfigDirection
The direction for the config One of:
maximize, minimize, none.EvalDataGranularityType
Data granularity (span, trace, or session). Defaults to span when omitted. One of:
span, trace, session.TemplateEvaluationLlmConfigInput
Llm Config for this evaluator
Show TemplateEvaluationLlmConfigInput fields
Show TemplateEvaluationLlmConfigInput fields
[CodeEvaluationConfigInput!]
Show CodeEvaluationConfigInput fields
Show CodeEvaluationConfigInput fields
EvalColumnName!
required
The name of the evaluation config
String
The evaluation type associated with the config
String
The code block for the evaluation class
JSON
The evaluation input params
String
The package imports for the evaluation class
Int!
required
The position of the config
String
Optional query used to filter over a given data granularity (ex. session, trace)
[String!]
The data column associated with the config
EvalDataGranularityType
The data granularity associated with the config (span, trace, or session) One of:
span, trace, session.[ID!]
[OnlineTaskPostProcessingStepInput!]
Show OnlineTaskPostProcessingStepInput fields
Show OnlineTaskPostProcessingStepInput fields
[OnlineTaskEvaluatorInput!]
Show OnlineTaskEvaluatorInput fields
Show OnlineTaskEvaluatorInput fields
ID!
required
id of evaluator to attach to online task
ID
Optional id of the evaluator version to pin. Omit (or null) to always run the latest version.
Int!
required
position of evaluator in ui
String
Optional query used to filter over a given data granularity (ex. session, trace)
JSON
Mapping of evaluator variable names to dataset column names
[SubqueryColumnMappingInput!]
Subquery-aware variable mappings. Populated instead of columnMappings on tasks that have a multiSpanQuery.
Show SubqueryColumnMappingInput fields
Show SubqueryColumnMappingInput fields
String!
required
Evaluator variable name
[String!]!
required
Source subquery names (A-E) from the taskās multi_span_query that may populate this variable. When multiple listed subqueries match for a unit, their resolved values are concatenated. The list must contain at least one subquery; an empty list matches nothing and is rejected.
String!
required
Span attribute path to resolve within each subquery
Boolean
Whether tracing is enabled for the online task
ID
Arize sandbox integration ID (BYO path for harness tasks). When provided for a harness task the integration is linked to this task so the sandbox pod receives the Arize API key. Omit to use auto-provisioning (phase 2).
String
Returns
OnlineTask!
Show OnlineTask fields
Show OnlineTask fields
Scalar fields:
id, taskType, name, samplingRate, lastRunAt, queryFilter, createdAt, updatedAt, deletedAt, runContinuously, tracingEnabled, experimentIds, hasUnavailableEvaluatorIntegration.Object fields (select subfields): model, multiSpanQuery, createdBy, llmConfig, taskConfig, evaluators, onlineTaskEvaluators, allEvaluatorConfigs, taskRuns, experiments, dataset, postProcessingSteps.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation CreateEvalTask($input: CreateEvalTaskMutationInput!) {
createEvalTask(input: $input) {
evalTask { id name }
}
}
{
"input": {
"name": "<string>",
"samplingRate": 1,
"runContinuously": true
}
}
patchEvalTask
patchEvalTask(input: PatchEvalTaskMutationInput!): PatchEvalTaskMutationPayload
Arguments
PatchEvalTaskMutationInput!
required
Show PatchEvalTaskMutationInput fields
Show PatchEvalTaskMutationInput fields
ID!
required
The online task id for the eval task to be patched
ID
The model id for the model to be evaluated
ID
The dataset id for the dataset to be evaluated
String
The name of the online task
[FilterItemInputType!]
The filters associated with the online task
Show FilterItemInputType fields
Show FilterItemInputType fields
FilterRowType!
required
The type of filter. One of:
featureLabel, tagLabel, predictionValue, actuals, modelVersion, batchId, predictionClass, predictionScore, actualClass, actualScore, topKPercentile, spanProperty, llmEval, annotation, userAnnotation.ComparisonOperator!
required
The operator of the filter. One of:
greaterThan, lessThan, equals, notEquals, greaterThanOrEqual, lessThanOrEqual, topN, contains, containsString, similarTo.DimensionInput
Show DimensionInput fields
Show DimensionInput fields
ID!
required
String!
required
The label of this dimension, e.g. āBank Nameā or āageā
DimensionDataType!
required
The data type of the values in dimensionName, ex. āFLOATā, āSTRINGā, etc. One of:
STRING, LONG, FLOAT, DOUBLE, EMBEDDING, STRING_LIST, DICTIONARY.DimensionCategory
The category of the dimension, e.g. ātagā or āfeatureLabelā, etc. One of:
featureLabel, prediction, actuals, actualScore, actualClass, predictionClass, predictionScore, tag, spanProperty, llmEval, annotation, userAnnotation, modelVersion, batchId.[DimensionValueInput!]
The dimension values of the filter.
String
Filters to apply to the dataset. Compatible with existing filters, joined with AND.
MultiSpanQueryDefinitionInput
Multi-span query definition for admission. Mutually exclusive with queryFilter.
Float
The sampling rate of the online task
Boolean
Whether the online task runs continuously
OnlineTaskLLMConfigInput
The LLM config for the online task
Show OnlineTaskLLMConfigInput fields
Show OnlineTaskLLMConfigInput fields
ID!
required
The selected named LLM integration id (relay global id). If provided, server will use its configuration.
String!
required
The LLM model
JSONObject!
required
parameters used when running the Llm
JSONObject!
required
parameters used to initialize the Llm
Float
The temperature of the LLM config
LLMIntegrationProvider
The LLM provider One of:
openAI, anthropic, awsBedrock, azureOpenAI, vertexAI, custom, nvidiaNim, gemini, litellm, fireworks, togetherAi, cursor, typeSafeAi.String
The deployment name of the LLM config. Required for azure openai provider
String
The api version of the LLM config. Required for azure openai provider
String
The api endpoint of the LLM config. Required for azure openai provider
String
The region of the LLM config
Float
The max tokens of the LLM config
Float
The max completion tokens of the LLM config
Float
The top P of the LLM config
Float
The top K of the LLM config
[String!]
The stop associated with the config
String
The id of the Custom Model Endpoint. Required if provider is custom.
[TemplateEvaluationConfigInput!]
Show TemplateEvaluationConfigInput fields
Show TemplateEvaluationConfigInput fields
EvalColumnName!
required
The name of the template evaluation config
[String!]
The rails associated with the config
String!
required
The template associated with the config
String
Optional turn-definition template for session-granularity evaluators. When non-empty, the evaluator operates in custom-turn mode: each session trace yields one turn rendered from this template. Uses the same {variable} syntax as the main template.
Int!
required
The position of the config
Boolean!
required
Whether the config includes explanations
Boolean!
required
Whether the config uses function calling if available
Boolean
Whether the config uses structured output if available
JSONObject
The classification choices for the config (Record<string, number>)
TemplateEvaluationConfigDirection
The direction for the config One of:
maximize, minimize, none.EvalDataGranularityType
Data granularity (span, trace, or session). Defaults to span when omitted. One of:
span, trace, session.TemplateEvaluationLlmConfigInput
Llm Config for this evaluator
Show TemplateEvaluationLlmConfigInput fields
Show TemplateEvaluationLlmConfigInput fields
[CodeEvaluationConfigInput!]
Show CodeEvaluationConfigInput fields
Show CodeEvaluationConfigInput fields
EvalColumnName!
required
The name of the evaluation config
String
The evaluation type associated with the config
String
The code block for the evaluation class
JSON
The evaluation input params
String
The package imports for the evaluation class
Int!
required
The position of the config
String
Optional query used to filter over a given data granularity (ex. session, trace)
[String!]
The data column associated with the config
EvalDataGranularityType
The data granularity associated with the config (span, trace, or session) One of:
span, trace, session.[ID!]
[OnlineTaskPostProcessingStepInput!]
Show OnlineTaskPostProcessingStepInput fields
Show OnlineTaskPostProcessingStepInput fields
[OnlineTaskEvaluatorInput!]
Show OnlineTaskEvaluatorInput fields
Show OnlineTaskEvaluatorInput fields
ID!
required
id of evaluator to attach to online task
ID
Optional id of the evaluator version to pin. Omit (or null) to always run the latest version.
Int!
required
position of evaluator in ui
String
Optional query used to filter over a given data granularity (ex. session, trace)
JSON
Mapping of evaluator variable names to dataset column names
[SubqueryColumnMappingInput!]
Subquery-aware variable mappings. Populated instead of columnMappings on tasks that have a multiSpanQuery.
Show SubqueryColumnMappingInput fields
Show SubqueryColumnMappingInput fields
String!
required
Evaluator variable name
[String!]!
required
Source subquery names (A-E) from the taskās multi_span_query that may populate this variable. When multiple listed subqueries match for a unit, their resolved values are concatenated. The list must contain at least one subquery; an empty list matches nothing and is rejected.
String!
required
Span attribute path to resolve within each subquery
Boolean
Whether tracing is enabled for the online task
String
Returns
OnlineTask!
Show OnlineTask fields
Show OnlineTask fields
Scalar fields:
id, taskType, name, samplingRate, lastRunAt, queryFilter, createdAt, updatedAt, deletedAt, runContinuously, tracingEnabled, experimentIds, hasUnavailableEvaluatorIntegration.Object fields (select subfields): model, multiSpanQuery, createdBy, llmConfig, taskConfig, evaluators, onlineTaskEvaluators, allEvaluatorConfigs, taskRuns, experiments, dataset, postProcessingSteps.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation PatchEvalTask($input: PatchEvalTaskMutationInput!) {
patchEvalTask(input: $input) {
evalTask { id name }
}
}
{
"input": {
"onlineTaskId": "<ID>"
}
}
patchEvalTaskWithEvalHub
patchEvalTaskWithEvalHub(input: PatchEvalTaskWithEvalHubMutationInput!): PatchEvalTaskWithEvalHubMutationPayload
Arguments
PatchEvalTaskWithEvalHubMutationInput!
required
Show PatchEvalTaskWithEvalHubMutationInput fields
Show PatchEvalTaskWithEvalHubMutationInput fields
ID!
required
The online task id for the eval task to be patched
ID
The model id for the model to be evaluated
ID
The dataset id for the dataset to be evaluated
String
The name of the online task
[FilterItemInputType!]
The filters associated with the online task
Show FilterItemInputType fields
Show FilterItemInputType fields
FilterRowType!
required
The type of filter. One of:
featureLabel, tagLabel, predictionValue, actuals, modelVersion, batchId, predictionClass, predictionScore, actualClass, actualScore, topKPercentile, spanProperty, llmEval, annotation, userAnnotation.ComparisonOperator!
required
The operator of the filter. One of:
greaterThan, lessThan, equals, notEquals, greaterThanOrEqual, lessThanOrEqual, topN, contains, containsString, similarTo.DimensionInput
Show DimensionInput fields
Show DimensionInput fields
ID!
required
String!
required
The label of this dimension, e.g. āBank Nameā or āageā
DimensionDataType!
required
The data type of the values in dimensionName, ex. āFLOATā, āSTRINGā, etc. One of:
STRING, LONG, FLOAT, DOUBLE, EMBEDDING, STRING_LIST, DICTIONARY.DimensionCategory
The category of the dimension, e.g. ātagā or āfeatureLabelā, etc. One of:
featureLabel, prediction, actuals, actualScore, actualClass, predictionClass, predictionScore, tag, spanProperty, llmEval, annotation, userAnnotation, modelVersion, batchId.[DimensionValueInput!]
The dimension values of the filter.
String
Filters to apply to the dataset. Compatible with existing filters, joined with AND.
MultiSpanQueryDefinitionInput
Multi-span query definition for admission. Mutually exclusive with queryFilter.
Float
The sampling rate of the online task
Boolean
Whether the online task runs continuously
OnlineTaskLLMConfigInput
The LLM config for the online task
Show OnlineTaskLLMConfigInput fields
Show OnlineTaskLLMConfigInput fields
ID!
required
The selected named LLM integration id (relay global id). If provided, server will use its configuration.
String!
required
The LLM model
JSONObject!
required
parameters used when running the Llm
JSONObject!
required
parameters used to initialize the Llm
Float
The temperature of the LLM config
LLMIntegrationProvider
The LLM provider One of:
openAI, anthropic, awsBedrock, azureOpenAI, vertexAI, custom, nvidiaNim, gemini, litellm, fireworks, togetherAi, cursor, typeSafeAi.String
The deployment name of the LLM config. Required for azure openai provider
String
The api version of the LLM config. Required for azure openai provider
String
The api endpoint of the LLM config. Required for azure openai provider
String
The region of the LLM config
Float
The max tokens of the LLM config
Float
The max completion tokens of the LLM config
Float
The top P of the LLM config
Float
The top K of the LLM config
[String!]
The stop associated with the config
String
The id of the Custom Model Endpoint. Required if provider is custom.
[OnlineTaskEvaluatorInput!]
Show OnlineTaskEvaluatorInput fields
Show OnlineTaskEvaluatorInput fields
ID!
required
id of evaluator to attach to online task
ID
Optional id of the evaluator version to pin. Omit (or null) to always run the latest version.
Int!
required
position of evaluator in ui
String
Optional query used to filter over a given data granularity (ex. session, trace)
JSON
Mapping of evaluator variable names to dataset column names
[SubqueryColumnMappingInput!]
Subquery-aware variable mappings. Populated instead of columnMappings on tasks that have a multiSpanQuery.
Show SubqueryColumnMappingInput fields
Show SubqueryColumnMappingInput fields
String!
required
Evaluator variable name
[String!]!
required
Source subquery names (A-E) from the taskās multi_span_query that may populate this variable. When multiple listed subqueries match for a unit, their resolved values are concatenated. The list must contain at least one subquery; an empty list matches nothing and is rejected.
String!
required
Span attribute path to resolve within each subquery
[ID!]
[OnlineTaskPostProcessingStepInput!]
Show OnlineTaskPostProcessingStepInput fields
Show OnlineTaskPostProcessingStepInput fields
String
Returns
OnlineTask!
Show OnlineTask fields
Show OnlineTask fields
Scalar fields:
id, taskType, name, samplingRate, lastRunAt, queryFilter, createdAt, updatedAt, deletedAt, runContinuously, tracingEnabled, experimentIds, hasUnavailableEvaluatorIntegration.Object fields (select subfields): model, multiSpanQuery, createdBy, llmConfig, taskConfig, evaluators, onlineTaskEvaluators, allEvaluatorConfigs, taskRuns, experiments, dataset, postProcessingSteps.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation PatchEvalTaskWithEvalHub($input: PatchEvalTaskWithEvalHubMutationInput!) {
patchEvalTaskWithEvalHub(input: $input) {
evalTask { id name }
}
}
{
"input": {
"onlineTaskId": "<ID>"
}
}
migrateTaskToEvalHub
migrates a taskās evaluators to eval hubmigrateTaskToEvalHub(input: MigrateTaskToEvalHubMutationInput!): MigrateTaskToEvalHubMutationPayload
Arguments
MigrateTaskToEvalHubMutationInput!
required
Returns
OnlineTask!
Show OnlineTask fields
Show OnlineTask fields
Scalar fields:
id, taskType, name, samplingRate, lastRunAt, queryFilter, createdAt, updatedAt, deletedAt, runContinuously, tracingEnabled, experimentIds, hasUnavailableEvaluatorIntegration.Object fields (select subfields): model, multiSpanQuery, createdBy, llmConfig, taskConfig, evaluators, onlineTaskEvaluators, allEvaluatorConfigs, taskRuns, experiments, dataset, postProcessingSteps.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation MigrateTaskToEvalHub($input: MigrateTaskToEvalHubMutationInput!) {
migrateTaskToEvalHub(input: $input) {
onlineTask { id name }
}
}
{
"input": {
"onlineTaskId": "<ID>"
}
}
runOnlineTask
Run an online taskrunOnlineTask(input: RunOnlineTaskMutationInput!): RunOnlineTaskMutationPayload
Arguments
RunOnlineTaskMutationInput!
required
Show RunOnlineTaskMutationInput fields
Show RunOnlineTaskMutationInput fields
ID
The task id for the task to be processed
DateTime
Start time of the data processed by the online task
DateTime
Start time of the data processed by the online task
Int
Maximum number of spans processed by the online task. If not specified, default to 10,000.
Boolean
Whether to override the existing evaluation labels for the task.
Boolean
When true, the worker reproduces the source runās capped row set by omitting eval-null scan filters. Used by Rerun on Errors.
[ID!]
The experiment ids to run the task on
String
Returns
RunOnlineTaskResponse!
Result for queuing online task run on date range Union of
TaskError, CreateTaskRunResponse. Use inline fragments (... on TypeName).Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation RunOnlineTask($input: RunOnlineTaskMutationInput!) {
runOnlineTask(input: $input) {
result { __typename }
}
}
{
"input": {}
}
cancelOnlineTaskRun
Cancel an online task runcancelOnlineTaskRun(input: CancelOnlineTaskRunMutationInput!): CancelOnlineTaskRunMutationPayload
Arguments
CancelOnlineTaskRunMutationInput!
required
Returns
CancelOnlineTaskResponse!
Result for cancelling online task run Union of
TaskError, CancelTaskRunResponse. Use inline fragments (... on TypeName).Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation CancelOnlineTaskRun($input: CancelOnlineTaskRunMutationInput!) {
cancelOnlineTaskRun(input: $input) {
result { __typename }
}
}
{
"input": {
"onlineTaskRunId": "<ID>"
}
}
deleteOnlineTask
deleteOnlineTask(input: DeleteOnlineTaskMutationInput!): DeleteOnlineTaskMutationPayload
Arguments
DeleteOnlineTaskMutationInput!
required
Returns
Boolean!
True when all requested tasks were deleted (failedTasks is empty). Kept for backward compatibility.
[ID!]!
IDs of the tasks that were successfully deleted.
[DeleteOnlineTaskFailure!]!
Tasks that could not be deleted, each with the reason for failure. The caller should inspect and retry or investigate individually.
Show DeleteOnlineTaskFailure fields
Show DeleteOnlineTaskFailure fields
Scalar fields:
taskId, message.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation DeleteOnlineTask($input: DeleteOnlineTaskMutationInput!) {
deleteOnlineTask(input: $input) {
success
deletedTaskIds
failedTasks { message }
}
}
{
"input": {
"onlineTaskIds": [
"<ID>"
]
}
}
runExperimentTask
Run an experiment task with configurable LLM interactionsrunExperimentTask(input: RunExperimentTaskMutationInput!): RunExperimentTaskMutationPayload
Arguments
RunExperimentTaskMutationInput!
required
Show RunExperimentTaskMutationInput fields
Show RunExperimentTaskMutationInput fields
ID!
required
The dataset id for the experiment
String!
required
The dataset name for the experiment
ID!
required
The dataset version id for the experiment
ID!
required
The space id for the experiment
[ExampleOverridesInput!]!
required
[String!]
The example ids to run the experiment on
Int
The number of examples to run the experiment on
String
Optional dataset query filter. When set, the experiment (and associated evaluations) run only on examples matching the filter.
[RunExperimentConfigInput!]!
required
The run configurations for the experiment
Show RunExperimentConfigInput fields
Show RunExperimentConfigInput fields
ExperimentTypeEnum!
required
Discriminant for what kind of work is performed per example. One of: āllm_generationā, ātemplate_evaluationā, ācode_evaluationā, āagent_callā. One of:
llm_generation, template_evaluation, code_evaluation, agent_call.LLMInput
LLM input (messages, model, integration). Required for llm_generation; provider/model fields also used for template_evaluation. Not used for agent_call.
Show LLMInput fields
Show LLMInput fields
ExternalLLMProvider!
required
The provider to use for the call. One of:
openAI, azureOpenAI, anthropic, vertexAI, awsBedrock, custom, nvidiaNim, gemini, litellm, fireworks, togetherAi.String
The model to use for the call.
[LLMMessageInput!]!
required
The full message to send to the LLM (e.g., a prompt template with all variables filled in). Nested
LLMMessageInput (same shape as above).InvocationParamsInput!
required
The params to call the LLM with. Nested
InvocationParamsInput (same shape as above).ProviderParamsInput!
required
The provider params to use for the call. Nested
ProviderParamsInput (same shape as above).ID
The selected named LLM integration id (relay global id). If provided, server will use its configuration.
String
Unique identifier for this input
PromptVersionInputVariableFormatEnum
The input variable format for determining prompt variables in the messages. Only applicable for llm_generation. One of:
F_STRING, MUSTACHE, NONE.String!
required
Name for the experiment
Int!
required
The instance id for the experiment
ID
The version ID of the prompt from Prompt Hub (if loaded from hub)
TemplateEvalConfigInput
Required when experimentType = ātemplate_evaluationā.
Show TemplateEvalConfigInput fields
Show TemplateEvalConfigInput fields
String!
required
The evaluation prompt template.
Boolean!
required
Whether to include LLM explanations in output.
JSON
Map of choice label to numeric score (e.g. {relevant: 1, irrelevant: 0}). Required when templateConfig is provided.
JSON
Maps named eval param keys to dataset column paths.
ID
The evaluator version ID from Eval Hub (if the template eval was loaded from hub)
CodeEvalConfigInput
Required when experimentType = ācode_evaluationā.
Show CodeEvalConfigInput fields
Show CodeEvalConfigInput fields
String!
required
Managed evaluator class (e.g. JSONParsable, MatchesRegex) or āCustomā.
String
JSON-encoded params for managed evaluator classes.
String
Class name for managed evaluators.
CustomCodeEvalInfoInput
Only set when codeEvalClass is āCustomā. Nested
CustomCodeEvalInfoInput (same shape as above).JSON!
required
Maps named eval param keys to dataset column paths.
AgentConfigInput
Int
Per-call LLM timeout in seconds (whole number, 30-1800) for llm_generation and template_evaluation runs; also applies to any LLM evaluator judge calls attached to the run. Rejected for other experiment types. Null uses the system default.
JSON
The tracing metadata for the experiment
[String!]
List of evaluation task IDs to run after the experiment completes
String
Returns
RunExperimentTaskResponse!
Result for running experiment task Union of
RunExperimentTaskSuccess, RunExperimentTaskError. Use inline fragments (... on TypeName).Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation RunExperimentTask($input: RunExperimentTaskMutationInput!) {
runExperimentTask(input: $input) {
result { __typename }
}
}
{
"input": {
"datasetId": "<ID>",
"datasetName": "<string>",
"datasetVersionId": "<ID>",
"spaceId": "<ID>",
"exampleOverrides": [
{
"exampleId": "<string>",
"overridesJSON": {}
}
],
"runConfigurations": [
{
"experimentType": "llm_generation",
"experimentName": "<string>",
"instanceId": 1
}
]
}
}
createPromptOptimizationTask
Create a new prompt optimization taskcreatePromptOptimizationTask(input: CreatePromptOptimizationTaskMutationInput!): CreatePromptOptimizationTaskMutationPayload
Arguments
CreatePromptOptimizationTaskMutationInput!
required
Show CreatePromptOptimizationTaskMutationInput fields
Show CreatePromptOptimizationTaskMutationInput fields
ID!
required
The dataset id to use for prompt optimization
ID
Optional experiment ID to use as datasource. Only one experiment is supported for prompt optimization tasks. If not provided, the dataset will be used.
String!
required
The name of the prompt optimization task
Boolean!
required
Whether the online task runs continuously
OnlineTaskLLMConfigInput!
required
The LLM config for the online task
Show OnlineTaskLLMConfigInput fields
Show OnlineTaskLLMConfigInput fields
ID!
required
The selected named LLM integration id (relay global id). If provided, server will use its configuration.
String!
required
The LLM model
JSONObject!
required
parameters used when running the Llm
JSONObject!
required
parameters used to initialize the Llm
Float
The temperature of the LLM config
LLMIntegrationProvider
The LLM provider One of:
openAI, anthropic, awsBedrock, azureOpenAI, vertexAI, custom, nvidiaNim, gemini, litellm, fireworks, togetherAi, cursor, typeSafeAi.String
The deployment name of the LLM config. Required for azure openai provider
String
The api version of the LLM config. Required for azure openai provider
String
The api endpoint of the LLM config. Required for azure openai provider
String
The region of the LLM config
Float
The max tokens of the LLM config
Float
The max completion tokens of the LLM config
Float
The top P of the LLM config
Float
The top K of the LLM config
[String!]
The stop associated with the config
String
The id of the Custom Model Endpoint. Required if provider is custom.
PromptOptimizationConfigInput!
required
The prompt optimization config
Boolean
Whether tracing is enabled for the online task
String
Returns
OnlineTask!
Show OnlineTask fields
Show OnlineTask fields
Scalar fields:
id, taskType, name, samplingRate, lastRunAt, queryFilter, createdAt, updatedAt, deletedAt, runContinuously, tracingEnabled, experimentIds, hasUnavailableEvaluatorIntegration.Object fields (select subfields): model, multiSpanQuery, createdBy, llmConfig, taskConfig, evaluators, onlineTaskEvaluators, allEvaluatorConfigs, taskRuns, experiments, dataset, postProcessingSteps.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation CreatePromptOptimizationTask($input: CreatePromptOptimizationTaskMutationInput!) {
createPromptOptimizationTask(input: $input) {
promptOptimizationTask { id name }
}
}
{
"input": {
"datasetId": "<ID>",
"name": "<string>",
"runContinuously": true,
"llmConfig": {
"integrationId": "<ID>",
"modelName": "<string>",
"invocationParameters": "<JSONObject>",
"providerParameters": "<JSONObject>"
},
"promptOptimizationConfig": {
"userInstructions": "<string>",
"outputColumn": "<string>",
"feedbackColumns": [
"<string>"
],
"batchSize": 1,
"promptId": "<ID>"
}
}
}
patchPromptOptimizationTask
patchPromptOptimizationTask(input: PatchPromptOptimizationTaskMutationInput!): PatchPromptOptimizationTaskMutationPayload
Arguments
PatchPromptOptimizationTaskMutationInput!
required
Show PatchPromptOptimizationTaskMutationInput fields
Show PatchPromptOptimizationTaskMutationInput fields
ID!
required
The online task id for the eval task to be patched
ID
The dataset id for the dataset to be evaluated
String
The name of the online task
Boolean
Whether the online task runs continuously
OnlineTaskLLMConfigInput
The LLM config for the online task
Show OnlineTaskLLMConfigInput fields
Show OnlineTaskLLMConfigInput fields
ID!
required
The selected named LLM integration id (relay global id). If provided, server will use its configuration.
String!
required
The LLM model
JSONObject!
required
parameters used when running the Llm
JSONObject!
required
parameters used to initialize the Llm
Float
The temperature of the LLM config
LLMIntegrationProvider
The LLM provider One of:
openAI, anthropic, awsBedrock, azureOpenAI, vertexAI, custom, nvidiaNim, gemini, litellm, fireworks, togetherAi, cursor, typeSafeAi.String
The deployment name of the LLM config. Required for azure openai provider
String
The api version of the LLM config. Required for azure openai provider
String
The api endpoint of the LLM config. Required for azure openai provider
String
The region of the LLM config
Float
The max tokens of the LLM config
Float
The max completion tokens of the LLM config
Float
The top P of the LLM config
Float
The top K of the LLM config
[String!]
The stop associated with the config
String
The id of the Custom Model Endpoint. Required if provider is custom.
PromptOptimizationConfigInput
The prompt optimization config
Boolean
Whether tracing is enabled for the online task
String
Returns
OnlineTask!
Show OnlineTask fields
Show OnlineTask fields
Scalar fields:
id, taskType, name, samplingRate, lastRunAt, queryFilter, createdAt, updatedAt, deletedAt, runContinuously, tracingEnabled, experimentIds, hasUnavailableEvaluatorIntegration.Object fields (select subfields): model, multiSpanQuery, createdBy, llmConfig, taskConfig, evaluators, onlineTaskEvaluators, allEvaluatorConfigs, taskRuns, experiments, dataset, postProcessingSteps.Example
Only required input fields are shown. Replace<ID> and <string> placeholders with real values; the object graph page shows how to look IDs up.
mutation PatchPromptOptimizationTask($input: PatchPromptOptimizationTaskMutationInput!) {
patchPromptOptimizationTask(input: $input) {
promptOptimizationTask { id name }
}
}
{
"input": {
"onlineTaskId": "<ID>"
}
}