Create task

Creates a new task. Supported task types:

For RUN_EXPERIMENT tasks the run configuration is stored on the task. Each trigger (POST /v2/tasks/{task_id}/trigger) supplies per-run fields (experiment_name, optional example subset, etc.) and starts an async run. Poll GET /v2/task-runs/{run_id} until status reaches a terminal state.

Payload Requirements (template_evaluation / code_evaluation)

• At least one evaluator is required.
• Duplicate evaluator IDs are not allowed.
• When dataset_id is provided, experiment_ids must contain at least one entry.
• sampling_rate and is_continuous are only supported on project-based tasks.
• System-managed fields (id, created_at, updated_at) are rejected on input.
• evaluator_version_id pins an evaluator to one version. Omit it (or send null) to run that evaluator’s latest version, which is the default. The version must belong to the evaluator named by evaluator_id, and every evaluator on the task must resolve to the same data scope — both return 422. List an evaluator’s versions with GET /v2/evaluators/{evaluator_id}/versions.
• Every evaluator must match the task type: template evaluators on TEMPLATE_EVALUATION, code evaluators on CODE_EVALUATION. An evaluator of a different kind (for example a DECISION_JUDGE evaluator) returns 422.
• All evaluators on a task must be of the same kind. A list that mixes evaluator kinds returns 400.

Payload Requirements (decision_judge_evaluation)

• project_id is required; dataset_id and experiment_ids return 422.
• Every evaluator must be a DECISION_JUDGE evaluator. A list that mixes Decision Judge and other evaluator kinds returns 400. A list with only other evaluator kinds returns 422, and a Decision Judge evaluator on any other task type returns 422.
• Only the span shape is accepted; query_filters and query_mappings return 422.
• Returns 403 when Decision Judge evaluators are not enabled for the account.

Payload Requirements (run_experiment)

• dataset_id is required; project_id must be omitted.
• run_configuration is required; evaluators, experiment_ids, sampling_rate, is_continuous, and query_filter must be omitted.

Valid example (template_evaluation, project-based)

{
  "name": "Production Hallucination Check",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "sampling_rate": 1.0,
  "is_continuous": true,
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}
    }
  ]
}

Valid example (pinned to a specific evaluator version)

{
  "name": "Hallucination Check v3",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjo5OTphQmNE",
      "column_mappings": {"input": "attributes.input.value"}
    }
  ]
}

Invalid example (run_experiment missing run_configuration)

{
  "name": "My Experiment",
  "type": "RUN_EXPERIMENT",
  "dataset_id": "RGF0YXNldDo1NjpxUndY"
}

Invalid example (422 — the version belongs to a different evaluator)

{
  "name": "Mismatched Pin",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjo3OmFCY0Q="
    }
  ]
}