Create task

Creates a new task. Supported task types:

For RUN_EXPERIMENT tasks the run configuration is stored on the task. Each trigger (POST /v2/tasks/{task_id}/trigger) supplies per-run fields (experiment_name, optional example subset, etc.) and starts an async run. Poll GET /v2/task-runs/{run_id} until status reaches a terminal state.

Payload Requirements (template_evaluation / code_evaluation)

• At least one evaluator is required.
• Duplicate evaluator IDs are not allowed.
• When dataset_id is provided, experiment_ids must contain at least one entry.
• sampling_rate and is_continuous are only supported on project-based tasks.
• System-managed fields (id, created_at, updated_at) are rejected on input.
• evaluator_version_id pins an evaluator to one version. Omit it (or send null) to run that evaluator’s latest version, which is the default. The version must belong to the evaluator named by evaluator_id, and every evaluator on the task must resolve to the same data scope — both return 422. List an evaluator’s versions with GET /v2/evaluators/{evaluator_id}/versions.

Payload Requirements (run_experiment)

• dataset_id is required; project_id must be omitted.
• run_configuration is required; evaluators, experiment_ids, sampling_rate, is_continuous, and query_filter must be omitted.

Valid example (template_evaluation, project-based)

{
  "name": "Production Hallucination Check",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "sampling_rate": 1.0,
  "is_continuous": true,
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}
    }
  ]
}

Valid example (pinned to a specific evaluator version)

{
  "name": "Hallucination Check v3",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjo5OTphQmNE",
      "column_mappings": {"input": "attributes.input.value"}
    }
  ]
}

Invalid example (run_experiment missing run_configuration)

{
  "name": "My Experiment",
  "type": "RUN_EXPERIMENT",
  "dataset_id": "RGF0YXNldDo1NjpxUndY"
}

Invalid example (422 — the version belongs to a different evaluator)

{
  "name": "Mismatched Pin",
  "type": "TEMPLATE_EVALUATION",
  "project_id": "TW9kZWw6MTIzOmFCY0Q=",
  "evaluators": [
    {
      "evaluator_id": "RXZhbHVhdG9yOjEyOmFCY0Q=",
      "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjo3OmFCY0Q="
    }
  ]
}