Create alignment run

Scores how closely one evaluator version’s labels matched the human annotations on a dataset version, using the column pair and comparison function named by the alignment task in the path.

Scoring is synchronous: the score is computed before the response is returned, so the created run already carries its score and comparable_row_count. There is no run to poll for.

A run that could not be scored still returns 201. The run resource was created, so it is returned with status: FAILED and a public failed_reason. Backend error details are kept in server logs. A failed score is not a server error and is not reported as 5xx. Check status on every response rather than relying on the status code alone.

Read score and comparable_row_count together. score is null, not 0, when no row had both a judge label and a human annotation: nothing was comparable, so no agreement was measured. A score of 0 means rows were compared and none agreed. A high score over a small comparable_row_count is not a meaningful result.

A finished run is never rewritten. Scoring the same version again creates another run, because both the annotations and the task’s columns can change between scores.

Payload Requirements

• evaluator_version_id is required: the evaluator version to score. The caller must have update access to the evaluator that owns it.
• dataset_version_id is required and must belong to the alignment task’s dataset. A dataset version from another dataset returns 404, the same as one that does not exist.
• predicted_entity is required and names what produced the judge labels: entity_type: DATASET_VERSION when the labels were already on the dataset, or entity_type: EXPERIMENT when an experiment produced them. An entity that does not belong to the same dataset returns 404.
• System-managed fields (id, created_at, updated_at, status, score, comparable_row_count, failed_reason) are rejected on input. alignment_task_id is taken from the path and is rejected in the body.

The response records predicted_entity on successful and failed runs. Malformed IDs, incorrect field types, and a body that is not an object return 400. Missing required values, unsupported entity types, and unknown fields return 422.

Valid example

{
  "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjoxMDpYeVp3",
  "dataset_version_id": "RGF0YXNldFZlcnNpb246NTY6UXJTdA==",
  "predicted_entity": {
    "id": "RGF0YXNldFZlcnNpb246NTY6UXJTdA==",
    "entity_type": "DATASET_VERSION"
  }
}

Invalid example (score is computed by the server and cannot be supplied)

{
  "evaluator_version_id": "RXZhbHVhdG9yVmVyc2lvbjoxMDpYeVp3",
  "dataset_version_id": "RGF0YXNldFZlcnNpb246NTY6UXJTdA==",
  "predicted_entity": {
    "id": "RGF0YXNldFZlcnNpb246NTY6UXJTdA==",
    "entity_type": "DATASET_VERSION"
  },
  "score": 0.9
}