Assign to a Labeling Queue

Instead of telling your team “go look at traces from yesterday,” create a labeling queue with the specific spans or traces that need review — a structured list with an annotation schema and progress tracking.

Add spans to a queue

Individual: Open a span → click Add to → Labeling Queue → choose or create a queue. Bulk: Select multiple spans in the traces table → Add to → Labeling Queue. [screenshot: add to labeling queue menu from span toolbar]

The review workflow

Open Annotation Queues from the left sidebar
Select a queue → review each record’s inputs, outputs, and existing evals
Add annotations using the queue’s annotation config
Move to the next record

[screenshot: annotator queue view with span details and annotation form]

A queue record can be a single span or an entire trace. When you annotate a trace-level record, the label is written as a trace-level annotation rather than on an individual span — use it when the judgment covers the whole request or conversation.

Choose what annotators see

When you create or edit a queue, use Annotator Columns to pick which record columns annotators see while labeling. Narrow the set to the fields the judgment actually depends on, so annotators are not reading past irrelevant metadata, and so sensitive columns stay out of the review view.

From queue to dataset

After labeling, create a dataset from the annotated results for experiments and fine-tuning.

With human review: Traces → Labeling Queue → Annotate → DatasetWithout human review: Traces → Dataset directly

Manage queues via API

You can create and manage annotation queues programmatically using the GraphQL API. This is useful for automating queue creation in CI/CD pipelines or building custom tooling.

mutation {
  createAnnotationQueue(input: {
    projectId: "PROJECT_ID"
    name: "Weekly Safety Review"
    description: "Spans flagged by safety eval for human review"
  }) {
    annotationQueue { id name }
  }
}

Annotation queue management also ships in the SDKs and the CLI, with list, create, get, update, and delete plus record management. See the annotation-queues reference for Python, TypeScript, Go, or the CLI. Most operations are generally available; annotating and assigning a record are still in beta.

Quickstart

Instrument

Observe

Agents

Evaluate

Improve

Machine Learning

Settings

Security

Assign to a Labeling Queue

Add spans to a queue

The review workflow

Choose what annotators see

From queue to dataset

Manage queues via API

​Add spans to a queue

​The review workflow

​Choose what annotators see

​From queue to dataset

​Manage queues via API

Add spans to a queue

The review workflow

Choose what annotators see

From queue to dataset

Manage queues via API