Build a dataset

​

Experiment planning

​

Experiment dataset requirements

• Column names are part of the contract. Placeholders and prompt variables resolve by exact column name, so renaming a column breaks every run that references it. Keep names stable across dataset versions. Names you choose read best as variables, such as question or expected_category. Trace-sourced columns keep their OpenInference names and work as placeholders unchanged, so attributes.input.value becomes {{dataset.attributes.input.value}}.
• A dataset built for one experiment type usually works for another. The same rows that fill a prompt variable can fill a field in an agent request body. A regression set collected while testing prompts stays useful once that logic moves into a deployed agent.

​

Choose the variable

1. Open a bad outcome in Traces: a thumbs-down, a low eval score, an exception, or anything a reviewer flagged.
2. Step through the trace span by span. Was the prompt ambiguous? Did the model ignore an instruction? Did the retriever return the wrong chunk? Did a tool call fail to parse?
3. Identify the step that most directly caused the bad outcome. That step is your experimental variable, and the spans around it are the rows worth collecting.

• Prompt: wording, few-shot examples, output schema, or tool definitions.
• Model: the same prompt against a different LLM.
• Invocation parameters: temperature, top-p, max tokens.
• Pipeline or agent behavior: retrieval strategy, routing, tool-using loops, or any logic that lives outside a single prompt.

​

Plan a baseline

​

Dataset design

​

What to include

​

Dataset types

• Regression: Examples where the app has already failed. Use these to verify a fix holds and doesn’t quietly reintroduce the bug.
• Golden: Inputs with hand-labeled expected outputs, a stable benchmark for comparing prompt and model changes.
• Synthetic: Generated examples that mimic real inputs. Useful when production data is thin, sensitive, or missing the edge cases you want to stress-test.

​

Dataset row schema

​

Common dataset row shapes

{
  "input": {
    "messages": [{"role": "system", "content": "You are an expert SQL assistant"}]
  },
  "output": {
    "messages": [{"role": "assistant", "content": "SELECT * FROM users;"}]
  }
}

​

Create a dataset

• By Arize Skills
• By Alyx
• By UI
• By Code

The Arize skills plugin wires dataset and trace workflows into your coding agent through the ax CLI.From traces. Combine arize-trace with arize-dataset. Try:

• “Export error spans from the last 7 days in my production-chatbot project and create a dataset called error-regression-v1.”
• “Find spans where annotation.hallucination.label = 'yes' over the past 14 days and save them as hallucination-examples.”

From a local file. Point the arize-dataset skill at a CSV, JSON, JSONL, or Parquet file you already have. Try:

• “Create a dataset called billing-qa-v1 from ./data/billing_qa.csv in my support space.”
• “Append the rows in new_edge_cases.jsonl to my existing edge-cases dataset.”

Generate synthetic rows. Have the agent draft examples for you. Try:

• “Generate 50 synthetic billing support tickets with query and expected_category fields, then save as support-synthetic-v1.”
• “Draft 20 adversarial inputs targeting prompt injection for my chat agent and save as adversarial-v1.”

Alyx builds datasets directly from the app. It’s available on the pages where you’re already looking at traces, datasets, and prompts (see where to find Alyx for the full list of surfaces). Ask Alyx to turn specific spans into dataset rows, or have it draft synthetic examples when you don’t have production data to pull from.From traces. Try:

• “Add this span and every similar error to a new regression dataset.”
• “Show me the most common failure patterns in the last 24 hours and add one example of each to a new dataset.”

Generate synthetic rows. Try:

• “Create a synthetic dataset of 100 examples covering billing, technical, and general support categories.”
• “Generate 30 edge cases for my router prompt and save them as a new dataset.”

From the Traces table. Filter by status, eval score, latency, annotations, or a natural-language query via AI Search. For example, status_code = 'ERROR' for exceptions, eval.groundedness.score < 0.5 for low-scoring spans, or “traces with hallucinations from yesterday”.

Select the spans you want and click Add to Dataset to create a new dataset or append to an existing one. Map at minimum the span’s input and output (stored under attributes.input.value and attributes.output.value); for classification tasks, also include a column with the expected label.

Upload a file. Go to Datasets & Experiments, click New Dataset, then choose Upload CSV and select a file you generated elsewhere.

Start with no datasets yet. Before your space contains a dataset, the Datasets & Experiments page offers three shortcuts. Start with Alyx generates a starter dataset, Create regression dataset from traces turns low-scoring interactions into a regression set, and Upload a dataset takes a CSV you already have.Add rows to an existing dataset. On a dataset, click New Examples to add rows: Add Manually for a row you type in, Select Traces to pull in spans from a project, Upload CSV, Add via Code, or Add via Agent (Skills) to work through the Arize skills in your coding agent.

Use the Arize SDK to create datasets programmatically. For the Python examples, install arize>=8.0.0 and set ARIZE_API_KEY and ARIZE_SPACE_ID in your environment.

Python

TS/JS

import os
from datetime import datetime, timedelta
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
space = os.environ["ARIZE_SPACE_ID"]

# From inline examples
client.datasets.create(
    name="support-qa-v1",
    space=space,
    examples=[
        {"input": "How do I cancel?", "expected_category": "account"},
        {"input": "I was charged twice.", "expected_category": "billing"},
    ],
)

# From traces (columns are OpenInference span attributes)
spans_df = client.spans.export_to_df(
    space_id=space,
    project_name="my-llm-app",
    start_time=datetime.now() - timedelta(days=30),
    end_time=datetime.now(),
)
client.datasets.create(name="traces-v1", space=space, examples=spans_df)

Optional: include prompt template metadata in each row

Continuing from the Python example above, if each row needs to carry the prompt template and its filled variables, store them on the OpenInference prompt-template columns so Playground and code can map them consistently:

Python SDK v8

import json
import pandas as pd

PROMPT_TEMPLATE = """
You are an expert in the history of technological inventions.
Identify the individual or organization that created the following invention.

Invention: {invention}
"""

prompt_rows = pd.DataFrame(
    [
        {
            "attributes.llm.prompt_template.template": PROMPT_TEMPLATE,
            "attributes.llm.prompt_template.variables": json.dumps(
                {"invention": "Telephone"}
            ),
            "attributes.output.value": "Alexander Graham Bell",
        }
    ]
)

client.datasets.create(
    name="prompt-invention-dataset",
    space=space,
    examples=prompt_rows,
)

If you’re migrating from Python SDK v7 dataset APIs, see the datasets client migration guide for create_dataset() and update_dataset() replacements.

​

Manage your dataset

• By Arize Skills
• By Alyx
• By UI
• By Code

Use the arize-dataset skill to append, export, or inspect datasets without leaving your editor. Try asking your agent:

• “Append the rows in new_examples.csv to my support-regression dataset.”
• “Export the latest version of my support-tickets dataset so I can review it offline.”
• “Show me the schema and the first five rows of my support-qa-v1 dataset.”

The Dataset Page Agent can append, annotate, or summarize datasets. Try:

• “Add the last 20 error spans to my regression dataset.”
• “Label the rows in this dataset with their expected category.”
• “Summarize how my regression dataset rows break down by failure type.”

Everything below happens on the dataset detail view, where you are working with a single dataset version at a time.To add a row, click + Example and fill in the fields inline.

To edit, open any row in the table and change values in place. That is the natural place to add or correct expected outputs so regression rows become golden rows. To remove rows from the latest version, select them and click Delete.When you want a file for offline review or to share outside AX, click Download as CSV on the dataset page.

Use the Arize SDK to append or export dataset rows.

Python

TS/JS

import os
from arize import ArizeClient

client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])

# Append rows — lands in the latest version in place.
# Pass `dataset_version_id` to target a specific version.
client.datasets.append_examples(
    dataset="YOUR_DATASET_ID",
    examples=[{"input": "...", "expected_category": "billing"}],
)

# Export examples for offline analysis. `all=True` fetches every row.
examples_df = client.datasets.list_examples(
    dataset="YOUR_DATASET_ID",
    all=True,
).to_df()

For the full reference, including get, delete, DataFrame input, and pagination, see the Python datasets client and TypeScript datasets client.

​

Auto-add rows from evaluator labels

​

Run your first experiment

​

Further reading

• Run evals on experiments: a walkthrough that replays a failing production trace and validates the fix across a dataset.
• View and manage traces: find spans worth turning into regression cases.
• Human review: turn reviewer feedback into dataset rows.
• Labeling queues: collect labels at scale before you build or update a golden dataset.
• Complete experiments notebook: an end-to-end dataset, evaluator, and comparison walkthrough.