Experiment configuration

Matched round-robin arena

Prepare data, inspect the design, then launch an immutable schedule.

1. Dataset

One record per model, task, and condition. All selected conditions must have distinct valid outputs for a task to be eligible.

Server metadata: data/arena-dataset.json. Local HTML paths are relative to data/sites/ and should use relative asset URLs.

Record format
{
  "model": "Qwen",
  "condition": "Original",
  "task_id": "task_017",
  "task_instruction": "Find a product and add it to the cart.",
  "url": "https://your-site.example",
  "category": "shopping",
  "metadata": {
    "website": "example-shop"
  }
}

Use html_path instead of url for local HTML. Set valid: false for failed outputs.

2. Sampling design

Six evaluators (E001–E006) share the schedule. Each comparison is assigned once. The seed fixes both the assignments and each evaluator’s order.

Models
Conditions

The random-pair design chooses an eligible website uniformly, then an eligible task within it uniformly. Duplicate website/task pairs are discarded and redrawn. MTA is excluded because it has no complete valid tasks.

Sessions & exports

No experiment launched yet.