The completed challenge used two tracks to explore complementary approaches to reliable LLM agents.
At a Glance
These were the model, compute, packaging, and evaluation conditions under which participants built their submissions.
| Dimension | Track 1: Open Track | Track 2: Cerebras Fast-Reasoning |
|---|---|---|
| Goal | Maximize Pass^3 on the hidden test set using any approach | Use direct Cerebras-hosted gpt-oss inference to convert fast inference into higher Pass^3 under Track 2 inference-compute constraints |
| Allowed Models | Any LLM, any provider (proprietary or open-source) | Direct Cerebras-hosted models through the starter kit, with optional self-hosted open-source development runs |
| Ranking / Awards | Rank Award decided solely by Pass^3, plus Innovation Award judged on real-world deployment quality | Cerebras Award judged on compute-time-aware faster-inference methods, latency, and performance |
| Suitable For | Teams exploring architecture, prompting, fine-tuning, multi-agent approaches | Teams interested in leveraging fast inference for reasoning-heavy strategies |
| Compute | Bring your own API keys for development. | Direct Cerebras-hosted gpt-oss inference, with optional self-hosted open-source development runs |
| Starter Package | src/track_1_agent_under_test/ |
src/track_2_agent_under_test_cerebras*/ |
| Evaluated artifact | A public digest-pinned GHCR agent image, scenario.toml using the official evaluator and hidden-set config, required environment-variable names, and a technical report. The exact requirements remain in the submission archive. |
|
Competition Workflow
Both tracks used the same agent interface. The difference was what each internal reasoning harness was allowed and expected to use.
| Step | What You Do |
|---|---|
| 1. Clone starter kit | Use the competition starter repository, run the setup script, and choose a Track 1 or Track 2 scenario folder. |
| 2. Implement the agent | Your agent receives A2A messages with user text, tool definitions, and tool results. It returns user-facing text and/or tool-call data. |
| 3. Validate locally | Run local_smoke.toml first, then public validation scenarios. Scenario TOML files are the run configs. |
| 4. Validate Docker/GHCR | Generate Docker Compose from a Docker scenario, then test the image/config style used by the organizers. |
| 5. Package for evaluation | Participants provided the public digest-pinned GHCR image, scenario.toml using the official evaluator and hidden-set config, required environment-variable names, and technical report. Secret values were never included. Review the archived requirements. |
The Open Track is designed for maximum flexibility: use any model, provider, framework, and architecture to build the most reliable agent possible. You can use Claude Agents SDK, LangGraph, AutoGen, custom orchestration, fine-tuning pipelines, retrieval systems, or any other harness that respects the benchmark boundary. The goal is to close the gap between frontier model capability and deployment-ready consistency on CAR-bench.
The organizer baseline achieved 58% consistent success (Pass^3), leaving room for improvements through better scaffolding, prompting, planning, and agent design.
A dockerized agent under test that communicates via the A2A (Agent-to-Agent) protocol with the CAR-bench evaluator. Your agent receives user requests, tool definitions, and tool results; it returns user-facing responses and/or tool calls. The evaluator executes tools and scores correctness, policy compliance, limit-awareness, and disambiguation behavior.
The Track 1 Rank Award is decided solely by Pass^3 on the hidden test set. A task scores 1 only if solved correctly in all 3 independent trials. Per-category breakdown (Base / Hallucination / Disambiguation) is shown alongside the aggregate score.
The Track 1 Innovation Award is judged on criteria important for real-world deployment: cost and token efficiency, caching, latency, innovative methods, and performance relative to the LLM models used. The overall product and research contribution matter more than raw performance alone.
Track 2 focuses on direct Cerebras gpt-oss inference for fast-reasoning agent harnesses. Fast inference makes it practical to explore more reasoning, verification, retry, and search work while keeping inference compute bounded.
This track explores how to turn that inference speed into higher Pass^3 scores. For each baseline LLM step, where the baseline directly decides on a tool call or final answer, participants may use up to 5 sequential LLM calls. Parallel calls within each step are allowed and do not count toward this sequential limit.
Token usage is limited to 500k tokens on average per task, including input, reasoning, and output tokens. As a reference, the baseline agent uses approximately 54k tokens on average per task. Token usage must be tracked through the A2A turn_metrics token fields and may be counterchecked.
Track 2 was limited to 15 participating teams.
The same dockerized A2A-compatible agent format as Track 1. The reference starter kit includes Cerebras SDK/direct API gpt-oss agents, including a direct executor and a planner/executor template. Participants may replace the internal planner, verifier, retry, or ensemble logic while preserving the same A2A boundary.
During development, participants could use Cerebras-hosted inference or self-host the open-source models used by the Cerebras gpt-oss executor, then validate the speed gain with Cerebras-hosted access.
gpt-oss inference setup described in the starter kitTrack 2 is judged with special focus on compute-time-aware development: innovative methods to leverage faster inference through harnesses, workflows, or architecture; latency; and performance. Cost is less central than in Track 1 because inference-time scaling is intentionally explored, but overall latency remains important. Evaluation is based on the required technical report alongside measured performance.
Use the starter kit to run the same public benchmark interface and review how the final submissions performed.