Frequently Asked Questions

In the Open Track, any LLM was allowed - proprietary, open-source, and any size. In the Cerebras Fast-Reasoning Track, submitted agents used direct Cerebras-hosted gpt-oss inference through the starter kit. Larger closed-source models could be used for internal planner, verifier, or condenser steps if the complete agent stayed within the Track 2 inference-compute constraints.

If your agent does not respond within the evaluator timeout, that trial is scored as a failure and evaluation continues with remaining tasks. Track 2 also has inference-compute constraints: up to 5 sequential LLM calls per baseline LLM step and token usage up to 500k input, reasoning, and output tokens on average per task.

The hidden test set is a completely novel set that is never shared with participants. Evaluation runs in a controlled environment. Agents cannot observe or extract test set tasks. Any attempt to probe the test set is a rule violation resulting in disqualification.

Post-competition release will be decided and announced separately. The public train and validation splits are available under the MIT License.

Hard-coding means encoding specific task answers or building lookup tables that map task descriptions to pre-determined solutions. This is prohibited. Clever prompting means crafting system prompts, few-shot examples, or reasoning strategies that help the agent generalize - this is encouraged. The litmus test: would your approach work on novel tasks it hasn’t seen? If yes, it’s prompting. If it only works on tasks whose answers you’ve memorized, it’s hard-coding.

Yes. Retrieval-augmented generation over the environment data (tools, policies, world data) is explicitly allowed and encouraged as a research direction.

Yes. You may fine-tune on the provided training data using any method (SFT, RLHF, DPO, etc.). You may also use the public validation split for model and harness selection. The public data is released under the MIT License.

Submitted technical reports are available through the results explorer. The requirements used for those reports remain in the submission archive.

The A2A (Agent-to-Agent) protocol is the structured communication format between the CAR-bench evaluator and your agent under test. Your agent receives user messages, tool definitions, and tool results. It sends back user-facing responses and/or tool calls. The evaluator is the only component that executes CAR-bench tools and computes scores. See the Data & Starter Kit page for the quickstart.

Scenario TOML files are run configs. They specify which evaluator and agent to start or pull, which task split to run, how many tasks and trials to execute, and which environment variables, Docker images, or GHCR images to use. Start with local_smoke.toml, then move to Docker and GHCR scenarios when reproducing the packaged competition setup.

Teams provided a public digest-pinned GHCR agent image, a scenario.toml configuration using the official evaluator and hidden-set settings, required environment-variable names without secret values, and a technical report. See the submission archive for the complete requirements.

Track 1 had a Rank Award decided solely by hidden-set Pass^3 and an Innovation Award judged on real-world deployment criteria such as cost, latency, methods, and performance relative to the LLM models used. Track 2 had a Cerebras Award focused on compute-time-aware faster-inference methods, latency, and performance. See the award winners and the archived criteria.

The Track 2 reference agents call Cerebras-hosted gpt-oss directly through the Cerebras SDK or direct API. Cerebras accounts can have strict limits, so use task scheduling or waiting between longer runs. Competition teams could self-host the open-source executor models during development before testing the speed gain on Cerebras-hosted inference.