Rules

1. Eligibility

• The competition is open to individuals and teams worldwide.
• There is no minimum or maximum team size, though we recommend 1–5 members.
• Participants may be affiliated with academic institutions, industry organizations, or be independent researchers.
• An individual may be a member of multiple teams. However, if two submissions from teams sharing a member are substantially overlapping, the worse-performing submission will be excluded from award eligibility.

2. Registration

• Teams must register before their first submission to the official leaderboard.
• Registration is preferably submitted through the official Google Form. If Google Forms is inaccessible from a participant’s location, teams may register by email using the fallback instructions on the registration page.
• Registration requires: team name, institution or organization, country or region, contact person name, contact email, team member names, optional team member emails, and track selection.
• Track 1 registration remains open.
• Track 2 registration is closed and remains limited to 15 participating teams.
• Registration opened on May 25, 2026.
• Teams may update their team members up to 7 days before the final evaluation deadline.

3. Allowed Approaches

We adopt a permissive stance. Any approach is permitted unless explicitly prohibited below. This includes but is not limited to:

• Any LLM (proprietary or open-source, any size)
• Prompting strategies (system prompts, few-shot, chain-of-thought, self-reflection)
• Agentic scaffolding (planning-execution separation, retry logic, verification)
• Multi-agent systems (subagents, orchestration, ensemble methods)
• Fine-tuning on the provided training data (SFT, RLHF, DPO, etc.)
• Retrieval-augmented generation (RAG) over environment data
• External knowledge bases or pre-computed lookup tables (provided they do not encode test set answers)

Cerebras Fast-Reasoning Track: Track 2 submissions must use the direct Cerebras-hosted gpt-oss inference setup described in the starter kit. Agents must expose the same dockerized A2A interface as Track 1. Multi-pass reasoning, private planning, self-verification, retries, ensembles, and parallel calls are allowed, but the agent must respect Track 2 inference-compute constraints and must not exploit evaluator internals. For each baseline LLM step, participants may use up to 5 sequential LLM calls; parallel calls inside each step do not count toward this sequential limit. Token usage may be up to 500k tokens on average per task, including input, reasoning, and output tokens. Token usage must be tracked through A2A turn_metrics token fields and may be counterchecked. Cerebras will provide increased rate limits compared with a free personal account; access details will follow soon.

4. Prohibited Behaviors

The following are strictly prohibited and will result in disqualification:

• Hard-coding answers: Encoding specific test-set answers or lookup tables that map task descriptions to solutions.
• Exploiting evaluation infrastructure: Reverse-engineering, probing, or exploiting vulnerabilities in the evaluation system. This includes self-evaluating against the task-level metrics (e.g., checking subscores and re-prompting the LLM to correct failures before the official evaluation scores the run). Agents must produce genuine responses, not iteratively repair them against the scoring criteria.
• Probing the hidden test set: Any attempt to extract, infer, or reconstruct hidden test set tasks through the evaluation API or any other channel.
• Overlapping submissions: Teams sharing a member may not submit substantially overlapping agents. If detected, the worse-performing submission will be excluded from award eligibility.

5. Submission Mechanics

• Submissions are dockerized A2A-compatible agents built from the starter kit interface.
• Participants submit a public GHCR image for the agent under test, pinned by digest.
• Participants submit scenario.toml, using the official organizer-published evaluator image and their agent-under-test runtime config.
• The official evaluator image must not be replaced, modified, self-hosted, or submitted as a participant artifact.
• Submission scenarios should list required and optional environment variable or secret names, but must never contain secret values.
• Official submission scenarios must use task_split = "hidden" and -1 for each task-count field.
• All LLM model names, provider routes, deployment names, API bases, service tiers, and reasoning-effort selectors must be configurable through environment variables.
• Development validation: Participants may run public local, Docker, and GHCR scenario configs themselves before submitting.
• Hidden test set: One final official evaluation round on Jul 19. The planned Jul 10 score-return test round will not run; organizers will contact teams if there are issues with their final submission.
• Award criteria: Track 1 Rank Award is decided solely by hidden-set Pass^3. Track 1 Innovation Award is judged on real-world deployment criteria including cost, latency, innovative methods, and performance relative to the LLM models used. Track 2 Cerebras Award focuses on compute-time-aware faster-inference methods, latency, and performance. See the submission page for details.

6. Code Release

• Participants are expected to release their code under a permissive open-source license (MIT, Apache 2.0, or equivalent) upon conclusion of the competition.
• Code must be sufficient for independent reproduction of reported results.

7. Data Use

• The starter kit and public CAR-bench data are released under the MIT License.
• Participants may use the training and dev data for other research, provided they cite the original paper.
• The hidden test set will not be released during the competition. Post-competition release will be announced separately.
• Results and rankings will be made publicly available on the competition website after the final evaluation.