Archived Competition Rules

1. Eligibility

• The competition was open to individuals and teams worldwide.
• There was no minimum or maximum team size, though 1–5 members were recommended.
• Participants could be affiliated with academic institutions, industry organizations, or be independent researchers.
• An individual could be a member of multiple teams. However, if two submissions from teams sharing a member were substantially overlapping, the worse-performing submission was excluded from award eligibility.

2. Registration

• Teams registered before their first submission to the official leaderboard.
• Registration was accepted through an official Google Form, with an email fallback where Google Forms was inaccessible.
• Registration required: team name, institution or organization, country or region, contact person name, contact email, team member names, optional team member emails, and track selection.
• Track 1 accepted new teams throughout the competition submission period.
• Track 2 was limited to 15 participating teams.
• Registration opened on May 25, 2026.
• Teams could update their team members up to 7 days before the final evaluation deadline.

3. Allowed Approaches

We adopt a permissive stance. Any approach is permitted unless explicitly prohibited below. This includes but is not limited to:

• Any LLM (proprietary or open-source, any size)
• Prompting strategies (system prompts, few-shot, chain-of-thought, self-reflection)
• Agentic scaffolding (planning-execution separation, retry logic, verification)
• Multi-agent systems (subagents, orchestration, ensemble methods)
• Fine-tuning on the provided training data (SFT, RLHF, DPO, etc.)
• Retrieval-augmented generation (RAG) over environment data
• External knowledge bases or pre-computed lookup tables (provided they do not encode test set answers)

Cerebras Fast-Reasoning Track: Track 2 submissions used the direct Cerebras-hosted gpt-oss inference setup described in the starter kit. Agents exposed the same dockerized A2A interface as Track 1. Multi-pass reasoning, private planning, self-verification, retries, ensembles, and parallel calls were allowed, but the agent had to respect Track 2 inference-compute constraints and could not exploit evaluator internals. For each baseline LLM step, participants could use up to 5 sequential LLM calls; parallel calls inside each step did not count toward this sequential limit. Token usage could be up to 500k tokens on average per task, including input, reasoning, and output tokens. Token usage was tracked through A2A turn_metrics token fields and could be counterchecked.

4. Prohibited Behaviors

The following are strictly prohibited and will result in disqualification:

• Hard-coding answers: Encoding specific test-set answers or lookup tables that map task descriptions to solutions.
• Exploiting evaluation infrastructure: Reverse-engineering, probing, or exploiting vulnerabilities in the evaluation system. This includes self-evaluating against the task-level metrics (e.g., checking subscores and re-prompting the LLM to correct failures before the official evaluation scores the run). Agents must produce genuine responses, not iteratively repair them against the scoring criteria.
• Probing the hidden test set: Any attempt to extract, infer, or reconstruct hidden test set tasks through the evaluation API or any other channel.
• Overlapping submissions: Teams sharing a member could not submit substantially overlapping agents. If detected, the worse-performing submission was excluded from award eligibility.

5. Submission Mechanics

• Submissions were dockerized A2A-compatible agents built from the starter kit interface.
• Participants provided a public GHCR image for the agent under test, pinned by digest.
• Participants provided scenario.toml, using the official organizer-published evaluator image and their agent-under-test runtime config.
• The official evaluator image could not be replaced, modified, self-hosted, or submitted as a participant artifact.
• Submission scenarios listed required and optional environment-variable or secret names, but never contained secret values.
• Official submission scenarios used task_split = "hidden" and -1 for each task-count field.
• All LLM model names, provider routes, deployment names, API bases, service tiers, and reasoning-effort selectors must be configurable through environment variables.
• Development validation: Participants could run public local, Docker, and GHCR scenario configs themselves before submitting.
• Hidden test set: One final official evaluation round ran on Jul 19. The planned Jul 10 score-return test round was cancelled.
• Award criteria: The Track 1 Rank Award was decided solely by hidden-set Pass^3. The Track 1 Innovation Award was judged on real-world deployment criteria including cost, latency, innovative methods, and performance relative to the LLM models used. The Track 2 Cerebras Award focused on compute-time-aware faster-inference methods, latency, and performance. See the archived criteria for details.

6. Code Release

• Participants are expected to release their code under a permissive open-source license (MIT, Apache 2.0, or equivalent) upon conclusion of the competition.
• Code must be sufficient for independent reproduction of reported results.

7. Data Use

• The starter kit and public CAR-bench data are released under the MIT License.
• Participants may use the training and dev data for other research, provided they cite the original paper.
• The hidden test set was not released during the competition. Any post-competition release will be announced separately.
• Results and rankings are publicly available on the results page.