Submission Requirements Archive

Report Requirements

• Format: PDF using the IJCAI author kit.
• Length: up to 4 pages of main text; references are excluded from the page limit.
• Authorship: include the report title, ordered author names, and affiliations in the PDF. Reports are not anonymous.
• Citations: cite the ACL 2026 CAR-bench paper using the BibTeX below, as well as relevant methods, frameworks, models, datasets, and prior work used in the submission.
• One report per track: teams participating in both tracks submitted two separate reports.
• Track 2 visualization: include an agent-architecture diagram that makes the maximum sequential LLM-call depth, parallel fan-out, branches, and retry loops clear for each baseline agent step. A baseline agent step directly decides on a tool call or final answer. Sequential depth must not exceed 5; parallel calls within a step do not count toward that sequential limit.
• Track 2 compute: report assumptions needed to interpret token use and latency. Track 2 allows up to 500k input, reasoning, and output tokens on average per task.
• Results: deadline versions could report public-task scores but did not include the hidden-test-set score or ranking, which organizers returned after evaluation.
• Publication: submitters confirmed that all authors approved publication of selected reports as non-archival proceedings on this website.

After receiving the hidden-test-set score and ranking, teams could update the report to a final version before uploading it to arXiv.

Optional Report Structure

This outline is a non-mandatory completeness check. Teams may organize the report differently.

1. Problem and contribution overview.
2. Method and agent architecture.
3. Validation results, ablations, or lessons learned.
4. Efficiency and compute assumptions, limitations, and discussion.

CAR-bench Citation

Use the published ACL 2026 citation:

@inproceedings{kirmayr-etal-2026-car,
    title = "{CAR}-bench: Evaluating the Consistency and Limit-Awareness of {LLM} Agents under Real-World Uncertainty",
    author = "Kirmayr, Johannes  and
      Stappen, Lukas  and
      Andre, Elisabeth",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1886/",
    doi = "10.18653/v1/2026.acl-long.1886",
    pages = "40599--40618",
    ISBN = "979-8-89176-390-6"
}

Final Agent Submission Reference

The final-agent submission stage has concluded. The details below remain available as a reference for the artifacts used in official evaluation.

• Agent Docker image: a public GHCR image for your agent under test, pinned by digest.
• Scenario TOML: a scenario.toml file using the official evaluator image and your agent-under-test image/config.
• Environment variables: required and optional env var names only. Never submit secret values.
• Model configurability: all LLM model names, provider routes, deployment names, API bases, service tiers, and reasoning-effort selectors must be configurable through env vars.

The evaluator image was official and organizer-published. Participants did not submit, modify, or self-host evaluator images for official evaluation.

Awards and Scoring Criteria

Track 1: Open Track

• Rank Award: 1 winning team receives $1,000 in Anthropic API credits. The hidden-set Pass^3 score is the sole deciding factor.
• Innovation Award: 3 winning teams shared a $4,000 Anthropic API credits pool.
• Openstream.ai-sponsored prize: The Open Track winner receives $1,500.

The Innovation Award is judged on criteria important for real-world deployment: cost (token efficiency, caching), latency, innovative methods, and performance relative to the LLM models used. For this award, the overall product and research contribution matter more than raw performance alone.

Track 2: Cerebras Fast-Reasoning

• Cerebras Award: 2 winning teams receive a 12-month Codex Pro allocation.
• Openstream.ai-sponsored prize: The Cerebras Fast-Reasoning Track winner receives $1,500.

Track 2 is judged with special focus on compute-time-aware development: innovative ways to leverage faster inference through harnesses, workflows, or architecture; latency; and performance. Cost is less central in this track because inference-time scaling is intentionally explored, but overall latency remains important.

Copy-Paste Scenario TOML

This is retained as a reference for the concluded final-agent submission stage. The evaluator stayed the official organizer-published evaluator.

[evaluator]
image = "ghcr.io/car-bench/car-bench-evaluator:latest"

[evaluator.env]
GEMINI_API_KEY = "${GEMINI_API_KEY:?Set GEMINI_API_KEY}"
LOGURU_LEVEL = "${LOGURU_LEVEL:-INFO}"

[agent_under_test]
image = "ghcr.io/your-org/your-agent@sha256:replace_with_digest"

[agent_under_test.env]
AGENT_LLM = "${AGENT_LLM:?Set AGENT_LLM}"
AGENT_API_BASE = "${AGENT_API_BASE:-}"
AGENT_API_KEY = "${AGENT_API_KEY:?Set AGENT_API_KEY}"
AGENT_TEMPERATURE = "${AGENT_TEMPERATURE:-}"
LOGURU_LEVEL = "${LOGURU_LEVEL:-INFO}"

[config]
num_trials = 3
task_split = "hidden"
tasks_base_num_tasks = -1
tasks_hallucination_num_tasks = -1
tasks_disambiguation_num_tasks = -1
max_steps = 50

Field Meanings

• [evaluator]: used the official evaluator image. Participants did not submit, modify, or self-host evaluator images for official evaluation.
• [evaluator.env]: may reference evaluator env var names, but organizers provide evaluator secrets for official runs.
• [agent_under_test].image: public GHCR image pinned by digest, for example ghcr.io/team/agent@sha256:....
• [agent_under_test.env]: env vars organizers must set. Use ${VAR:?message} for required vars and ${VAR:-} or ${VAR:-default} for optional vars. Do not include secret values.
• [config]: use task_split = "hidden" and -1 for each task-count field so the full hidden set is selected.
• Name environment variables however you want, but every model/provider choice needed to run the agent must be configurable through env vars.

Final Agent Hosting Reference

Track 2 inference used Cerebras-hosted gpt-oss as described in the starter kit. Track 1 allowed configurable model and provider hosting. Submitted agents exposed relevant model, provider, deployment, API-base, service-tier, and reasoning-effort settings through environment variables where applicable.

Track 2 token usage must be reported through existing A2A turn_metrics fields: prompt_tokens, completion_tokens, and thinking_tokens. Do not add a custom sequential-depth metadata field; document the sequential-call structure in the report architecture diagram.