Technical Report Submission

Report Requirements

• Format: PDF using the IJCAI author kit.
• Length: up to 4 pages of main text; references are excluded from the page limit.
• Authorship: include the report title, ordered author names, and affiliations in the PDF. Reports are not anonymous.
• Citations: cite the ACL 2026 CAR-bench paper using the BibTeX below, as well as relevant methods, frameworks, models, datasets, and prior work used in the submission.
• One report per track: a team participating in both tracks submits two separate reports and two form responses.
• Track 2 visualization: include an agent-architecture diagram that makes the maximum sequential LLM-call depth, parallel fan-out, branches, and retry loops clear for each baseline agent step. A baseline agent step directly decides on a tool call or final answer. Sequential depth must not exceed 5; parallel calls within a step do not count toward that sequential limit.
• Track 2 compute: report assumptions needed to interpret token use and latency. Track 2 allows up to 500k input, reasoning, and output tokens on average per task.
• Results: scores on public tasks may be reported. Submit the deadline version without a hidden-test-set score or ranking; organizers will return the hidden-test-set score and ranking after evaluation.
• Publication: selected reports will be published as non-archival proceedings on this website. The submitter must confirm that all authors approve publication if selected.

After receiving the hidden-test-set score and ranking, teams may update the report to a final version before uploading it to arXiv. arXiv submission is encouraged and endorsed.

How to Submit

1. Open the technical report form and select one track.
2. Enter the team name and contact email, then upload one PDF of at most 10 MB.
3. Complete the formatting, citation, authorship, and publication confirmations. Track 2 submissions also complete the architecture-visualization confirmation.
4. Submit and retain the edit-response link. You may replace the PDF through that link until the deadline.

Open Technical Report Form

Email Fallback

If Google Forms or its file-upload sign-in is inaccessible, email the PDF to johannes1.kirmayr@uni-a.de by the same deadline. Use one email per track and include the confirmation shown below.

Prepare Fallback Email

Subject: CAR-bench Technical Report – <Team Name> – <Track>

Team name:
Track:
Contact email:

I confirm that all authors approve publication of this report on the CAR-bench competition website if it is selected.

Attachment: one technical-report PDF for the selected track

Optional Report Structure

This outline is a non-mandatory completeness check. Teams may organize the report differently.

1. Problem and contribution overview.
2. Method and agent architecture.
3. Validation results, ablations, or lessons learned.
4. Efficiency and compute assumptions, limitations, and discussion.

CAR-bench Citation

Use the published ACL 2026 citation:

@inproceedings{kirmayr-etal-2026-car,
    title = "{CAR}-bench: Evaluating the Consistency and Limit-Awareness of {LLM} Agents under Real-World Uncertainty",
    author = "Kirmayr, Johannes  and
      Stappen, Lukas  and
      Andre, Elisabeth",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1886/",
    doi = "10.18653/v1/2026.acl-long.1886",
    pages = "40599--40618",
    ISBN = "979-8-89176-390-6"
}

Final Agent Submission Reference

The final-agent submission stage has concluded. The details below remain available as a reference for the artifacts used in official evaluation.

• Agent Docker image: a public GHCR image for your agent under test, pinned by digest.
• Scenario TOML: a scenario.toml file using the official evaluator image and your agent-under-test image/config.
• Environment variables: required and optional env var names only. Never submit secret values.
• Model configurability: all LLM model names, provider routes, deployment names, API bases, service tiers, and reasoning-effort selectors must be configurable through env vars.

The evaluator image is official and organizer-published. Participants do not submit, modify, or self-host evaluator images for official evaluation.

Awards and Scoring Criteria

Track 1: Open Track

• Rank Award: 1 winning team receives $1,000 in Anthropic API credits. The hidden-set Pass^3 score is the sole deciding factor.
• Innovation Award: estimated 2-3 winning teams share a $4,000 Anthropic API credits pool. The final number of winners and distribution will be decided based on submission quality and ordering.

The Innovation Award is judged on criteria important for real-world deployment: cost (token efficiency, caching), latency, innovative methods, and performance relative to the LLM models used. For this award, the overall product and research contribution matter more than raw performance alone.

Track 2: Cerebras Fast-Reasoning

• Cerebras Award: 2 winning teams receive a 12-month Codex Pro allocation.

Track 2 is judged with special focus on compute-time-aware development: innovative ways to leverage faster inference through harnesses, workflows, or architecture; latency; and performance. Cost is less central in this track because inference-time scaling is intentionally explored, but overall latency remains important.

Copy-Paste Scenario TOML

This is retained as a reference for the concluded final-agent submission stage. The evaluator stayed the official organizer-published evaluator.

[evaluator]
image = "ghcr.io/car-bench/car-bench-evaluator:latest"

[evaluator.env]
GEMINI_API_KEY = "${GEMINI_API_KEY:?Set GEMINI_API_KEY}"
LOGURU_LEVEL = "${LOGURU_LEVEL:-INFO}"

[agent_under_test]
image = "ghcr.io/your-org/your-agent@sha256:replace_with_digest"

[agent_under_test.env]
AGENT_LLM = "${AGENT_LLM:?Set AGENT_LLM}"
AGENT_API_BASE = "${AGENT_API_BASE:-}"
AGENT_API_KEY = "${AGENT_API_KEY:?Set AGENT_API_KEY}"
AGENT_TEMPERATURE = "${AGENT_TEMPERATURE:-}"
LOGURU_LEVEL = "${LOGURU_LEVEL:-INFO}"

[config]
num_trials = 3
task_split = "hidden"
tasks_base_num_tasks = -1
tasks_hallucination_num_tasks = -1
tasks_disambiguation_num_tasks = -1
max_steps = 50

Field Meanings

• [evaluator]: must use the official evaluator image. Participants do not submit, modify, or self-host evaluator images for official evaluation.
• [evaluator.env]: may reference evaluator env var names, but organizers provide evaluator secrets for official runs.
• [agent_under_test].image: public GHCR image pinned by digest, for example ghcr.io/team/agent@sha256:....
• [agent_under_test.env]: env vars organizers must set. Use ${VAR:?message} for required vars and ${VAR:-} or ${VAR:-default} for optional vars. Do not include secret values.
• [config]: use task_split = "hidden" and -1 for each task-count field so the full hidden set is selected.
• Name environment variables however you want, but every model/provider choice needed to run the agent must be configurable through env vars.

Final Agent Hosting Reference

Track 2 inference is Cerebras-hosted gpt-oss as described in the starter kit. Track 1 hosting details will follow soon. Participants should still expose relevant model, provider, deployment, API-base, service-tier, and reasoning-effort settings through environment variables where applicable.

Track 2 token usage must be reported through existing A2A turn_metrics fields: prompt_tokens, completion_tokens, and thinking_tokens. Do not add a custom sequential-depth metadata field; document the sequential-call structure in the report architecture diagram.