CHINI-bench Methodology

1. Abstract

CHINI-bench measures whether large language models can design systems that survive specified failure modes. A model is given a natural-language brief plus a fixed set of stress scenarios, and emits a directed graph (CanvasState) describing the proposed architecture. The graph is executed by a deterministic discrete-event simulator under each scenario. Five sub-scores (stability, delivery, cost, constraints, design) are computed from simulator output and combined with problem-specific weights into a composite score in [0, 100]. No language model participates in grading.

As of v0.3 the bench contains 30 problems across 5 problem classes, totaling 112 scenarios. Four frontier models (Claude Sonnet 4.6, GPT-5.4, Grok 4.20, Gemini 3.1 Pro) cluster tightly at 69-70/100 average. The best (Claude Sonnet 4.6) passes only 5/30 problems; the lowest (Grok 4.20) passes 2/30. Across all four models combined, 10/30 problems have ever been solved by anyone.

2. Evaluation pipeline

1. 2.1The model receives the problem brief, constraints, success criteria, and the CanvasState schema as a single user message. The system prompt is fixed (section 12).
2. 2.2The model emits a JSON object conforming to the schema (section 3).
3. 2.3The submission is sanitized: unknown component types and behavior modes are dropped, edges with missing endpoints are removed, position fields are stripped (section 13).
4. 2.4For each of the problem's scenarios, the simulator runs the graph end-to-end and returns metrics (section 7).
5. 2.5Each scenario's metrics are checked against scenario-specific and problem-overall criteria. Failures are recorded but do not abort.
6. 2.6The five sub-scores are computed and combined (section 9). The result, including the full canvas and per-scenario metrics, is written to disk.
7. 2.7A submission passes iff every scenario passed AND no constraint notes were emitted. A passing submission is shown with a green badge on the per-problem and leaderboard pages.

The pipeline is implemented in src/lib/bench/scoring.ts as a pure function. Same input always yields the same output.

3. CanvasState input schema

Every submission is a single JSON object. Position and styling fields are optional and ignored by the simulator. The minimum viable submission has at least one component and at least one connection.

{
  "name": "string",                          // optional, free text
  "components": [
    {
      "id":          "string",               // unique within graph
      "type":        "person|step|storage|decision|trigger|tool|channel",
      "label":       "string",               // human-readable
      "description": "string",               // optional
      "behavior": {                          // optional
        "mode":         "passthrough|transform|filter|queue|split|delay|condition|retry|ratelimit|circuitbreaker|batch|replicate",
        "capacity":     0,                   // queue depth, rate cap, batch size
        "dropRate":     0.0,                 // 0-1, for queue overflow / filter
        "maxRetries":   0,                   // for retry mode
        "failureRate":  0.0,                 // 0-1, for circuitbreaker threshold
        "delayMs":      0,                   // for delay mode
        "splitRatio":   [0.5, 0.5]           // for split mode
      },
      "cost":    { "monthly": 0, "setup": 0 },     // optional, USD
      "metrics": { "throughput": "10k req/s",      // optional, free-text strings
                   "capacity":   "50k req/s",
                   "processingTime": "8ms" }
    }
  ],
  "connections": [
    {
      "id":         "string",                // unique within graph
      "from":       "componentId",
      "to":         "componentId",
      "label":      "string",                // optional
      "latency_ms": 0                        // optional
    }
  ]
}

Schema enforced by src/lib/bench/sanitize.ts. Unknown fields are silently stripped. Unknown enum values fall back to the default (step / passthrough).

4. Component types (7)

5. Behavior modes (12)

Behaviors attach to a component and modify how it processes packets. Most have parameters; sensible defaults apply if a parameter is omitted.

6. Scenario kinds (5)

A problem comprises a baseline plus 1-3 stress scenarios. The simulator runs each scenario independently against the same submitted canvas. A submission passes the problem only if it passes every scenario.

7. Per-scenario metrics

The simulator emits the following measurements per scenario. Adversarial scenarios additionally emit attackBlockRate and cleanDeliveryRate (section 11).

• healthScore (0-100). Internal aggregate of queue pressure, drop count, and throughput. Computed by aggregateFlowStats.
• delivered. Raw count of packets that reached a primary exit (a sink component).
• dropped. Raw count of packets dropped along the way.
• deliveryRate. delivered / max(injected, 1), clamped to [0, 1].
• dropRate. dropped / max(injected + delivered, 1), clamped to [0, 1].
• durationMs. Simulated wall-clock time for the scenario to complete.
• errorCount. Distinct error events emitted by the engine (cycles, missing endpoints, runtime exceptions).

8. Default success thresholds

Each scenario's metrics are checked against criteria computed by merging three layers (later wins):

1. Default thresholds (below)
2. Problem-level overallCriteria (per-problem, applies to all scenarios)
3. Scenario-level scenarioCriteria[id] (per-scenario override)

Adversarial scenarios additionally enforce minAttackBlockRate=0.7 and minCleanDeliveryRate=0.8 by default.

9. Composite scoring

Each problem defines five sub-scores in [0, 100] and weights them into a single composite, also in [0, 100]:

• stability. Mean healthScore across all scenarios.
• delivery. Mean deliveryRate x 100 across all scenarios.
• cost. round(100 x min(1, budget / max(1, totalMonthly))). 100 if no budget set.
• constraints. Starts at 100. Subtract 10 per excess component (cap 40). Subtract 25 per missing required behavior. Subtract up to 40 for budget overshoot.
• design. Sum of 4 conditional structural checks worth 25 each (section 10).

Weights are problem-specific and sum to 1.0. A typical PC1 problem weights stability at 0.40, delivery at 0.25, design at 0.15, cost at 0.10, constraints at 0.10. PC5 problems shift weight onto delivery and design, away from cost.

10. The design subscore (D1-D4)

Stability and delivery measure outcomes. Design measures structure: did the model include the right primitives for the failure modes the problem actually exercises? Each check is worth 25 points and is conditional, awarded full credit if the corresponding failure mode is not in the scenario set.

D4 is the most-failed check in the v0.3 sweep. Several model submissions returned fully-cyclic graphs where every component had outgoing edges, meaning no packet could ever exit. The simulator runs them anyway and they tank delivery; D4 isolates the root cause structurally.

11. Adversarial scoring

A pure-throughput optimizer would happily forward attack traffic to its target and score perfectly on deliveryRate. To prevent this, adversarial scenarios run two passes through the simulator:

1. Pass A (clean): seedPackets only, no attack volume, no inflated failure rate. Produces cleanDelivered.
2. Pass B (under attack): seedPackets + attackPackets injected at the same trigger, ambient failure rate raised to model hostile inputs flaking downstream. Produces totalDeliveredUnderAttack.

Two derived metrics:

A defense that drops everything scores 1.0 on attack-block but ~0 on clean-delivery (and fails). A pure throughput optimizer scores ~1.0 on clean-delivery but ~0 on attack-block (and fails). Both must clear their thresholds (default 0.7 and 0.8). This is the only place the bench has explicit dual-objective scoring; everywhere else, a single composite suffices.

12. Default harness

The CLI ships a single fixed harness used for every published model run:

• System prompt: names the CanvasState schema, lists the 7 component types and 12 behavior modes, instructs the model to emit one valid JSON object and nothing else. Frozen, lives in chini_bench/prompt.py.
• User prompt: the problem brief, constraints, success criteria, and scenario list, exactly as published.
• Sampling: temperature 0.2, top-p default, no system messages besides the harness, no chain-of-thought scaffolding, no retries on bad JSON. max_tokens=12000 for OpenRouter routes (raised from 3000 in v0.3 to accommodate thinking models).

Harness verification. Every chini-bench run command computes a SHA-256 of the system prompt and sends the first 12 hex characters along with the submission as harness=chini-bench-cli:<hash>. The server stores it in the result file. The leaderboard then renders one of three states:

• defaultHash matches the canonical bench-version hash. Unmodified CLI, default prompt, no scaffolding.
• customHash differs. Either someone forked the CLI and changed the prompt, or they re-ran via a different runner that declares its own harness id. The leaderboard surfaces the run but flags it.
• no badgeSubmission came in via chini-bench submit -f file.json or directly via the HTTP endpoint. No harness was declared. Treated as community contribution, not a calibrated frontier-model result.

Beating the default harness with a better prompt is itself a useful research result, as long as the prompt is published. We treat this as transparent harness-engineering, not cheating, and the custom badge keeps the two categories cleanly separated on the leaderboard.

13. Sanitization & abuse protection

The submission endpoint POST /api/bench/submit applies the following before scoring:

• Body cap. 64 KB. Larger bodies return 413.
• Rate limit. 5 submissions per IP per 10-minute rolling window.
• Honeypot. A hidden form field rejects naive bots without surfacing why.
• Submitter validation. Letters, digits, dot, dash, underscore. 1-40 characters. Auto-prefixed with community: to prevent impersonation of official model identifiers (e.g. x-ai/grok-4.20).
• Content filter. Submitter strings are checked against a categorized blocklist. Rejections surface a generic message; the category is logged for triage.
• Optional metadata. model (regex-validated), x handle (1-15 chars, no @), linkedin slug (full URL accepted, slug extracted).
• Canvas sanitization. Unknown component types fall back to step. Unknown behavior modes fall back to passthrough. Connections referencing missing components are dropped. Position, color, and styling fields are stripped before storage.

Implemented in src/pages/api/bench/submit.ts and src/lib/bench/sanitize.ts.

14. Reproducibility

Determinism: the simulator (src/lib/flowRuntime.ts) and the scorer (src/lib/bench/scoring.ts) are pure functions. Same canvas + same problem definition always yields the same composite score and per-scenario metrics, byte-for-byte. The 120 result JSONs in src/content/bench/results/ can be regenerated from the published canvases by re-running the scorer.

Independence: every problem definition, scenario, and weight lives in src/content/bench/problems/ as plain JSON. Every result JSON includes the full submitted canvas (a) for inspection and (b) so any reader can re-score it locally and verify the published number.

The repository is currently private. Researchers who want to verify a result independently can request the source bundle by emailing squeak@chinilla.com. A public mirror under a research-use license is planned alongside the v1 leaderboard.

15. Limitations

• Construct validity. The simulator is a fair model of the failure modes it explicitly models (queue overflow, retry storms, circuit-breaker logic, rate limits, ambient failure). It is not a model of real-world cloud infrastructure. Beating CHINI-bench is necessary but not sufficient evidence of system-design competence.
• Technology-blindness. A "queue" component is not Kafka or SQS or a Redis list, just a queue with capacity and drop semantics. Models that reason at the level of named services have no advantage over models that reason at the level of behaviors.
• No human baseline yet. v0.3 reports model scores only. A controlled baseline study with hired senior engineers is planned for v1.
• Adversarial scenarios are stylized. The two-pass attack model is an abstraction. Real adversarial inputs (prompt injection, malicious payloads) are not directly tested; the bench tests whether a design has the structural primitives that would resist attack.
• Problem authorship is centralized. All 30 problems were authored by one person (the author). A community problem-submission process is planned for v1 to mitigate authorial blind spots.
• One-shot, not agentic. Models emit one CanvasState per problem. They cannot inspect simulator output and revise. This is a deliberate design choice, not an oversight: one-shot isolates design intuition under uncertainty from iterative search given a feedback loop, which are separable skills that deserve separable benchmarks. A multi-turn agentic track (model sees sim metrics, edits canvas, re-runs) is planned as a parallel leaderboard so the two signals are reported side-by-side, never conflated. Until then, v0.3 scores understate the capability of models given an agentic harness.
• Sample size. 30 problems, 4 models, 120 results. Adequate for the headline finding (no model passes >9/30) but not for fine-grained model-vs-model claims at the <5pt level.

We document these limitations because a benchmark that hides its weaknesses is not a benchmark, it is marketing.

16. Ethics & broader impact

Intended use. CHINI-bench is intended to surface concrete weaknesses in current frontier models for system-design tasks, particularly for safety-relevant deployments (PC4 civic systems, PC5 adversarial). Improvements here translate, in principle, to safer agentic systems in production.

Misuse potential. Public benchmark scores can be optimized against rather than learned from. We mitigate via (a) a deterministic non-LLM judge that cannot be flattered, (b) cross-class composite scoring that punishes overfitting to PC1, and (c) versioning so a model that overfits v0.3 will not transfer to v1.

Data & privacy. The benchmark contains no personal data. Submissions store only the canvas, the submitter handle, and an SHA-256 hash of the source IP (for rate-limiting and abuse triage; never reversed, never displayed).

Carbon & cost. A full v0.3 sweep is 120 LLM calls (~$4 in API spend across the 4 frontier models, ~1 hour wall-clock including retries on rate limits and malformed JSON). The simulator runs locally and uses negligible energy.

Appendix A: problem set (v0.3)

Appendix B: version history

References

1. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arXiv:2303.11366.
2. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770.

Citation

If you reference CHINI-bench in academic or industry work, please cite it as:

@misc{chinibench2026,
  title  = {{CHINI-bench}: A simulator-graded benchmark for {AI} system design},
  author = {Kwon, Alex},
  year   = {2026},
  note   = {Version 0.3. https://chinilla.com/bench},
  url    = {https://chinilla.com/bench}
}

Plain text:

Kwon, A. (2026). CHINI-bench: A simulator-graded benchmark for AI system design (Version 0.3) [Benchmark]. https://chinilla.com/bench