CHINI-bench Methodology

1. Abstract

CHINI-bench measures whether large language models can design systems that survive specified failure modes. A model is given a natural-language brief plus a fixed set of stress scenarios, and emits a directed graph (CanvasState) describing the proposed architecture. The graph is executed by a deterministic discrete-event simulator under each scenario. Five sub-scores (stability, delivery, cost, constraints, design) are computed from simulator output and combined with problem-specific weights into a composite score in [0, 100]. No language model participates in grading.

As of v0.3 the bench contains 450 problems across 5 problem classes, totaling 1539 scenarios. Four frontier models (Claude Sonnet 4.6, GPT-5.4, Grok 4.20, Gemini 3.1 Pro) cluster tightly at 69-70/100 average. The best (Claude Sonnet 4.6) passes only 5/30 problems; the lowest (Grok 4.20) passes 2/30. Across all four models combined, 10/30 problems have ever been solved by anyone.

2. Evaluation pipeline

1. 2.1The model receives the problem brief, constraints, success criteria, and the CanvasState schema as a single user message. The system prompt is fixed (section 12).
2. 2.2The model emits a JSON object conforming to the schema (section 3).
3. 2.3The submission is sanitized: unknown component types and behavior modes are dropped, edges with missing endpoints are removed, position fields are stripped (section 13).
4. 2.4For each of the problem's scenarios, the simulator runs the graph end-to-end and returns metrics (section 7).
5. 2.5Each scenario's metrics are checked against scenario-specific and problem-overall criteria. Failures are recorded but do not abort.
6. 2.6The five sub-scores are computed and combined (section 9). The result, including the full canvas and per-scenario metrics, is written to disk.
7. 2.7A submission passes iff every scenario passed AND no constraint notes were emitted. A passing submission is shown with a green badge on the per-problem and leaderboard pages.

The pipeline is implemented in src/lib/bench/scoring.ts as a pure function. Same input always yields the same output.

3. CanvasState input schema

Every submission is a single JSON object. Position and styling fields are optional and ignored by the simulator. The minimum viable submission has at least one component and at least one connection.

{
  "name": "string",                          // optional, free text
  "components": [
    {
      "id":          "string",               // unique within graph
      "type":        "person|step|storage|decision|trigger|tool|channel",
      "label":       "string",               // human-readable
      "description": "string",               // optional
      "behavior": {                          // optional
        "mode":         "passthrough|transform|filter|queue|split|delay|condition|retry|ratelimit|circuitbreaker|batch|replicate",
        "capacity":     0,                   // queue depth, rate cap, batch size
        "dropRate":     0.0,                 // 0-1, for queue overflow / filter
        "maxRetries":   0,                   // for retry mode
        "failureRate":  0.0,                 // 0-1, for circuitbreaker threshold
        "delayMs":      0,                   // for delay mode
        "splitRatio":   [0.5, 0.5]           // for split mode
      },
      "cost":    { "monthly": 0, "setup": 0 },     // optional, USD
      "metrics": { "throughput": "10k req/s",      // optional, free-text strings
                   "capacity":   "50k req/s",
                   "processingTime": "8ms" }
    }
  ],
  "connections": [
    {
      "id":         "string",                // unique within graph
      "from":       "componentId",
      "to":         "componentId",
      "label":      "string",                // optional
      "latency_ms": 0                        // optional
    }
  ]
}

Schema enforced by src/lib/bench/sanitize.ts. Unknown fields are silently stripped. Unknown enum values fall back to the default (step / passthrough).

4. Component types (7)

5. Behavior modes (11)

Behaviors attach to a component and modify how it processes packets. Most have parameters; sensible defaults apply if a parameter is omitted.

6. Scenario kinds (5)

A problem comprises a baseline plus 1-3 stress scenarios. The simulator runs each scenario independently against the same submitted canvas. A submission passes the problem only if it passes every scenario.

7. Per-scenario metrics

The simulator emits the following measurements per scenario. Adversarial scenarios additionally emit attackBlockRate and cleanDeliveryRate (section 11).

• healthScore (0-100). Internal aggregate of queue pressure, drop count, and throughput. Computed by aggregateFlowStats.
• delivered. Raw count of packets that reached a primary exit (a sink component).
• dropped. Raw count of packets dropped along the way.
• deliveryRate. delivered / max(injected, 1), clamped to [0, 1].
• dropRate. dropped / max(injected + delivered, 1), clamped to [0, 1].
• durationMs. Simulated wall-clock time for the scenario to complete.
• errorCount. Distinct error events emitted by the engine (cycles, missing endpoints, runtime exceptions).

8. Default success thresholds

Each scenario's metrics are checked against criteria computed by merging three layers (later wins):

1. Default thresholds (below)
2. Problem-level overallCriteria (per-problem, applies to all scenarios)
3. Scenario-level scenarioCriteria[id] (per-scenario override)

Adversarial scenarios additionally enforce minAttackBlockRate=0.7 and minCleanDeliveryRate=0.8 by default.

9. Composite scoring

Each problem defines five sub-scores in [0, 100] and weights them into a single composite, also in [0, 100]:

• stability. Mean healthScore across all scenarios.
• delivery. Mean deliveryRate x 100 across all scenarios.
• cost. round(100 x min(1, budget / max(1, totalMonthly))). 100 if no budget set.
• constraints. Starts at 100. Subtract 10 per excess component (cap 40). Subtract 25 per missing required behavior. Subtract up to 40 for budget overshoot.
• design. Sum of 4 conditional structural checks worth 25 each (section 10).

Weights are problem-specific and sum to 1.0. A typical PC1 problem weights stability at 0.40, delivery at 0.25, design at 0.15, cost at 0.10, constraints at 0.10. PC5 problems shift weight onto delivery and design, away from cost.

10. The design subscore (D1-D4)

Stability and delivery measure outcomes. Design measures structure: did the model include the right primitives for the failure modes the problem actually exercises, and did it put them where the stress actually flows? Each check is worth 25 points and is conditional, awarded full credit if the corresponding failure mode is not in the scenario set.

As of v0.7, D1-D3 split into two halves: 10 points for presence (the primitive exists somewhere in the graph) and 15 points for placement (the primitive sits on a source-to-sink forward path; for D2 with a named outage target, upstream of that target). This closes a v0.6-and-earlier gap where a queue parked off to the side of the actual flow earned the same credit as a queue inline between trigger and storage. D4 remains binary 25/0 because a graph either has a terminal sink or it does not.

Source / sink definitions. A source is a trigger-typed component whenever the canvas contains any; this pins placement scoring to the components the simulator actually injects packets into and prevents a model from earning placement credit by attaching a disconnected dummy-source → queue → dummy-sink subgraph. When the canvas has no triggers (free-form designs), source falls back to "any component with no inbound forward edges". A sink is any component with no outbound forward edges. Forward direction respects each Connection.direction value (forward / backward / both / none). An orphan node (no edges at all) is excluded from placement credit so it cannot trivially count as both endpoint of a zero-length path.

Known limits of v0.7 placement scoring. Placement is binary "on a source-to-sink forward path", not "effective at the placed point". A primitive on the path earns full +15 even if it is undersized (1-capacity queue under a 5x spike), placed too late (queue downstream of the fragile chain it should have been protecting), or only protects one of several stressed branches. Outcome metrics (stability, delivery) still punish those cases at the simulator level, so they are score distortions rather than leaderboard-breaking exploits, but the design subscore alone will overstate competence in those edge cases. Future work: capacity-vs-load sanity gates for D1 (queue capacity must be a non-trivial fraction of expected spike traffic), upstream-of-fragile-chain placement for D1/D3 (currently only D2 enforces upstream of a named target), and coverage scoring (multiple buffers / distributed retries) under "defense in depth" problems.

D4 was the most-failed check in the v0.3 sweep and remains a frequent failure: several model submissions return fully-cyclic graphs where every component has outgoing edges, meaning no packet can ever exit. The simulator runs them anyway and they tank delivery; D4 isolates the root cause structurally. The v0.7 placement split was added in response to a public review pointing out that pre-v0.7 D1-D3 rewarded primitive presence anywhere in the graph rather than primitive placement on the stressed path; under the new scheme, an off-flow queue earns 10 instead of 25.

11. Adversarial scoring

A pure-throughput optimizer would happily forward attack traffic to its target and score perfectly on deliveryRate. To prevent this, adversarial scenarios run two passes through the simulator:

1. Pass A (clean): seedPackets only, no attack volume, no inflated failure rate. Produces cleanDelivered.
2. Pass B (under attack): seedPackets + attackPackets injected at the same trigger, ambient failure rate raised to model hostile inputs flaking downstream. Produces totalDeliveredUnderAttack.

Two derived metrics:

A defense that drops everything scores 1.0 on attack-block but ~0 on clean-delivery (and fails). A pure throughput optimizer scores ~1.0 on clean-delivery but ~0 on attack-block (and fails). Both must clear their thresholds (default 0.7 and 0.8). This is the only place the bench has explicit dual-objective scoring; everywhere else, a single composite suffices.

12. Default harness

The CLI ships a single fixed harness used for every published model run:

• System prompt: names the CanvasState schema, lists the 7 component types and 8 behavior modes, instructs the model to emit one valid JSON object and nothing else. Frozen, lives in chini_bench/prompt.py.
• User prompt: the problem brief, constraints, success criteria, and scenario list, exactly as published.
• Sampling: temperature 0.2, top-p default, no system messages besides the harness, no chain-of-thought scaffolding, no retries on bad JSON. max_tokens=12000 for OpenRouter routes (raised from 3000 in v0.3 to accommodate thinking models).

Harness verification. Every chini-bench run command computes a SHA-256 of the system prompt and sends the first 12 hex characters along with the submission as harness=chini-bench-cli:<hash>. The server stores it in the result file. The leaderboard then renders one of three states:

• defaultHash matches the canonical bench-version hash. Unmodified CLI, default prompt, no scaffolding.
• customHash differs. Either someone forked the CLI and changed the prompt, or they re-ran via a different runner that declares its own harness id. The leaderboard surfaces the run but flags it.
• no badgeSubmission came in via chini-bench submit -f file.json or directly via the HTTP endpoint. No harness was declared. Treated as community contribution, not a calibrated frontier-model result.

Beating the default harness with a better prompt is itself a useful research result, as long as the prompt is published. We treat this as transparent harness-engineering, not cheating, and the custom badge keeps the two categories cleanly separated on the leaderboard.

12.5 Reflexion track (multi-turn agentic)

The multi-turn (agentic) track answers a different question than single-shot: given simulator feedback from a failed first attempt, can a model fix its own architecture in one revision? Same model, same problem, two attempts. Server runs the simulator on the v1 canvas and returns a redacted FeedbackPacket; the model writes v2; the server scores v2. The technique is Reflexion-style verbal self-reflection, capped at exactly one revision.

Localization rules. The model sees scenario names, per-metric {observed, target, delta} triples, failed structural-check ids (D1-D4), missing canonical behavior ids, observation phrases from a controlled vocabulary, and budget overspend. The model never sees the composite score, sub-scores, the passing threshold, or any other submission. Free-form natural-language advice from the simulator is forbidden.

One revision, no tools. N-shot loops let weak models brute-force their way to a passing canvas, which conflates "model is smart" with "model has compute budget." The track is hard-capped at one revision. The model cannot probe the simulator with extra requests.

Headline metric. Pass after revision: percentage of runs where v2 passed every scenario and structural check. Tie-breakers: structural fix rate, then v1→v2 delta, then fewer tokens (cheaper wins).

Audit trail. Every multi-turn result file stores the v1 canvas, v1 pass flag, the FeedbackPacket the model saw, the v2 canvas, and an estimated tokensTotal. Anyone can re-run the simulator on v1Canvas and re-derive the FeedbackPacket; no part of the pipeline is sealed.

Full design and schema: docs/agentic-track.md.

13. Sanitization & abuse protection

The submission endpoint POST /api/bench/submit applies the following before scoring:

• Body cap. 64 KB. Larger bodies return 413.
• Rate limit. 5 submissions per IP per 10-minute rolling window.
• Honeypot. A hidden form field rejects naive bots without surfacing why.
• Submitter validation. Letters, digits, dot, dash, underscore. 1-40 characters. Auto-prefixed with community: to prevent impersonation of official model identifiers (e.g. x-ai/grok-4.20).
• Content filter. Submitter strings are checked against a categorized blocklist. Rejections surface a generic message; the category is logged for triage.
• Optional metadata. model is regex-validated against the standard provider id charset.
• Canvas sanitization. Unknown component types fall back to step. Unknown behavior modes fall back to passthrough. Connections referencing missing components are dropped. Position, color, and styling fields are stripped before storage.

Implemented in src/pages/api/bench/submit.ts and src/lib/bench/sanitize.ts.

14. Reproducibility

Determinism: the simulator (src/lib/flowRuntime.ts) and the scorer (src/lib/bench/scoring.ts) are pure functions. Same canvas + same problem definition always yields the same composite score and per-scenario metrics, byte-for-byte. The 120 result JSONs in src/content/bench/results/ can be regenerated from the published canvases by re-running the scorer.

Independence: every problem definition, scenario, and weight lives in src/content/bench/problems/ as plain JSON. Every result JSON includes the full submitted canvas (a) for inspection and (b) so any reader can re-score it locally and verify the published number.

The repository is currently private. Researchers who want to verify a result independently can request the source bundle by emailing squeak@chinilla.com. A public mirror under a research-use license is planned alongside the v1 leaderboard.

15. Limitations

• Construct validity. The simulator is a fair model of the failure modes it explicitly models (queue overflow, retry storms, circuit-breaker logic, rate limits, ambient failure). It is not a model of real-world cloud infrastructure. Beating CHINI-bench is necessary but not sufficient evidence of system-design competence.
• Technology-blindness. A "queue" component is not Kafka or SQS or a Redis list, just a queue with capacity and drop semantics. Models that reason at the level of named services have no advantage over models that reason at the level of behaviors.
• No human baseline yet. v0.3 reports model scores only. A controlled baseline study with hired senior engineers is planned for v1.
• Adversarial scenarios are stylized. The two-pass attack model is an abstraction. Real adversarial inputs (prompt injection, malicious payloads) are not directly tested; the bench tests whether a design has the structural primitives that would resist attack.
• Problem authorship is centralized. All 30 problems were authored by one person (the author). A community problem-submission process is planned for v1 to mitigate authorial blind spots.
• One-shot, not agentic. Models emit one CanvasState per problem. They cannot inspect simulator output and revise. This is a deliberate design choice, not an oversight: one-shot isolates design intuition under uncertainty from iterative search given a feedback loop, which are separable skills that deserve separable benchmarks. A multi-turn agentic track (model sees sim metrics, edits canvas, re-runs) is planned as a parallel leaderboard so the two signals are reported side-by-side, never conflated. Until then, v0.3 scores understate the capability of models given an agentic harness.
• Sample size. 30 problems, 4 models, 120 results. Adequate for the headline finding (no model passes >9/30) but not for fine-grained model-vs-model claims at the <5pt level.

We document these limitations because a benchmark that hides its weaknesses is not a benchmark, it is marketing.

16. Ethics & broader impact

Intended use. CHINI-bench is intended to surface concrete weaknesses in current frontier models for system-design tasks, particularly for safety-relevant deployments (PC4 civic systems, PC5 adversarial). Improvements here translate, in principle, to safer agentic systems in production.

Misuse potential. Public benchmark scores can be optimized against rather than learned from. We mitigate via (a) a deterministic non-LLM judge that cannot be flattered, (b) cross-class composite scoring that punishes overfitting to PC1, and (c) versioning so a model that overfits v0.3 will not transfer to v1.

Data & privacy. The benchmark contains no personal data. Submissions store only the canvas, the submitter handle, and an SHA-256 hash of the source IP (for rate-limiting and abuse triage; never reversed, never displayed).

Carbon & cost. A full v0.3 sweep is 120 LLM calls (~$4 in API spend across the 4 frontier models, ~1 hour wall-clock including retries on rate limits and malformed JSON). The simulator runs locally and uses negligible energy.

Appendix A: problem set

Appendix B: version history

References

1. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS 2023. arXiv:2303.11366.
2. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770.

Citation

If you reference CHINI-bench in academic or industry work, please cite it as:

@misc{chinibench2026,
  title  = {{CHINI-bench}: A simulator-graded benchmark for {AI} system design},
  author = {Kwon, Alex},
  year   = {2026},
  note   = {Version 0.3. https://chinilla.com/bench},
  url    = {https://chinilla.com/bench}
}

Plain text:

Kwon, A. (2026). CHINI-bench: A simulator-graded benchmark for AI system design (Version 0.3) [Benchmark]. https://chinilla.com/bench