chini-train-heldout-0010-dp4-infra
Session-Store With Cache Stampede Risk
hard infra problem: session-store with cache stampede risk
Source: chini-train synth generator v0.1
Prompt
Design a system for: session-store with cache stampede risk (domain: SWE backend / infra). Tier DP4 (hard). 7-10 nodes, three stress scenarios including adversarial, tight criteria. Constraints: - At most 11 components on the canvas. - Monthly cost ceiling: $366 USD. Required behaviors: storage, queue, circuitbreaker, retry, ratelimit. Return a Chinilla CanvasState that handles the listed scenarios. Include trigger components for each entry point and at least one terminal storage / sink so the simulator can score delivery.
Constraints
- Max components
- 11
- Required behaviors
- storage, queue, circuitbreaker, retry, ratelimit
- Monthly budget
- $366
Stress scenarios
Baseline traffic
baselineSteady ambient load with no failures.
Dependency outage
outageA downstream component is disabled. System must degrade gracefully.
Adversarial burst
adversarialHostile packets injected on top of clean traffic. Defenses must block them without dropping good requests.
Traffic spike
spikeTraffic suddenly multiplies. The hot path must hold.
Pass criteria (overall)
- Min stability score
- 79
- Max drop rate
- 8.8%
- Min delivery rate
- 87.5%
- Max errors
- 6
Submit your run
Submissions go through the chini-bench CLI. It calls your model with your key, scores the result locally, and posts to the leaderboard. Nothing leaves your machine except the canvas it produces.
End-to-end:
pip install git+https://github.com/collapseindex/chini-bench-cli.git
export OPENROUTER_API_KEY=...
chini-bench run chini-train-heldout-0010-dp4-infra \
--provider openrouter --model google/gemini-2.0-flash-001 \
--as aliceOr inspect the prompt first:
chini-bench prompt chini-train-heldout-0010-dp4-infra Providers: openai · anthropic · google · openrouter · ollama
Leaderboard
| Rank | Submitter | Model | Score | Stability | Delivery | Design | Pass |
|---|---|---|---|---|---|---|---|
| #1 | chini-train-08 | base_7b single-shot | 82 | 37.0 | 100.0 | 100.0 | ✗ |
| #2 | chini-train-08 | base_7b single-shot | 82 | 37.0 | 100.0 | 100.0 | ✗ |
| #3 | chini-train-08 | base_7b single-shot | 82 | 37.0 | 100.0 | 100.0 | ✗ |
| #4 | chini-train-08 | base single-shot | 82 | 37.0 | 100.0 | 100.0 | ✗ |
| #5 | chini-train-08 | fmt_a_7b single-shot | 77 | 58.0 | 45.0 | 100.0 | ✗ |
| #6 | chini-train-08 | fmt_a_7b single-shot | 77 | 58.0 | 45.0 | 100.0 | ✗ |
| #7 | chini-train-08 | fmt_a_7b single-shot | 77 | 58.0 | 45.0 | 100.0 | ✗ |
| #8 | chini-train-08 | fmt_a_v5_mixed_7b single-shot | 76 | 55.0 | 42.0 | 100.0 | ✗ |
| #9 | chini-train-08 | fmt_a_v5 single-shot | 76 | 55.0 | 42.0 | 100.0 | ✗ |
| #10 | chini-train-08 | rl_v07_pilot_a10b_k8_s3 single-shot | 76 | 55.0 | 42.0 | 100.0 | ✗ |
| #11 | chini-train-08 | rl_v07_pilot_a10b_k8_s5 single-shot | 76 | 56.0 | 43.0 | 100.0 | ✗ |
| #12 | chini-train-08 | rl_v07_pilot_a10b_k8_s6 single-shot | 76 | 55.0 | 42.0 | 100.0 | ✗ |
| #13 | chini-train-08 | fmtA single-shot | 76 | 55.0 | 42.0 | 100.0 | ✗ |
| #14 | chini-train-08 | rl_v07_full single-shot | 76 | 56.0 | 43.0 | 100.0 | ✗ |
| #15 | chini-train-08 | fmt_a single-shot | 75 | 55.0 | 40.0 | 100.0 | ✗ |
| #16 | chini-train-08 | fmt_a_v2 single-shot | 75 | 53.0 | 39.0 | 100.0 | ✗ |
| #17 | chini-train-08 | fmt_a_3b single-shot | 75 | 53.0 | 39.0 | 100.0 | ✗ |
| #18 | chini-train-08 | fmt_a_3b single-shot | 75 | 53.0 | 39.0 | 100.0 | ✗ |
| #19 | chini-train-08 | rl_v06_run2 single-shot | 75 | 54.0 | 40.0 | 100.0 | ✗ |
| #20 | chini-train-08 | rl_v07_pilot_a10b_k8_s0 single-shot | 75 | 54.0 | 40.0 | 100.0 | ✗ |
| #21 | chini-train-08 | rl_v07_pilot_a10b_k8_s1 single-shot | 75 | 54.0 | 40.0 | 100.0 | ✗ |
| #22 | chini-train-08 | rl_v07_pilot_a10b_k8_s2 single-shot | 75 | 54.0 | 40.0 | 100.0 | ✗ |
| #23 | chini-train-08 | rl_v07_pilot_a10b_k8_s4 single-shot | 75 | 54.0 | 40.0 | 100.0 | ✗ |
| #24 | chini-train-08 | rl_v07_pilot_a10b_k8_s7 single-shot | 75 | 54.0 | 41.0 | 100.0 | ✗ |
| #25 | chini-train-08 | fmt_a_v4_opus_7b single-shot | 73 | 60.0 | 48.0 | 100.0 | ✗ |
| #26 | chini-train-08 | fmt_a_v4_opus_7b single-shot | 73 | 60.0 | 48.0 | 100.0 | ✗ |
| #27 | chini-train-08 | base single-shot | 62 | 37.0 | 32.0 | 70.0 | ✗ |
| #28 | chini-train-08 | base_3b single-shot | 62 | 37.0 | 32.0 | 70.0 | ✗ |
| #29 | chini-train-08 | base_3b single-shot | 62 | 37.0 | 32.0 | 70.0 | ✗ |
| #30 | chini-train-08 | base single-shot | 62 | 37.0 | 32.0 | 70.0 | ✗ |
Per-scenario breakdown of the top run
| Scenario | Health | Drop rate | Delivered | Pass |
|---|---|---|---|---|
| baseline | 50.0 | 8.3% | 132 | ✗ |
| outage-1 | 42.0 | 9.3% | 110 | ✗ |
| adversarial-2 | 15.0 | 100.0% | 346 | ✗ |
| spike-3 | 41.0 | 18.5% | 4101 | ✗ |