CHINI-train

CHINI-train is an open RL training pipeline for graph-valued outputs. The training target is a CanvasState JSON document, the kind a Chinilla user would build by hand: nodes, edges, behaviors, parameters. The reward signal is the CHINI-bench simulator, called over a local HTTP endpoint, returning a deterministic 0 to 100 score and a breakdown across stability, delivery, cost, constraints, and design quality.

The stack is small on purpose. ~2,000 lines of Python. SFT followed by GRPO (Group Relative Policy Optimization). Trained on a single A10G via Modal for under $15 per full run. The base model is Qwen2.5-1.5B. No frontier-scale training, no closed reward model, no human preference data.

The reason the parameter count is small is the same reason the simulator is real: we are testing a recipe, not chasing a leaderboard. If a 1.5B model can learn to design systems that survive a deterministic simulator, that recipe transfers to anything else with executable ground truth: code that compiles, tool-use DAGs that return 2xx, queries that return correct rows. System design is the testbed because we already built the simulator.

The RL-trained model produces canvases that are systematically smaller than the base model. Mean component count dropped from 8.6 to 6.6. Mean cost utilization dropped from 96% of budget to 77%. Mean capacity utilization from 83% to 65%. The policy learned to stay safely under every constraint by underbuilding.

It also collapsed in distribution. The base model produced 7.0 distinct canvas topologies per K=8 sample. The RL model produced 3.65. Per-problem standard deviation of component count fell from 1.84 (base) to 0.16 (RL). Effectively deterministic given a problem.

Both pathologies trace to the same root cause: the bench's cost subscore is asymmetric. Going over budget hurts. Coming in under budget caps at full credit the moment you are below the line. The reward function was the raw bench composite divided by 100, with no shaping. Once the policy figured out that minimal canvases reliably score above the median, gradient descent did the rest.

This is a clean, replicable, mechanistically-explained failure of a flat reward. It is also fixable, which is what v0.7 is about.

The shaping landed in the way it could. Mean composite climbed +2.8 over the warm-start (+8.7 over the cold base). Distinct topologies per K=8 went from 3.65 (v0.6 RL collapse) to 5.10. The quality distribution of canvases the model already knows how to produce got better.

Pass@1 did not move. 2/20 problems passed before training, 2/20 passed after. The diagnosis is in the per-tier breakdown: dp1 = 100%, dp2 = 33%, dp3 through dp6 = 0%. Mean score on dp3-dp6 sits at 78-80, uniformly. The model produces near-miss canvases on hard problems but never crosses the line.

Mechanism finding from the pilot. Of the four shaping terms, only one bit. The diversity bonus (added to the GRPO advantage at the group level) was ~10x more effective than the per-rollout reward shaping (utilization, size, coverage) at the same alpha. The reason is gradient amplitude: 0.10 alpha × bounded ±0.05 = ~0.005 reward delta per canvas, dwarfed by the 0.1-0.3 bench reward variance within a K=4 group. Advantage-level shaping is a much stronger gradient signal because it isn't fighting noise.

Most published RL-on-LLM work uses RLHF with a learned reward model. The reward model is itself a noisy LLM trained on preference data. You cannot tell whether the policy is gaming the reward model or actually getting better. We have a reward signal that is a deterministic discrete-event simulator. There is no reward model to game; either the simulated traffic survives or it does not.

That makes CHINI-train a useful environment for studying reward-shaping mechanics directly. The v0.6 collapse-into-minimalism result above is a clean example: a textbook reward-hacking failure caught and explained in a setting where you can actually see what happened, because the grader has no opinions.

The recipe is also small enough to reproduce. Total v0.6 training cost was under $15 on a single A10G. Inference replays are free. The full v0.6 inference autopsy and v0.7 reward specification are in the repo.