Every story game opens with the same promise. This story is tailored by how you play. Telltale put it on a title card. BioWare built a trilogy on it. My own game, a card game about the 1126–1127 siege of Kaifeng, makes the promise too, just more quietly: three routes, dozens of decisions, and a hand of historical figures you can win over or lose.
Here is the problem. I had no idea whether it was true.
I could feel that some choices mattered. I wrote them. But the writer is the worst judge of this, because I know what each option is supposed to do. So I treated the game like any other system I did not trust. I replayed it with one decision flipped at a time, and I read the numbers.
This post is about those numbers. What I measured, what the first run said, what I changed, and what the second run said sixteen days later. There is also a short detour through three much bigger games, because the mistakes I found are not new. They are just rarely counted.
What a choice has to do to count
"Choices matter" is a slogan. To measure it, I first had to say what it would look like if a choice did not matter. Then I could look for that.
I ended up with four questions, each with a number attached.
1. Divergence. Take one playthrough. At one decision, flip the option the player picked and replay everything after it with the same random seed. Compare the two end states: the cards in hand, trust levels, story flags, the ending. How much changed? If the answer is nothing, the choice was cosmetic. I report the share of flipped pairs where nothing changed, and how much changed in the typical pair.
2. Latency. When a choice does change something, how many turns pass before the player can see it? A result that lands next turn teaches. A result that lands eighty turns later, in the epilogue, is a trivia answer.
3. Ending diversity. Across all runs on a route, how many different endings show up, and does one of them take most of the runs? Three endings that split 10/2/2 is one ending with decoration.
4. Reward drought. How many turns in a row can a player go without the game giving them anything that counts: a new person card, or a story flag that something later in the game actually reads? Choices cannot matter if the player has stopped paying attention by the time the payoff arrives.
The test harness plays the real game page, not a model of it. Every run uses a fixed random seed, so it can be repeated exactly. For each route it plays two base runs under different policies, then flips one option at a time and replays: 103 flipped pairs in the current build. The ending and reach numbers come from a wider sweep of 14 scripted runs per route. No language model is involved in these four metrics. The only judgement calls are the pass/fail limits, which I set, and which I come back to at the end.
Bottom line: If you cannot write down what "this choice did nothing" looks like in your game's data, you cannot claim the opposite either. Define the failure first.
The first run: Oct 9
The nightly job ran the harness, checked every metric against its limit, and posted a digest. Here are the rows that answer the four questions above.
| Metric | What it measures | Oct 9 | Target |
|---|---|---|---|
| Zero-divergence choices | Share of flipped pairs where nothing changed | 14.5% | ≤ 10% |
| Max consequence latency (turns) | Longest wait between a choice and its first visible effect | 86 | ≤ 20 |
| Distinct endings (worst route) | Fewest different endings reached on any one route | 1 | ≥ 4 |
| Reward drought (turns) | Longest run of turns with no new person card or live story flag | 14 | ≤ 3 |
| Unvisited written content | Options and outcomes in the game that no simulated run ever reached | 102 | 0 |
| Dead story flags | Flags that get set but nothing later ever reads | 99 | 0 |
Roughly one choice in seven was a coin with the same face on both sides. One route ended the same way in every simulated run. And 102 written options and outcomes were never reached by any run, which means either the simulated players cannot find them or nobody can. A separate static check explained some of them: historical figures you could meet at the first level of trust, with no event on that route ever offering the next level. The option is real, the good result is written, and the path to it does not exist.
The dead flags are the quietest failure and, I think, the most instructive. When you write a branching story, every option sets a flag: the emperor refused to leave the capital, the official sided with the war party. Ninety-nine of those flags were set and never read by anything. The game remembered. It just never did anything with the memory.
I also ran a separate pass where language-model "readers" scored the same transcripts, piece by piece, on a 1–5 scale. They are not ground truth, and I treat them as a second opinion rather than a measurement. But the second opinion agreed with the counting: fork quality averaged 2.39 out of 5 over 103 pieces, impact 2.59, pacing 2.25. Same transcripts, different method, same verdict.
Bottom line: The game's biggest problem was not that choices had no consequences. It was that the consequences were written and then left unplugged: flags nobody read, outcomes nobody could reach, payoffs nobody could feel.
What bigger games already learned the hard way
None of this is new. What is new is counting it.
Sid Meier's definition, from his 2012 GDC talk, is that a game is a series of interesting decisions, and that the quickest test is the negative one: if the player always takes the first option, or might as well pick at random, the decision is not interesting.¹ My zero-divergence metric is the random-pick test done by machine. If swapping the option changes nothing, picking at random was the right strategy.
Telltale's The Walking Dead is the famous case of the other failure. The "Clementine will remember that" card became a meme because players learned that the remembering rarely changed where the story went. Critics have argued the choices shaped who Lee was far more than what happened.² That is a fair design, but it is a latency design: the payoff is in tone and relationships, not plot, and players who were promised plot felt the gap. My 86-turn maximum latency is the same gap, as a number.
And Mass Effect 3 is the ending-diversity failure at the largest scale ever tried. Three games of choices came down to a small set of endings that many players felt did not reflect them, and the reaction was loud enough that BioWare published an open letter and later an extended cut.³ My worst route had collapsed to one ending. The route is a few hundred lines of JSON rather than a trilogy, but the complaint would have been the same.
There is one more source, closer to home. For my agency-management game I sort Steam reviews of eight similar tycoon games into a fixed list of eighteen themes. When I wrote that list, I did not need a theme called my decisions had no consequences. I needed randomness feels arbitrary, I cannot tell why that happened, and nothing changes between runs. Players rarely diagnose the unplugged flag. They report the symptom: the game feels like it is happening to them.
Bottom line: Players do not complain that choices are meaningless. They complain that they cannot see the meaning. Latency and clarity predict the review, not divergence.
What I changed
Most fixes were plumbing, not writing.
The Oct 9 digest did not just report the failures. It filed seven proposals, each tied to a failing metric and naming the gap. Over the next two weeks they were built on their own branches and merged on Oct 25. In rough order of impact:
Dead flags were plugged in. A flag that nothing read now has a reader: an incident that is skipped if you already fled the city, an ending that only happens if you were held in the enemy camp. If a consequence is not worth reading later, it is not worth setting.
The collapsed route got new endings, and the default ending stopped being where every run landed unless it had explicitly earned something else.
Knowledge-card slots, which let the player spend a remembered precedent to strengthen an action, had been mostly for show: in early October only 17% of them changed the game state. Slot effects now grant, unlock or remove cards instead of just changing the text.
Weak outcomes stopped being dead ends. A weak result now opens its own line, with a way back, so taking the worse option gives a different story rather than a shorter one.
I did not touch the pass/fail limits, and I did not add content just to move a number. The point was to find out whether the game I had already written does what it claims.
The second run: Oct 25
| Metric | Oct 9 | Oct 25 | Target | Status |
|---|---|---|---|---|
| Zero-divergence choices | 14.5% | 13.6% | ≤ 10% | Still failing |
| Typical state change per choice (fields) | — | 6 | ≥ 3 | Passing |
| Typical consequence latency (turns) | — | 1 | ≤ 6 | Passing |
| Max consequence latency (turns) | 86 | 95 | ≤ 20 | Worse |
| Distinct endings (worst route) | 1 | 3 | ≥ 4 | Improving |
| Runs that fall through to the default ending | — | 21% | ≤ 40% | Passing |
| Reward drought (turns) | 14 | 17 | ≤ 3 | Worse |
| Slot effects that change state | 17% | 39% | ≥ 50% | Improving |
| Unvisited written content | 102 | 79 | 0 | Improving |
| Dead story flags | 99 | 1 | 0 | Nearly there |
| Recovery rate after a weak outcome | — | 78% | ≥ 50% | Passing |
The plumbing worked. Ninety-nine dead flags became one. Unvisited content fell by a quarter. The worst route now reaches three endings instead of one, and only 21% of runs fall through to the default ending. The typical choice changes six fields of game state and shows its first effect on the very next turn.
And two numbers got worse.
Maximum latency went from 86 to 95 turns. My reading, not yet proven, is that plugging dead flags into late-game gates created new long-range consequences: more things are remembered, and some of the remembering only pays off at the end. That is the Telltale trade. Reward drought grew from 14 to 17 turns on the princess route, and I suspect the same cause: a flag only counts as a reward if something reads it, and on that route the readers come late. Both are real regressions, and the harness flagged them in the same run that confirmed the fixes.
Zero-divergence barely moved, 14.5% to 13.6%. That one is not plumbing. The remaining same-outcome pairs are mostly options I wrote as real alternatives that turn out to lead to the same place. Fixing them means writing, not wiring.
Bottom line: Structural fixes move structural metrics. Dead flags, unreached outcomes and ending collapse all responded to plumbing. The share of choices that truly fork did not, because that is a writing problem dressed up as a metrics problem.
Real life is in the details. Cautions.
The limits are mine. "At most 10% of choices may be cosmetic" is a design opinion I wrote down, not a law. Several limits are marked calibrate in the harness, meaning I am not yet sure the threshold is right. A metric failing its limit tells me to look, not necessarily to change the game.
Simulated players are not players. The harness picks options with a fixed policy that explores the tree evenly. Humans do not. A choice that is cosmetic in 14% of simulated pairs might be one no human ever faces, or the one every human faces. The metrics describe the content, not the experience.
The sample is small. A hundred-odd flipped pairs and fourteen ending runs per route is enough to find unplugged plumbing. It is not enough to put error bars on "14%". Treat the percentages as a count of what is in this build, not an estimate of anything beyond it.
Divergence counts state, not meaning. A choice that changes six fields of bookkeeping is not necessarily more meaningful than one that changes a single line of dialogue a player remembers for years. Telltale's whole argument is that the second kind is the point. My harness cannot see it.
Two snapshots are not a trend. Oct 9 and Oct 25 sit either side of a burst of fixes. They do not tell me whether the game is heading somewhere. The nightly job will.
And the language-model reader scores are a second opinion I chose to report because they agreed with the counting. Had they disagreed, I would have trusted the counting and looked into the readers, not the other way round.
Why bother
Because the alternative is the title card. This story is tailored by how you play. I can now say, for one small game, how tailored: about 86% of decisions change the game state, the typical change shows up next turn, every route reaches at least three endings, and one flag in a hundred is still set and never read.
That is not a slogan. It is a measurement, with a date on it, that I expect to be wrong about within a week, and able to prove it.
¹ Leigh Alexander, "GDC 2012: Sid Meier on how to see games as sets of interesting decisions," Game Developer, March 2012. https://www.gamedeveloper.com/design/gdc-2012-sid-meier-on-how-to-see-games-as-sets-of-interesting-decisions
² "The Walking Dead, Choices, and Kenny," Game Developer. https://www.gamedeveloper.com/design/the-walking-dead-choices-and-kenny
³ "Mass Effect 3 ending controversy," Wikipedia; Ray Muzyka, "To Mass Effect 3 players," BioWare Blog, March 21, 2012. https://blog.bioware.com/2012/03/21/4108/
Metrics come from the game's nightly harness: the Oct 9 figures from the cloud digest of run 9a746cb, the Oct 25 figures from a local run at commit 18b61cd. The harness plays two seeded base runs per route, builds 103 flipped pairs by swapping one option and replaying the same seed, and sweeps 14 scripted runs per route for ending and reach statistics.