Do chess improvement apps actually work?

What 167 users' ratings actually did

Rating change from each user's first to last post-signup game, inside their dominant time class. Every user in the sample is on this chart — including all 42 who lost rating.

This is the whole sample, not a highlight reel. 122 of 167 users gained rating; the spread runs from +552 down to −254, and the middle of the distribution — the number a skeptic should care about — is +35. The biggest gainer (1073 → 1625 in five days) is an account climbing fast on a small number of games, exactly the profile you'd flag as a still-calibrating rating. We flag it too: it's one reason the headline here is a median with a sample size attached, not the mean (+48) it inflates.

That blended median also understates what a settled user sees, because most of this sample is new. Among the 50 users we have now been measuring for 18 days or more, the median is +55 and 84% gained; among the 117 measured for less, it's +26 and 68%. The full split is in the run table, and the reason it matters is arithmetic, not spin: absolute rating gain scales with how long you've been playing.

For context on what those users are actually fixing while they climb, the companion reports cover what each Elo level struggles with and how chess games are actually lost — short version: games are decided by one collapse, and shrinking that collapse is where the rating points live.

The stat we got wrong

Published Jul 13 and Aug 3, 2026. Retracted Sep 7, 2026. Here is the error in full, because a page arguing about honest denominators does not get to quietly edit its own numbers.

The first two versions of this article carried a second headline next to the rating gain: active users improved ≈1.7× faster per game than their own pace before joining. It was the number we told people to watch. It was wrong. Not the data, the arithmetic on top of it.

We computed it as median(post-signup pace) ÷ median(pre-signup pace). Those are two separately sorted lists. Dividing one by the other tells you how two groups compare; it says nothing about what happened to any person, which is exactly what "faster than their own past" claims. The correct computation is one difference per user, then the median of those differences. Do that, and the effect disappears.

Read the last two columns. The paired lift was negative on the day we first published the claim, and it is indistinguishable from zero now, with almost exactly half the sample beating their own prior pace — a coin flip. The two methods diverge because of regression to the mean: users who arrive on a hot streak cool off, users who arrive on a cold one heat up, so the two sorted lists keep their gap while the per-person differences collapse onto zero. The July and August rows are genuine as-of replays — we capped both game dates and signup dates at each run date — so those are the cohorts we published, recomputed, not a convenient new sample.

Nothing else in the article moved. The absolute gains (+35 median, 73% gaining, the mature and fresh split) were computed one user at a time from the start and are unaffected. What we lost is the per-game multiple, and we've replaced it with a comparison that can't make the same mistake — directly below.

Faster than their own past — the honest version

Median rating change over a fixed number of games: each user's post-signup run against their own equally long stretch immediately before it. Same person, same game count, same time class (n=135).

The obvious objection to any before/after stat is that maybe these players were improving anyway. So we compare them to themselves — carefully, because the careless version of this is exactly what we got wrong. Take each user's post-signup games in their dominant time class, count them, then take that same number of their games immediately before signing up. One difference per user, no sorted lists.

The median user gained +39 across their post-signup run, against +14 across their own previous run of identical length. The gap between those two medians (+25) is still not the effect — pair it person by person and the median lift is +7 rating points, with 56% of users beating their own prior stretch against a 50% coin-flip baseline. That is a real edge and a small one, and it is roughly an order of magnitude less impressive than the number we used to print.

The encouraging thing in that table is the direction of the last column: 50.0% → 52.6% → 56.3% across three runs. Three points is not a trend yet, it's a thing to keep measuring, which is why the table exists rather than a single number.

And here's the claim we killed first, back in July: measured per month, the same data says users improve 3.6× faster after joining. That number is real, spectacular, and dishonest. Users simply play more after joining an improvement app, so a per-month framing smuggles play frequency in as skill. Fixing the denominator was right; we then got the arithmetic on the fixed version wrong too, which is a decent argument for re-running your own numbers on a schedule. If you're comparing tools, track your progress in per-game terms — and pair it against yourself, not against a median.

How that pace compares to the rest of chess

Per-game rating pace — our users before they signed up, versus what published data implies is typical at each level. The grey bars are estimates (caveats below); our number is measured directly.

Comparing someone to their own past only means something if that past was normal to begin with. It was. Our users' pre-signup pace of 0.40 rating points per game lands right on what the wider data implies for their level. Published medians from Chess.com's SmarterChess analysis put a studying 800-rated player near +239 rating a year, and Lichess's public rating data (hundreds of millions of games) shows beginners around 800–1000 gaining their first ~100 points within a few months. Convert either into a per-game figure for an active player and you land in the 0.3–0.5 range — exactly where our users started before joining.

Two honest caveats, because this is a comparison. First, improvement gets harder as you climb: the same external data has a 1600-rated player gaining only about +35 a year, so the grey bars drop fast with level — and our sample skews toward beginners, whose ceiling for quick gains is naturally higher. Second, turning an annual figure into a per-game one means assuming a game volume — the exact move we refused to make with our own per-month stat. So treat the grey bars as a rough band, not a precise line. What is not an estimate is the 0.40: it's counted per game, directly, on real accounts. Earlier versions of this chart carried a second teal bar for our post-signup pace, with the gap between the two as the punchline. We removed it in September 2026 — putting two medians of two different distributions side by side is precisely the error this article now documents, and a chart is no place to repeat it.

Five ways we tried to kill the stat

Before publishing in July, we attacked the result the way a skeptical reviewer would. The attacks and verdicts, unedited. A sixth attack, run in September, killed a claim these five let through — that one gets its own section.

What we will and won't claim

We will claim: among active users, 73% gained rating within their first weeks (median +35, n=167; +55 and 84% among the 50 measured for 18 days or longer), and that over the same number of games the median user did +7 rating points better than their own immediately preceding run, with 56% beating their own prior stretch. Both are descriptive, both are computed one user at a time, and both come with their sample size attached.

We won't claim: “Chess DNA adds X rating” — that's causal, and observational data can't support causality no matter how much a marketing page wants it to. We won't claim “3.6× faster improvement,” because we know exactly which confound produces it. And we no longer claim “≈1.7× faster per game,” because we checked our own arithmetic and it didn't hold. When any training tool shows you an improvement stat, ask which attacks they ran against it, what denominator the claim is hiding, and whether the comparison was made person by person or list against list. If you'd rather test it on your own games, Chess DNA analyzes them free and shows you the same per-game curve we used here.

Three runs, and what moved between them

A static stat is easy to cherry-pick; a re-run schedule is not. We re-run the identical query monthly — same ≥20-game bar, same dominant-time-class rule, same exclusions. Every run gets a row here, including the ones that dip and the one that caught us out.

Read the blended rows honestly and every new run looks like a dip: +50, then +30, then +35. The reason is boring, and it's not the app: each run pulls in a wave of fresh signups whose measurement windows are short — the median is down to 9.6 days now — and absolute rating gain scales with time on the clock. Split by window and the story is stable. Users measured over the same horizon as run 1 (≥18 days) show +55 with 84% gaining in run 3, against +96 and 88% in run 2 on just 17 users, and +50 and 82% in run 1. The fresh two-thirds sit at +26, which is what "hasn't had time yet" looks like in data.

And the stat this article originally told you to watch — per-game pace against your own pre-signup self — didn't move between runs either. It just wasn't measuring what we said it was. Run 3 is where we checked it properly and retracted it. The replacement, a paired equal-length comparison, has been backfilled onto all three runs so the series stays comparable: +4.5, +15.0, +7.0 rating points of median lift, and 50.0% → 52.6% → 56.3% of users beating their own prior run. No new accounts needed excluding this run, and the five attacks in the grill apply unchanged, including the selection-bias disclosure. That's the whole point of re-running: the flattering number moved with cohort mix, and the honest number turned out to need honest arithmetic too.