Page 11 of 12
Measuring and claiming
Brain-training products fail on measurement in a predictable way. They show reaction-time difference scores, which are unreliable for one person, and they report gains on the trained task as if they were gains in life. This page is how we avoid both.
The reliability paradox
A task can produce a rock-solid group effect and still be useless for tracking one person, because the spread between people is tiny relative to the noise between trials. Hedge, Powell and Sumner (2018) named this the reliability paradox, and the numbers are worse than most people expect.
- Enkavi et al. (2019) found task measures averaged a test–retest intraclass correlation of about 0.3, against about 0.6 for questionnaires - and contrast scores were worse than raw scores.
- Difference scores - incongruent minus congruent, and stop-signal reaction time - are the worst hit of all.
- Rouder and Haaf estimate that trial noise in common attention tasks is roughly eight times the true individual variability, which means hundreds to thousands of trials before a difference score means anything about one person.
An intraclass correlation is a 0-to-1 figure for how well a score reproduces on retest; 0.7 is the usual floor for tracking individuals and 0.9 for decisions that matter. Most of the field's favourite measures do not clear the first bar.
What we track, and what we refuse to
| Measure | Reliability | How it is used |
|---|---|---|
| Commission error rate | Good - accuracy-based measures held up best | Primary progress metric |
| Premature response rate | Not formally studied, but it is a count, so with 60+ trials it should behave like commission errors | Primary progress metric |
| Mean go reaction time and its variability | Moderate to good; raw times are far more stable than differences | Secondary |
| Rush-error ratio | Untested - treat as exploratory | Shown to the teen as a style gauge, never as a score or a gate |
| Stop-signal reaction time | Poor to moderate for individuals (roughly 0.3–0.6); needs 50+ stop trials and the exclusion rules | Per-block feedback only; averaged across a whole week before any trend is shown |
| Stroop and flanker interference | Poor (about 0.6 at best, and 0.4 for flanker) | Never used for individual progress |
| Resting breathing rate | Stable and non-medical | The fair number to show improving for Settle |
| Accident count | Unvalidated | A trend line, and the outcome that actually matters |
The design rules that follow
- Two burn-in sessions before any number is shown to anyone. They are discarded, and they are also where the personal baseline median comes from.
- Rates and raw times, never a difference score as a progress measure.
- Weekly rolling averages with an uncertainty band, as your own trend - never against norms, percentiles or "healthy ranges".
- A fixed assessment block: same stimuli, same timings, same order rules, even while the training blocks adapt around it.
- Held-out item sets for Notice and Connect, so an in-app gain is not memorisation of the item bank.
- Frustration and dropout are first-class metrics, reported alongside performance rather than under it.
External evaluation, and where it lives
Questionnaires are much more stable than tasks, so the sensible far-transfer outcome for any pilot is a parent-completed instrument. The BRIEF-2 is the standard one: internal consistency above 0.90 on the parent form, retest around 0.81–0.86, inter-rater 0.96–0.98.
The BRIEF-2 is a licensed clinical instrument and it produces clinical-range labels, which is exactly what this product must not do. It is used only in an opt-in, parent-completed, external evaluation, and it is never surfaced inside the product to a teenager as a self-check. The in-product alternative is a five-item, non-clinical weekly self-rating worded as habits rather than symptoms - "this week I answered before the question finished: never / sometimes / often".
The same logic governs the other external measures: a short social-inference instrument with adolescent norms, the figural Test of Relational Reasoning (figural, so it is a genuine transfer test rather than a copy of the training), the written Autobiographical Memory Test, and parent forms. Results go to the family as a report. Nothing from them appears in the teenager's interface.
Cadence is baseline, six to eight weeks, and a three-month follow-up - because both memory specificity training and the specificity induction fade, and a design that stops at post-test will overstate them. And if we ever want to claim anything, the comparison is an active control: a fun-but-inert activity, not a waitlist. Every task in this document has a practice effect, so a waitlist would flatter us by construction.
What can and cannot be said
Two case studies set the ceiling.
Lumosity paid a $2 million settlement to the FTC in January 2016 (against a suspended $50 million judgment) for claiming its games improved school and work performance and delayed cognitive decline without competent and reliable scientific evidence. The order requires such evidence for any future real-world-performance or health claim. Note precisely what happened: the product was not fined for being useless. It was fined for the claims. Marketing copy is the risk surface.
EndeavorRx is the honest ceiling for a game-only product. In STARS-ADHD - 348 children aged 8 to 12, 25 minutes a day, five days a week, four weeks, double-blind against an active game control - the primary endpoint was met: the TOVA attention index improved +0.93 against +0.03 (p = 0.006), with a responder rate of 47% against 32%. But all five clinical secondaries, including parent and clinician ratings and the BRIEF, did not separate from the control game. The label says benefits were shown on a laboratory attention test and may not appear in typical behavioural symptoms. Adolescent and adult follow-up trials were single-arm.
An FDA-cleared, well-funded, double-blind attention game achieved near transfer, honestly labelled. That is the realistic upper bound, and any product claiming more without that evidence is making a Lumosity claim.
Practice at specific skills, on things that look like the exercises, with your own numbers shown improving over weeks. The claims page says this, the drills are framed this way inside the product, and if we run an evaluation we publish the result whatever it says.
One more note from the same trial, relevant to this profile: 4.3% of participants had treatment-related adverse events, mostly frustration and headache. Frustration is a real side effect of a hard attention game. Adaptive difficulty holds success around 70–85%, a visible bail-out costs nothing, and the regulation ritual is the reset.
What the model may never touch
A local model steers the session: it picks one of three exercises the scheduler offers and says why in a line, generates scenario text from templates against a verified ground truth, rates free text against a fixed rubric, and gives one observation, one thing that went well and one suggestion. It answers "why am I doing this?" from a curated document set rather than from open knowledge.
It decides none of the following: scores, ability estimates, difficulty step sizes, spacing schedules, session caps, the credit ledger, safety thresholds and escalation, or what a parent sees. It never infers, stores or mentions a condition, and a post-filter strips clinical terms from its output before display. The model proposes; the rule engine disposes. Small models are more suggestible than large ones, which is one more reason the engine owns every decision that matters.
Version 133 · Privacy · Terms of use · Appearance