How we chose the question algorithm

A good way of picking questions has to do four things at once:

• Fix the mistakes a child really makes, not ones they don’t.
• Keep questions at the right level, so practice stretches a child without swamping them.
• Raise few false alarms, because a mistake the app names may be shown to a parent.
• Help the child improve over the month.

Every version was tested the same way. The simulation ran the same month of practice from the same starting point, with Kudos choosing every question, its re-tests included (Kudos comes back to a mistake later with a fresh question, to check it has really gone), and with every question picked at random. Each figure below is the difference between the two, such as “25% more”. How the simulation works is explained in the note before this one.

It pitched questions well. Compared with random questions, 30% more of its questions were at the right level for the child, children answered 20% more correctly, and it fixed 7% more of the mistakes they really made.

It was poor at telling which mistakes were real. It spotted 13% fewer of children’s real mistakes than random questions did, because many of the mistakes it named were ones the child didn’t actually make. Most wrong answers come from a child who can’t do the question yet, not from a habit, and it treated too many of them as habits.

We changed when Kudos treats a mistake as real, and had it look more widely for mistakes in a child’s first weeks.

Compared with version 1, it spotted 17% more of the mistakes children really made, and fixed 8% more. Against random questions, it now fixed 13% more.

Each wrong answer in a multiple-choice question is written to catch one particular mistake. In this version, choosing a wrong answer named that exact mistake, instead of a general one for the whole question.

That made Kudos more precise, but many of the mistakes it named were still ones the child didn’t really make. The reason is simple: most wrong answers come from a child who can’t do the question yet, and those land on the named wrong answers like any other. One multiple-choice answer on its own says very little.

What helped was also counting the times a child avoided a mistake, not only the times they made it. That roughly halved the number of mistakes Kudos went after (49% fewer), so far fewer false alarms.

By now, daily practice was moving to twenty questions a day, and the weekly set towards fifty. Version 4 decides whether a mistake is real by weighing up everything a child’s answers say about it, rather than counting wrong answers. Why we moved to twenty a day is a note of its own.

Each chart sets random questions to 100 on every measure, in the same simulation, so Kudos can be read straight against them. For mistakes fixed, Kudos now stands at 116: for every 100 mistakes random practice fixed, Kudos fixed 116, which is 16% more.

Version 4 fixed 25% more real mistakes than version 3, and kept questions at the right level just as well. The same comparison at fifty questions a day:

Against random questions, version 4 fixes 16% more of the mistakes children really make, keeps 33% more of its questions at the right level, children answer 19% more correctly, and they make 24% more progress. For a ten-year-old, a session at the right level is the difference between one that feels doable and one that doesn’t.

The re-tests are a big part of the fixing, so any that are due still come first in a session. And because one multiple-choice answer says so little on its own, a parent is only shown a mistake when the app has the same evidence it acts on, and it is described as one their child may be making.

Results from real children. The simulation assumes children learn the way its model says, so these tests help us choose between versions; they don’t show how much real children learn. The app checks again, later, whether the mistakes it names stay fixed, and we ask families after the exam how it went.