Evaluating our question algorithm

Trying a new way of picking questions on real children would take a term to show anything, and the children would be the experiment. A simulation runs a month of practice in minutes, and it can be run again from exactly the same starting point, so two versions get a fair comparison.

It also tells us something we can never know for certain about a real child: which mistakes they truly make. In the simulation we know, so we can check whether the app spotted the right ones.

An AI chatbot already knows the maths, so we would have to tell it which mistakes to make. If those came from the same list Kudos looks for, Kudos would find them every time, and the test would prove nothing. Instead the simulation uses a small mathematical model, of the kind education researchers use to describe how children answer questions.

The model behind it captures five things about how children answer and learn:

Strengths and weaknesses

Children are stronger in some topics than others, so the same question can be easy for one and hard for another.

Real mistakes

A child can have a habit, such as adding the bottoms of two fractions, that leads to the wrong answer even on a question they could otherwise do.

Slips and lucky guesses

Children sometimes get a question wrong that they can do, or guess one right that they cannot.

Learning from feedback

Being told what went wrong weakens a habit, and practice weakens it a little more. A habit that has faded can come back.

Getting better

Children improve with practice, most of all on questions they have a fair chance at.

Where we had real figures from Kudos, such as how much children differ from one another, the simulation was set to match them. Everything else is an assumption we wrote down, so it can be checked and argued with.

Each simulated month of practice (30 days) is run from the same starting point with Kudos and with random questions:

• Kudos. The real thing: the same code the app runs, with the clock sped up so a day passes in seconds. That includes its re-tests: Kudos comes back to a mistake later with a fresh question, to check it has really gone.
• Random questions. Every question picked at random.

The help after a wrong answer is the same both ways, and so are the questions, which are the app’s own. The only difference is which question comes next. The first study ran 200 months of practice at 10 questions a day; later studies ran 100 at 20 and at 50 a day.

• Real mistakes spotted. Whether, and how soon, the app correctly names a mistake that is really being made.
• Real mistakes fixed. Whether the mistake stops, and whether it comes back.
• False alarms. Mistakes the app names that aren’t really being made. These matter because a parent may be shown them.
• Questions at the right level. Questions with a fair chance of being answered: hard enough to stretch, not so hard they swamp.
• Questions answered correctly, and progress made over the month.

Because both start from the same point, we can compare them run by run. These notes give the differences between them, such as “25% more”, rather than the raw scores.

Two of our assumptions are the easiest to argue with: how much a child learns from being told what went wrong, and whether children improve faster on questions at the right level. We changed each one and ran the simulation again.

The number of mistakes fixed went up and down with them, as it should. But Kudos still kept about a third more of its questions at the right level than random questions did (31% more and 30% more), so that finding does not depend on either assumption.

A simulation shows how a way of picking questions behaves when children learn the way the model says. It cannot show how real ten-year-olds learn, or that Kudos raises 11 plus scores, and we never quote it as if it did. That needs real children over a term. So the app checks again, later, whether the mistakes it names stay fixed, and we ask families after the exam how it went.

What the simulations found, and the version of Kudos they led to, is in how we chose the question algorithm.