A map of the field

Knowledge tracing asks: given a student answering exercise after exercise, can we estimate what they actually know, and how that changes as they practice? I spent part of my first year reading the canon and reproducing some of the models taken to be state of the art.

The problem

A learner produces a sequence of interactions : an exercise , tagged with one or more knowledge components (KCs), and a response . A knowledge-tracing model keeps a belief about the learner’s mastery and predicts how they’ll do next, .

What makes this hard is that the thing we care about is not observed, i.e., mastery is latent; it (at least is assumed to) changes over time, going up with practice and down with forgetting; and it’s revealed only through a single bit per step – that mixes knowing with guessing, slipping, and exercise difficulty. A lot of the field’s ingenuity is about recovering a rich, moving, hidden state from this very thin signal.

To model the latent state

Probabilistic models. The foundation is Bayesian Knowledge Tracing (Corbett & Anderson, 1994), per skill, a two-state hidden Markov model – a KC is learned or unlearned – with four parameters, the prior , a learning transition , a guess rate , and a slip rate . Its great virtue is that every parameter means something we could explain to a teacher. While classic BKT has no forgetting and assumes KCs are independent, much of what follows is the field relaxing those assumptions one at a time, for instance with dynamic Bayesian networks that model prerequisite structure between skills (Käser et al., 2017).

Logistic / factor models. Instead of a latent Markov chain, this family predicts correctness with a logistic function of additive features, close in spirit to item response theory. Learning Factors Analysis (Cen et al., 2006) uses initial knowledge, KC easiness, and a learning rate; Performance Factors Analysis (Pavlik et al., 2009) swaps in counts of prior successes and failures, which lets it tell students apart; Knowledge Tracing Machines (Vie & Kashima, 2019) generalize the whole family as factorization machines, so arbitrary side information can be encoded as features.

Deep models. After 2015 the field moved here, trading interpretability for fit. Deep Knowledge Tracing (Piech et al., 2015) runs an LSTM over the interaction sequence and reads mastery off the hidden state. Dynamic Key-Value Memory Networks (Zhang et al., 2017) make memory explicit, with a static key matrix over latent concepts and a dynamic value matrix for mastery. Attention models – SAKT (Pandey & Karypis, 2019), AKT (Ghosh et al., 2020), which uses a monotonic, distance-decaying attention – replace recurrence with self-attention; graph-based models like GKT (Nakagawa et al., 2019) and structure-based KT (Tong et al., 2020) put the KC graph into the architecture itself. These predict the next answer better; the cost is that the latent state becomes hard to read.

Forgetting and structure

Two extensions reappear across all three families, which I think is a useful signal about what the base models are missing.

The first is forgetting. BKT’s zero-forgetting assumption is plainly unrealistic, so the field keeps reintroducing time. The cheap version adds a lag-time feature (Nagatani et al., 2019); the more principled versions model decay like DAS3H for scheduling distributed practice (Choffin et al., 2019), or HawkesKT (Wang et al., 2021), which treats a learning history as a point process where each past interaction excites future correctness through a kernel that decays over time.

The second is structure dropping the pretense that KCs are independent. Prerequisite-driven DKT (Chen et al., 2018) regularizes toward a prerequisite graph; GKT and structure-based KT propagate mastery along the knowledge graph; RKT (Pandey & Srivastava, 2020) folds exercise relations and a time-decay kernel into attention.

How the field measures itself

I reproduced the temporal and structural end of this: HawkesKT, SKT, and RKT. This was only possible because of how much the community has open-sourced, from reference implementations like USTC’s EduKTM to benchmarks such as pyKT (Liu et al., 2022) that standardize datasets and evaluation.

In practice, almost everything is judged by AUC on next-answer prediction (sometimes accuracy or RMSE). But predicting the next answer is not the same as recovering the latent state these models claim to estimate, and the two come apart. The clearest demonstration I found is “How Deep is Knowledge Tracing?” (Khajah, Lindsey & Mozer, 2016): give BKT the same advantages the deep models enjoy, forgetting, student ability, item difficulty, and most of deep learning’s lead disappears. Good next-answer prediction, then, can come from capturing effects a simple interpretable model captures too.

However, when the latent state is legible enough to inspect, it is often visibly incoherent. Yeung & Yeung (2018) show that DKT’s mastery estimates jump around non-monotonically – knowledge spiking after a wrong answer, or for skills the student never practiced – even while the model predicts well. So a model can win on next-answer prediction while the trajectory it reports is not a believable account of learning at all.

My worry is also larger as the field has gradually swapped its real goal, modeling and improving how people learn, for a tractable proxy, predicting the next answer on passively logged data, scored by AUC. “State of the art” rests on datasets from a few tutoring platforms, and some of them are even from decades ago. It is hard to say how much of the steady AUC creep is insight into how people learn, and how much is overfitting to the quirks of ASSISTments and its cousins. When pyKT re-ran much of the field under one fixed pipeline, several widely cited models turned out to have been leaking the current response into their own inputs. In other words, some of the creep was never about learning at all.

AUC on next-answer prediction can go up for years while our account of how people learn stands still. No leaderboard scores these questions: what memory is, how concepts depend on one another, what it even means to know something.

References