The importance of verifiable learning

A common train of thought in the age of AI is: “do children need to learn math anymore?” Now that AI is capable of completing long-horizon tasks in computer programming and solving increasingly difficult math problems, people are starting to ask: what’s the point of doing something that an AI can readily do for me? Andreessen Horowitz has backed the Horowitz Andreessen Academy that aims to replace traditional college, emphasizing building over studying. Some people are starting to push philosophy and humanities education in response to AI, as less-verifiable skills that AI cannot as rigorously be proven to replace. For some, there’s pervasive doomerism that the marginal value of human cognition is declining to zero in general.

Maybe this theory is defensible for college students, who should (but often do not) already have a solid foundation in more rigorous skills. Being agentic and proactive is certainly a necessary condition when it comes to “building” in the age of AI. But I would argue that early training in verifiable domains is more important than ever.

First, a substantive reason: verifiable domains are really good training grounds for reasoning. There’s an analogous effect in AI model training. Some of the strongest improvements in reasoning have come from reinforcement learning on tasks like math and programming. This is implicitly believed in quant trading: manual trades and non-stationary portfolio decisions are notoriously difficult to backtest. They have no meaningful estimation statistics available and any quantitatively modeled approaches are contrived and empirically unverifiable. Yet, the quant trading population is constantly selected from Olympiad medalists, programming competitors, chess champions, and strategy game champions. Obviously, this is confounded by this population’s innate ability, or obsessiveness and work ethic, or educational backgrounds. But it seems plausible that reasoning learned in highly-structured, verifiable environments can transfer to less-verifiable reasoning tasks. And it’s a supported theory amongst top competitors in a very competitive industry.

A reason that verifiable tasks may be useful training applies to both models and children: verifiable tasks have the highest sample size and feedback is quick and deterministic. They can have an infinitely high and continuous bar for rigor, much like long-horizon tasks for AI, which can keep ramping up in difficulty. Similarly in math, there is a continuum for rigor – AMC, AIME, USAMO, MOP, IMO. Long-horizon tasks can become so difficult that we need to reward the machines with partial credit, because if binary success rates are low, the information content from that binary reward is low as well. Partial credit and intermediate rewards, and gradual ramps in difficulty all make a particular task a useful learning tool. When kids train to “build” just by building, à la the Horowitz Andreessen Academy, none of these criteria are satisfied. The difficulty ramp can be discontinuous and the outcomes are extremely probabilistic and low information content, making improvement hard. Without a smooth gradient, how do children know how to search for improvements?

Second, a signaling reason. Deedy Das posted an informal analysis suggesting that IMO, IOI, and IPhO medalists are thousands of times more likely than the average person to become venture-backed unicorn founders. The number itself should be taken cautiously, and the result is obviously heavily confounded. But for the signaling argument, this matters less. An Olympiad medal is useful because it gives a lot of information about human traits that are difficult to observe directly and deterministically.

As AI makes tasks easier and cheaper, competitive effects will make outcomes like college admissions and founding startups a less informative signal about human ability than ever. Any essay, any extracurricular, will be boosted by AI, and no longer cleanly dependent upon verifiable human merit. In this world, skill in verifiable domains like the Olympiad subjects will become more important, not less, as the only ways to robustly distinguish human marginal value add. Even if the Horowitz Andreessen Academy wants to front-run college, I bet their cohort will be disproportionately dense in people who are proven in verifiable domains, just people who have done so in high school rather than college. This academy will be exploitation, in the explore/exploit sense, for kids that are already fully-formed, not a training ground.

While AI makes the value of a verifiable output itself lower, the information quality in training and signaling with rigorous education in verifiable domains will become more valuable than ever. But, one difference I posit with the old world, is that the rigorous education should come earlier and be more diversified amongst different domains. If the goal is transfer rather than caring about the output, then a rotation of math, programming, chess, experimental science, and other rigorous activities will expose children to different kinds of reasoning skills: search, abstraction, proof, planning, statistical intuition, etc. The earlier a rigorous and diversified education is mastered, the earlier these fully-formed children with transfer-learned reasoning can move onto applying that to endeavors where the output is actually economically meaningful, like “building.”