Whither teacher-student models


If you haven't already tried an agentic workflow: congratulations on not running up an unexpectedly high AI compute bill!

Back when I worked at Microsoft, circa 2018 or so, I was of the firm belief that the future for corporate ML was Teacher-Student Models. Basically, once the, what-became-known-as hyperscalars' what-became-known-as Frontier Models reached Economic AGI the competition between ML-driven competitors would reduce to the economic of inference. In any field, Frontier Models would keep advancing the competitive state of the art, so every corporation in Field X would be working with very similar Frontier Model-derived solutions, for which everyone would be paying very similar amounts to very similar cloud-inference providers.

Where Spacely Space Sprockets could compete with Cogswell Cosmic Cogs was in cost of inference: rather than using the concept space of a Frontier Model (Byzantine History, Videogame Strategy, Cog-Manufacturing, etc.) you would pare down to a smaller, proprietary model with less knowledge of early South American Literature. Where your competitor using the Frontier Model would incur $1 in inference costs, your use of your distilled model would cost you $.10. For "distillation" is what the concept became known as, and to the extent that anyone at Microsoft vaguely remembers me talking about Economic AGI competition, they'll remember me going on about "Teacher-Student models," and not "distillation."

"Distillation" though, is nowadays mostly used in the phrase "distillation attack," and the hyperscalars now have anti-distillation defenses. Perhaps the duration of the Economic AGI epoch is destined to be so short before ASI World Dominance eliminates the quaint concept of "competitive firms" that it doesn't matter.