New theories and new software
Here's an excerpt from Dwarkesh Patel's interview with Terence Tao that's stuck with me:
Often, the ultimately correct theory initially is worse in many ways. Copernicus’s theory of the planets was less accurate than Ptolemy’s theory. Geocentrism had been developed for a millennium by that point, and they had made many tweaks and increasingly complicated ad hoc fixes to make it more and more accurate. [...] When you only get part of the solution, it looks worse than a theory which is incorrect but somehow has been completed to the point where it kind of answers all the questions.
I often think about this now when I'm doing migrations (which is often). The new system can be more correct and still perform worse, because the old system has more fitting (or overfitting) to the relevant tasks. Some of these legacy-system features tend to be good and noble: for example, handling observed (if unofficial) bad inputs that are hard to anticipate a priori. Some are not: for example, handling a race condition by making sure that one subsystem stays very slow.
So, when you're evaluating old and new versions of a system, and the old version is doing better in some respect, it's good to ask whether this is a deep advantage of the old system or just a sign that the old system is better tuned than the new one. A few other notes:
- This can be a lot less obvious than initial examples make it seem, especially since the new system often won't be handling real data or working at scale.
- AI is very good at finding and porting the "good" fine-tuning, but if the structures of the old and new systems are different enough, you might need to prod it a bit.1
- Here is a link to Peter Naur's "Programming as Theory Building," which might have influenced me a bit more than I've known. I have no actual allegiance to Naur's view, and I'm not that Naur is using "theory" the way we find it in "Copernican theory," but I'm glad I read Naur's paper a couple years ago. (It's short.)
AI is so good at this that I suspect it will be hard to convince people just how influential this Joel on Software piece was.↩