Generative AI and the psychology of failed experiments
Deciding that an experiment has failed, and throwing away the work, can feel bad. I suspect that it often feels worse to throw away a day of work than a year of work. Why might this be true? There are some psychological reasons:
- A year-long project probably felt successful, or at least partially successful, for a while.
- Or, at least, it felt real. Doing something for a year is satisfying and looks significant. There's a floating1 anecdote about a writer who says that he spent all morning inserting a comma and all afternoon taking it out. Many people find it amusing and a bit sad to think of undoing a morning's work. But if the same person writes a whole bad novel, it's often taken to indicate a certain phase of a brilliant artist's career; it makes their corpus feel more weighty and respectable.
- Many year-long projects only take a fraction of our day-to-day time, whereas a day-long project is often our entire focus for the day. A failed one-day project can therefore feel like a complete waste of time in a way that a failed year-long project doesn't.
If these projects happen in the context of a job, there are reasons of organizational behavior, too:
- A year-long (or longer) project is officially approved in ways that day-long projects might not be. I've never had a manager reprimand me for working to deprecate years of work, but I've definitely gotten feedback for small-scale failed experimentation, even if they had approved it.
- A programmer experimenting for a day is unlikely to have consulted much with product managers and other cross-functional partners. This kind of experimentation can therefore be disruptive to organizational politics.2
- When a project fails, it tends to be very bad for only a few people, if any. To vastly over-simplify: if one of 50 people on a big failed project will be fired for it, 98% of them will be OK. If you do a one-day experiment yourself, there's only you to blame for it.
All of this is much more relevant in a world with generative AI, where you can make enormous progress in one focused day of work, and where it often makes sense to experiment a lot more. Some of the difficulty in adjusting to generative-AI programming stems from this phenomenon, I think. There's a dilemma: we're either experiencing this particular kind of feel-bad a lot more, or else we're avoiding the feel-bad by failing to abandon experiments we ought to abandon.3
I am not sure how best to improve at declaring this kind of small-scale failure while staying appropriately persistent. I suspect it varies a lot across people and contexts. This is another respect in which my poker experience feels relevant; I often find myself remembering that a good way not to go on tilt after a loss is to first curb one's exultation at winning, because it trains one not to over-react to the short term, but that's a different post.
I don't know exactly what psychological adjustments are best for a world with many more short-term experiments, but I'm confident that those experiments are far more powerful than they used to be, and that we should be working to build all the technical and psychological skills they require.
It's attributed to Flaubert, Wilde, and others.↩
I don't mean to imply that all programmer-level experimentation is great, that we'd get so much more done if those pesky non-technical partners would get out of our way, and that this is doubly true in a gen-AI world. Aspects of that are sometimes true, but here I'm only saying that the politics of day-scale work are in general different from that of year-scale work. (I'm also not at all sure that this sort of experimentation is more threatening to non-programmers than to programmers, but that's another post.)↩
I suspect this is an underrated cause of "technical debt," and I'm still much more optimistic than others on this subject.↩