The Marco saga as a case for 10x engineering
Here is this week's episode of Accidental Tech. There's more discussion of the episode-transcription project that fascinates me. In short: Marco's quest to add automated transcripts to his podcast player turned into a year-long (so far) project to set up a fleet of Mac minis and run them for this purpose.
Besides being a fun story, this is a great case study to inform the old "does 10x engineering exist?" debate. I am on Team Yes It Does (see here and here). Here's an extremely partial list of topics relevant to the Marco Transcript Saga:
- Physical data-center infrastructure;
- Fleet management software;
- Queueing and retries;
- Evaluating current and possible future APIs for a task;
- Choosing and defining a data structure for the central domain object;
- Reconciling data from different sources purporting to describe the same thing;
- Weighing the value of choosing the currently best option and maintaining the adaptability that will allow you to migrate to potential better future options;
- Any number of UX and data-display questions.
These episodes vivify the lesson that good decisions in any of these have large payoffs, and a single wrong step can carry enormous costs.1 More importantly and less obviously, we see how combinations of these decisions create and avoid whole categories of work and whole recurring tasks (the sort of thing you'd be tempted to assign a full- or part-time person to).
All sorts of opinions and ideas fly through my head as I listen to these episodes: they're informed by 13-plus years as a software professional and much more time as a serious user and programmer of computers. Even so, in my imaginary design conversations with Marco, I find myself just deferring to him on decision after decision, because he simply knows so much more than I do about Apple hardware and audio software.
Over and over and over again, we see decisions being made that affect the quality of the output and the system's maintainability and cost, in ways that are likely to compound over time. A hypothetical cloud-based solution that makes choices of API and model highly modular is (i) reasonable a priori, (ii) totally different from what Marco did, and (iii) probably either a lot better or a lot worse over the long term. However one might choose to measure engineering output on this project, it seems inescapable to me that an excellent engineer's output will be more than 10x an average one's.
For example: if I understood correctly, a bug in some queueing or processing logic caused Marco to buy perhaps 10 extra computers. And who knows how many extra computers he avoided buying by avoiding other bugs?↩