Nate Meyvis

AI and some new division-of-labor problems

Here are some of the many software-design questions on my mind:

  1. I run a spaced-repetition flashcard site, and flashcarding is obviously a good fit for LLM assistance, but questions of how to implement that LLM assistance are sometimes awkward. Some functionality should clearly be server-side, automatic work (e.g., auto-fact-checking and auto-category-tagging of new flashcards). Some of it, on the other hand, is computationally expensive, hard to predict, and user-specific. It makes more sense to offload the specification and execution of that latter work to the user. So, for example, I recently rescheduled a backlog of overdue work according to a complicated, specific-to-me algorithm, and I've been doing a lot of other flashcarding projects that make a lot more sense in the "let users try their own crazy ideas" paradigm than they would in the "implement every possible crazy idea and make them available to all users" paradigm.
  2. Because AI enables blog-post-driven development, it seems I now also run a family research curation site. Some of its LLM-related functionality is cheap and predictable, and makes sense to implement server-side (e.g., "are there any obvious errors or research gaps in this project?").1 But there are an unlimited number of hard-to-predict sub-projects and questions a user might have, and human assistance is still necessary for most research work, so there's no good alternative to enabling users to do arbitrary LLM-enabled work themselves.

I've had analogous problems in other domains.2 The forces making it more attractive to implement AI-powered tools as part of the service often simultaneously push us to simply give our users the means to plug in their own AI tools:

  1. It's less and less cost-prohibitive to pay for LLM calls in the service, but more powerful versions of those tools often (i) give even better results and (ii) do have significant costs, which it often makes more sense for users to bear directly.3
  2. There are more and more things it makes sense to ask AI to do, but in at least some domains, for every predictable, generalizable AI-assisted task, there will be many unpredictable and ungeneralizable, but legitimately useful, AI-assisted tasks users will want done.
  3. AI makes it feasible to create Web services for things that used to be spreadsheets or emails, but these things tend to be irreducibly connected with other services and data (email, Google Drive, or whatever else). It's often possible to build service-side integrations with those, but (i) these have limitations and (ii) many users find them confusing and hard to manage. Giving your users the ability to combine their own data with your service in arbitrary ways ("here are my notes from the trivia season; do I have flashcards covering the questions I missed?") is often best done by the user, on the user's computer.

When I think about this, I'm reminded of an old question poker players like to debate: does pot-limit Omaha have more luck or more skill than no-limit hold'em? The excellent Ben Morris liked to answer that it had more luck and more skill. AI-forward services, it seems to me, have both more reason to do things themselves for the user4 and more reason to offload work to the user. If this is correct, implementing this balance,5 and choosing what goes where, is a newly important skill in software.

  1. As of this writing, I haven't actually implemented this yet. It could happen this hour, today, or not at all. Time estimates are hard, and the AI era makes some of them harder: easy but low-priority (in the context of my whole life) tasks are tough to reason about.↩

  2. E.g., recipe management, but note that I've deprecated that project, so please don't try to sign up.↩

  3. It's of course possible to pass charges through to users, offer subscription tiers, and so on, but this all adds complexity, and model capabilities and costs are changing so quickly that it's hard to implement these in a way that aren't immediately obsolete.↩

  4. Update: I have now implemented automatic, LLM-driven updates in TurboTree, while I was writing this blog post. The process of writing the first footnote shamed me into doing it. The feature seems to be working well so far.↩

  5. ...whether that's with MCP or by other means. Note that those posts were written a long time ago, and MCP is treating me just fine these days, but it's still a complicated subject I hope to revisit soon.↩

#generative AI #software #system design