Nate Meyvis

Scripting, testing, and dry runs

Here is Henrik Warne on using dry runs and adding --dry-run options. It's a good piece; here are some related thoughts.

A --dry-run flag is often used right before something dangerous or irreversible happens, and switches the behavior from "do the thing" to "tell the user about the thing." This is a standard technique, but it's worth considering alternatives. In particular, you can create a helper function (get_all_pending_file_behaviors() or similar, in Warne's example) to list all those actions, and test and call that function directly instead of using --dry-run. Instead of calling dangerous-script --dry-run, the user can run a separate script (or pass some other flag to dangerous-script) that reports the output of get_all_pending_file_behaviors directly.

Making helpers like get_all_pending_file_behaviors allows you to test its behavior more cleanly and thoroughly. Most often, scripts are tested by manual examination, whether the examination is of a dry run or of the script's effect on a lower environment. Here as elsewhere, it's wise not to rely on manual examination unless it's combined with automated testing.

Engineers routinely overestimate the effort of doing the work of factorization and testing. It's usually fast, and modern tools are making it even faster. This effort can feel purposeless, but the long run comes quickly, and you're likely to want to change and verify this code more than you initially think you will.

Even if you do have to spend significant time making your code testable and testing it, it's often still worth doing. When you want to do a dry run, you're usually modifying data or doing something else you don't want to have to undo.1 As the cost of mistakes rises, so does the benefit of investing effort in preventing them.

You can use the testing-helpers technique together with a --dry-run flag. The best state is one in which the main part of the script isn't doing much except calling helpers and doing the main dangerous things (functions in Web frameworks are a useful analogy here), but you can still put your disjunctive dry-run logic in that main part.

Encapsulation and modularity apply even to small and short-lived projects, like scripts. These principles lead us to factor our code and test it in other contexts, and we should do the same thing with scripts.

  1. It tends also to be the sort of thing you don't mistakenly want to believe you've done. False negatives and false positives will both hurt.

#nuts and bolts #software