Fantasy baseball: how good is PFM at using PECOTA?

Although I’m a baseball fan first and a fantasy baseball player second, I do play fantasy vigorously. As I’ve transitioned into being a professional computer/data guy, I’ve used baseball as a way to learn tools and practice “data science” skills. One nice feature for a beginner is that there’s enough data-cleaning work to give me a small taste of what it’s like, but not nearly as much as in other domains. With all the excellent data sets out there, I can get to the fun stuff quickly without completely neglecting data collection.

Valuing fantasy baseball players divides, as a first approximation, into two projects: predicting players’ outputs and using those outputs to generate fantasy values. I know many serious fantasy players do their own projections, but at least for now I’m guessing I can help myself more by working on the second step than on the first. I’ll make some obvious adjustments for injuries and similar news, but that’s about it for now.

That second step, generating fantasy values from projected statistics, is complicated, but my league’s (fairly common) structure lets me make certain simplifications. It’s a head-to-head league in which each of twelve categories is worth a point, and whoever gets the most points wins.

The suppositions I’ve chosen to make so far are:

(1)    That a player’s fantasy value is, for each category, proportional to the sum of the probabilities that his contributions to a given category cause a fantasy team to win that category.

(2)    That this value is proportional to the standard deviation of the distribution of team scores in that category in the fantasy league.

(3)    That a player’s output in a category can be represented by his average projected weekly output in that category.

(4)    That the means and standard deviations I’ve derived from my league’s last two years of results are a fair guide to how the league will play this year.

One could criticize these suppositions in many ways. The last one, for example, is particularly suspect. It might be better to create a model based on 2014 projections and estimate the values that way, but there are some advantages to my method. One advantage is that it was great fun to scrape all the results from (Many complications arose here, but BeautifulSoup and Requests were enough for the bulk of the work.)

(2) and (3) are similarly approximate. Supposition (2) might cause me to overvalue Billy Hamilton, for example, who will get many of his stolen bases when his fantasy teams have already locked up the category for the week. Using supposition (3) leads me to ignore differences in variance of outcomes between players, which is probably an oversimplification. And criticisms like these could be multiplied.

All that said, I think that a very useful first approximation results from this kind of strategy: use the projections to guess at what a player will contribute every week, use standard deviations to estimate the number of category wins that those contributions will add, and add those category-wins up.

Nothing I’ve described is statistically or technologically difficult, and I’ve chosen a similarly simple-but-reasonable method for adjusting player values at each position. (Estimate how many players n will be drafted at each position and set the n+1th-best player at that position’s value to zero, adjusting everyone else accordingly.)

My results are already surprising: they differ significantly from Baseball Prospectus’ Player Forecast Manager. What’s surprising about this is that I’m using the same PECOTA projections that BP’s tool is, so that any differences in our fantasy valuations are not due to differences in projections, only to differences in deriving fantasy value from projections. (I’ve also configured the tool to use the same categories that my league uses.) Even differences in determining positional value can’t account for too much of the differences, because even filtering each set of results by position leaves wide disparities. (I can’t give many examples here, because PECOTA and PFM data are subscriber-only, but two of the many claims that result from my method are that BP is--for my fantasy league--overvaluing Starlin Castro and undervaluing Jason Kipnis.)

Perhaps metarationality requires me to defer to BP completely here; it seems clear that I at least ought to partially adjust my beliefs once I see that a company of very smart people, plenty of whom get hired away by MLB teams, disagree with me. However I decide to weight my own results and PFM's come draft day, right now I’m left with a disjunctive conclusion. I think one of these must be true:

(1) BP’s calculations are way better than mine. This would be interesting as an example of a situation where a sensible basic method diverges enormously from the best method.

(2) The only major league feature that PFM does not explicitly account for--that my league is head-to-head--makes a big difference to player values. This would be interesting as a fact about fantasy baseball.

(3) BP’s calculations are not way better than mine. This would be interesting for many reasons. Although I’m not at all sure that my algorithm is better than or even roughly as good as theirs, this is a possibility worth at least considering. If it were true, it might mean that my league is idiosyncratic and that I’m doing a good job of adjusting to it; it might also mean that I got lucky to choose a different and better basic approach from PFM’s.

There’s much more to say here, but I’ll say it better after I work harder to find patterns in the differences between my results and PFM’s, and after I work to improve my algorithms. (The answer to the title question is currently: I don't know.) For now I’ll simply suggest that serious fantasy players might do better to spend less time projecting players and more time valuing players based on projections.