Wednesday, October 4, 2017

Comparing 2016-17 Preseason Rankings

Quick FYI: Our preseason SI college basketball material begins to roll out next week.

One piece of business I wanted to take care of before the rollout of this year's material, is a comparison of last year's preseason rankings.

2017 was a weird year. Every ranking system under the sun had Duke #1 and they did not live up to the hype despite immense talent. Meanwhile many of the top freshman (I’m thinking of Dennis Smith Jr. and Markelle Fultz) were great statistically, but failed to elevate their teams. Meanwhile, Ken Pomeroy added transfers to his model, but it turned out to be a year where transfers were not as impactful as usual, and Pomeroy was actually hurt by this addition in a lot of places. (See Syracuse). Finally, ESPN launched a second preseason rankings, the BPI preseason rankings, but the new system actually performed worse than the system their own ESPN Insider John Gasaway put online at the same time. Gasaway's preseason 351 crushed the BPI preseason 351.

Now, some people will look at the variance in college basketball and say that predicting the season is a fool's errand. And while there is always a lot of uncertainty, that doesn't mean that things like star ratings and AAU stats don't have some predictive power. I happen to believe that all of these rankings are useful, and together they paint a fair picture of preseason expectations. In fact, I personally consider the CBS rankings, that have fallen at the back of the pack in recent years, to be among the most important because they are based on coaching interviews and opinions, and that's an important additional data-point that many of the similar statistical systems don't catch.

As you will see below, last year our SI rankings beat CBS and ESPN again, so of the major websites, we won for a third year in a row. But the folks behind Torvik Rank actually took the top spot this year. After finishing 5th in 2016, I'm not quite convinced Torvik Rank has found the special sauce yet. I'd like to see a little more consistency first. But after last year, I highly recommend you follow them and read their work. Some of their ideas for evolving coach effects, i.e. allowing for the possibility that Thad Matta and John Thompson III got worse over time, turned out to be an important part of Torvik Rank winning last year.

OK, so now onto the numbers. In the table below, I compared Sports Illustrated preseason rankings, the ESPN preseason rankings by John Gasaway, the ESPN BPI preseason rankings, the CBS Sports preseason rankings, Ken Pomeroy’s preseason rankings, David Hess’s preseason rankings, and the Torvik Rank preseason rankings.

Then I calculated the total absolute error in each ranking system. The total absolute error is found by taking the absolute value of the difference between each team’s preseason ranking and the end of season Sagarin ranking and adding up the total.

For the end of season rankings, David Hess asked me to use Sagarin instead of Pomeroy so we were not using Pomeroy to score Pomeroy, but I actually ran the numbers both ways and it didn’t make a major difference this year. The astute reader will notice that switching from a Pomeroy system to a Sagarin system did raise David Hess’s ranking in 2016, however. (I kid, I'm sure that was unintentional.)

This is certainly not the only way to compare the rankings. You may prefer to look at NCAA bids or conference titles or something else. But if you care about where every team is ranked, last year Torvik Rank finished first and Sports Illustrated finished second:

I have intentionally left John Gasaway's rankings out of the second table, since they were only available behind a paywall, but I can assure you, he did in fact finish 3rd.

Onto the new season!