Comparing 2017-18 Preseason Rankings

Five teams received first place votes in either the preseason AP or coach’s poll last year; Villanova was not one of those teams. But Ken Pomeroy's computer pegged Villanova as the best team in the nation in the preseason. Perhaps this is a reminder to poll voters to avoid group-think and not be afraid to stray from the consensus. If Ken was best at the very top last year, John Gasaway had the most accurate Top 50 last year. And for the third time in four year's Sports Illustrated's projection of all 351 teams proved to be the most accurate for the full rankings.

Relative to the baseline (simply running the final Pomeroy rankings from the prior year), David Hess of Team Rankings, Ken Pomeroy, and John Gasaway all improved the average team's ranking by 9.5 to 9.9 slots. Clearly, their model's moved the needle quite far in the correct direction. The SI model, a combination of effort from me (Dan Hanner), Chris Johnson, and Jeremy Fuchs ended up being the most accurate, improving the average team's ranking by 11.65 spots relative to simply running the prior year's final standings.

Rank, Publication (Author), Improvement over Baseline

1^st, SI (Hanner, Johnson, Fuchs), +11.65

2^nd, Team Rankings (Hess), +9.90
3^rd, Pomeroy Preseason, + 9.55
4^th, ESPN (Gasaway), +9.53
5^th, ESPN BPI, +6.70
6^th, Torvik Rank, +6.68
7^th, CBS Sports, +5.49

Full details about these numbers are found at the end of this post.

For the last seven years, I have had the honor of ranking 351 teams for either ESPN the Magazine or Sports Illustrated. I am thankful to everyone who gave me this opportunity and worked with me along the way. I still remember when my late grandfather gave me my first subscription to SI. I fell in love with the SI preview editions and it was a dream come true to contribute to these for so many years.

But it shouldn't be a huge surprise when I announce I am stepping away from this process. After I gave up my column on RealGM.com a few years ago, I have been cutting back on how much college basketball I watch. And I don't feel it is appropriate to rank all these players and teams as I continue to cut-back on how much I watch.

Our SI player level projections have always been partly based on crunching the numbers, and partly based on scouting. We would take input from coaches and beat reporters to tweak the rotations for teams (usually the minutes for players, but also sometimes ORtg and usage rates), based on what people were seeing in practice. We got to the point of even including summer tour data in our analysis. It was awesome to see the results every year, but it became almost too time consuming to include all these inputs. Once you allow the ability to make these manual scouting adjustments, you essentially add a potentially unlimited amount of work to the project.

No one has quite matched what we have done, but I also go back to the above numbers and conclude that the effort is not quite worth it. At SI we were able to improve on what Pomeroy, Gasaway, and Hess have done, but not to a huge degree. And there remains substantial noise, substantial uncertainty each year, that will probably never be overcome, due to the fact that college basketball players are at a very developmental point of their lives.

You probably assume my departure from SI is related to all the media shake-ups that have happened. I would say only barely so. Sure, if there were huge amounts of money being thrown around for basketball columns, I might stick around. But this is more about personal time than money. I was fortunate to do this as a part-time job for as long as I did.

I also don't assume this will be the end of my sports writing career. I hope that some day I have the energy to blog every day of the NCAA tournament again. I hope that I can do more to publicize sporting events the public is missing out on. (For example, I think it is a crime that more people did not hear about this year’s NCAA women’s gymnastics final. Oklahoma posted a dominant score only to see UCLA’s red-shirt senior Peng Peng Lee close the meet with back to back perfect 10’s to give UCLA the National Title by the slimmest of margins. It was easily one of the most compelling sports moments of this entire year.) There are still stories to tell, and I am not done telling them. But I am done ranking college basketball teams for now.

...If you are sad to see the player projections go away…

Please continue to follow the work of Bart Torvik. While his team model did a little worse last year, his website started to show some player projections last year, and I feel he is on the brink of greatness.

...If you want some input on doing this yourself…

I continue to make two points. First, scouting matters. Follow the twitter feeds of beat reporters. College players develop rapidly and what beat reporters see happening in practice is real.

Second, I continue to believe that the AAU data is getting more and more accurate at predicting college. Trae Young wasn’t a Top 10 recruit, but his AAU data was off the charts. He had a 32% usage rate and 130 ORtg on the AAU circuit. His success in college should not have been a surprise. Whether it translates to the NBA is another question, but when projecting college basketball, don’t overlook the statistically dominant AAU players who don’t have NBA size or quickness. If someone is efficient and high volume on the AAU circuit, they can play college basketball.

Final Details
 

And for you raw number nerds, here is how I evaluated the preseason rankings from last year.  For each set of preseason predictions, I take the difference between each team’s preseason ranking and its final Sagarin ranking, take the absolute value of each difference, and add up the total over 351 teams. (I am using the final Sagarin rating rather than the final Pomeroy ranking, since one of the things we are evaluating is Pomeroy’s system. Nonetheless, if we used the Final Pomeroy instead, the results are very similar.)

I start with a baseline which is simply the final Pomeroy ranking of teams 1-351 from 2016-2017, and I take the difference between each ranking system and this baseline in the table above.

Here was the raw data before the calculations:
 

2017-18 Preseason Predictions:

Comparing 2016-17 Preseason Rankings

Quick FYI: Our preseason SI college basketball material begins to roll out next week.

One piece of business I wanted to take care of before the rollout of this year's material, is a comparison of last year's preseason rankings.

2017 was a weird year. Every ranking system under the sun had Duke #1 and they did not live up to the hype despite immense talent. Meanwhile many of the top freshman (I’m thinking of Dennis Smith Jr. and Markelle Fultz) were great statistically, but failed to elevate their teams. Meanwhile, Ken Pomeroy added transfers to his model, but it turned out to be a year where transfers were not as impactful as usual, and Pomeroy was actually hurt by this addition in a lot of places. (See Syracuse). Finally, ESPN launched a second preseason rankings, the BPI preseason rankings, but the new system actually performed worse than the system their own ESPN Insider John Gasaway put online at the same time. Gasaway's preseason 351 crushed the BPI preseason 351.

Now, some people will look at the variance in college basketball and say that predicting the season is a fool's errand. And while there is always a lot of uncertainty, that doesn't mean that things like star ratings and AAU stats don't have some predictive power. I happen to believe that all of these rankings are useful, and together they paint a fair picture of preseason expectations. In fact, I personally consider the CBS rankings, that have fallen at the back of the pack in recent years, to be among the most important because they are based on coaching interviews and opinions, and that's an important additional data-point that many of the similar statistical systems don't catch.

As you will see below, last year our SI rankings beat CBS and ESPN again, so of the major websites, we won for a third year in a row. But the folks behind Torvik Rank actually took the top spot this year. After finishing 5th in 2016, I'm not quite convinced Torvik Rank has found the special sauce yet. I'd like to see a little more consistency first. But after last year, I highly recommend you follow them and read their work. Some of their ideas for evolving coach effects, i.e. allowing for the possibility that Thad Matta and John Thompson III got worse over time, turned out to be an important part of Torvik Rank winning last year. http://adamcwisports.blogspot.com/2015/09/t-rank-2016-preview-nuts-and-bolts.html

OK, so now onto the numbers. In the table below, I compared Sports Illustrated preseason rankings, the ESPN preseason rankings by John Gasaway, the ESPN BPI preseason rankings, the CBS Sports preseason rankings, Ken Pomeroy’s preseason rankings, David Hess’s preseason rankings, and the Torvik Rank preseason rankings.

Then I calculated the total absolute error in each ranking system. The total absolute error is found by taking the absolute value of the difference between each team’s preseason ranking and the end of season Sagarin ranking and adding up the total.

For the end of season rankings, David Hess asked me to use Sagarin instead of Pomeroy so we were not using Pomeroy to score Pomeroy, but I actually ran the numbers both ways and it didn’t make a major difference this year. The astute reader will notice that switching from a Pomeroy system to a Sagarin system did raise David Hess’s ranking in 2016, however. (I kid, I'm sure that was unintentional.)

This is certainly not the only way to compare the rankings. You may prefer to look at NCAA bids or conference titles or something else. But if you care about where every team is ranked, last year Torvik Rank finished first and Sports Illustrated finished second:

I have intentionally left John Gasaway's rankings out of the second table, since they were only available behind a paywall, but I can assure you, he did in fact finish 3rd.

Onto the new season!

Returning Minutes and Number of Players Who Were Former Top 100 Recruits

In our SI projections, we project every player and lineup to get our team projections.

But I still get lots of requests for a list of returning minutes. That isn't a direct input into our model, though it is something I can easily calculate with the roster data I have.

I cannot say that these numbers will be 100% accurate. We typically don't pull walk-on data unless those walk-ons are expected to play a lot. And we have to make some decisions about certain players. For example, this assumes Coastal Carolina's Shivaughn Wiggins will be able to return in the second semester. But it should be mostly accurate.

I also list the number of RSCI Top 100 recruits on each roster. This includes current RSCI Top 100 freshmen, former RSCI Top 100 recruits (who are now sophomores, juniors, and seniors), and players that we think were incorrectly ranked by RSCI because they changed classes.