Last year, I set about at the start of January to document how the Pomeroy ranking are pretty useless this early in the season. But I noticed a funny thing. The early January numbers were shockingly similar in February and March. So what's going on here?
Well, last time I checked the RPI, Pomeroy, and Sagarin Predictor ratings weight the entire season equally. There are other nice ratings systems that put additional weight on the last 5 or last 10. But the RPI, Pomeroy, and Sagarin ratings do not. Thus in principle, as the sample size grows, each game provides less and less information, so that the rankings should become more stable over time.
The key difference however, is the model underlying the systems. Both the Pomeroy and Sagarin ratings can be thought of like this: Each game is a performance draw from some distribution, weighted relative to the strength of the opponent. And while some draws will be outliers, the distribution should be approximately normal and thus as the sample size gets bigger, both rating systems should converge on the truth.
But that won't necessarily happen with the RPI ratings. It doesn't matter how well a team plays in a game. Future games can have an inherently positive or negative effect. I.e., playing a great team can increase a team's RPI, even if they lose. And playing a bad team can decrease a team's RPI, even if they win. This isn't some draw from a distribution, but instead something determined ahead of time by the strength of schedule. The draws of schedule and not performance make the RPI much less stable. And we see that every year when late season wins over great teams can lead to substantial swings (30 spots) even in the final week of the regular season.
This is a long way for me to say that the Pomeroy and Sagarin ratings mean something at this point in the season. Texas and Kansas are not only preseason national favorites, they've played like it. The same can be said for Big East favorite West Virginia, annual ACC favorite Duke, and Big Ten favorite Purdue.
So what do we make of the other puzzling teams at the top of the ratings. See Minnesota, Missouri, Cal, and Arizona St. already with three losses. I think there are a couple of things to keep in mind here. Simply having good predictive numbers doesn't always mean you've had a successful year. Georgetown proved last year that they could maintain a high Pomeroy rating all year, but by the end of the year it was clear they were not a good team in tight games.
Second, having a limited number of freshman has made some of these teams appear better than they really are. I.e., these teams know how to crush the smaller schools at home, but it doesn't say that much about how they'll do on the road and against real BCS competition. I think that in particular with Minnesota. The Gophers expected to be mixing in top 100 freshman Royce White and getting him experience against some early season opponents. But with White not playing, and now "leaving college basketball" today, the Gophers have just had a bunch of veteran players playing meaningless games bashing cupcakes.
Along this line, the biggest reason the Pomeroy ratings should not be stable at this point of the year is that some teams will get better. Every year Michigan St. and Louisville seem to be substantially better in March than November. And in the case of Kentucky and North Carolina, very young teams, I don't think close early wins over Miami (OH) or Valparaiso are a good reason to be down on the season. Those teams will figure out how to dominate by March.
But there is information out there. Looking at the Big East, the current rankings are very similar to what I had in my pre-season preview. Marquette has played surprisingly well with a number of freshman. Villanova and Cincinnati are still adapting some key newplayers into the rotation. Louisville seems to be in the midst of its annual early season toe-stubbing. But little else is that surprising.
DePaul has a few quality wins, but is still clearly a bad team. Notre Dame still doesn't play any defense. My prediction that Providence could have a decent offense but no defense, holds true. And a number of the traditional names are at the top.
For all that we don't know about this season, we already know a lot. And along those lines, Ken Pomeroy has finally made the individual player stats available. The time to start commenting on who is good, and who is bad is here.