See Me Squeegee

Instead of an apology/excuse for not having posted in months, let’s get this first-post-in-forever started with a different cliche: year-in-review, albeit with this blog’s own cliche, the rant.

Every Christmas I spend a ton of time recapping the state of education and politics with family and friends, most of whom these days are pretty like-minded as educators and progressive thinkers.  And this time of year it’s common to take a look back and recap on the events of the year.  And through all of those discussions and reflections, I’ve come up with two rant-worthy conclusions:

1) All that glitters is not (Nate) Silver

2) One of the main differences in philosophy among most of us is short-term vs. long-term thinking…and I’ll openly campaign for long-term

A big portion of this comes from discussions about educational reform and the charter school and common-core movements, and spills over into much of what I’ve seen/heard in business and decision making.  With education, two major pet peeves include:

-An emphasis on “raising test scores”, tying school funding and teacher pay to those test scores, and trying to keep up with the world as it pertains to those test scores, and

-The statistics that charter schools are outpacing traditional public schools

Why those stats bug me – just a moment.  But here’s why they’re so popular…Moneyball and Nate Silver.  Those two things (Moneyball the movie…what last year?  And Silver all this year) have been perverted quite a bit from their true nature, which is absolutely intelligent and correct,  to the pop culture version of them:  “use statistics to make good decisions!”.  And the pop version is just about dead wrong, because neither Moneyball nor Nate Silver claimed to be the first to consider statistics – they’re just the  best in the game at looking at the most valuable statistics (and thereby writing off the less valuable statistics) when it comes to making decisions.

Here’s a quick synopsis of Moneyball, the book:

For years, baseball decision makers have relied heavily on a certain set of statistics, including RBIs, batting averages, ERAs, and pitcher W-L record.  But as a cash-strapped Oakland A’s franchise realized that it couldn’t afford to compete with the Yankees, Red Sox, and other big-market teams to buy the players best at those stats, it had to dig deeper and found that:

-Batting average is a little overrated, as it doesn’t factor in walks (or hit-by-pitch), and since most hits are singles anyway why does it really matter if you walked or  singled?  You’re on base, and that’s a good thing. On-base percentage is a more useful stat, particularly because there’s value in a player who can get on base after a 10-pitch at bat vs. a 5-pitch at bat…the former makes the pitcher work harder and therefore means you get to the middle reliever (who’s usually not as good) sooner.

-RBIs aren’t as highly correlated with success as one might think, because too much of that stat is based on one’s teammates and position in the batting order, and those variables can skew the data.  A leadoff hitter won’t get many RBIs since at least 1/4 of his at bats, by nature, will not have anyone on base.

Moneyball the book was focused on finding the most predictive statistics – it absolutely conceded the fact (which the movie didn’t) that people had been using stats for a century…it just questioned whether they were using the best possible stats to make their decisions.

Similarly, with Nate Silver – he didn’t conduct his own polls, but rather created a system to determine bias in polls and reliability of polls so that he could  combine several polls in each state to find the most predictive algorithm out there.  He wasn’t the first at all to “use stats” – he just did a much better job than anyone else of determining which stats were the most meaningful, and which needed handicaps (say, the Rasmussen poll which had value but always came with a Republican bias because its methodology tended to lead toward more GOP-friendly responses).

So back to education – the problem with what’s happening with our use of test scores and charter school trends is that we’re blindly “focusing on stats” without going Nate Silver / Moneyball and trying to determine what those stats really tell us:

-Do the standardized tests that compare us against China and India really measure what we as a nation feel is most important?  Do they have biases that favor some students over others?

-Are the tests flawed?  Do they measure (as I predict) what’s “easy  to measure” over what’s truly valuable?

-Which kids take the tests (and which don’t)?  And what are their incentives?  In Michigan, the MEAP statewide test comes with a carrot – those who pass get (at least they did last time I looked) $1,000 toward college…but that assumes that the kids on the verge of passing are even interested in college.  And my contention – if you’re struggling with state-level proficiency, college really isn’t the carrot that motivates you, at least for a lot of those students. So why do they  care?

In the realm of the charter school, their charters essentially won’t allow them to make admission based on merit, but rather require that every student who chooses to apply has an equal chance of admission.  But here’s the catch – there’s a pretty substantial hurdle to jump to apply to a  charter school and it’s one that gives those schools a huge leg up on the public schools: you have to actually apply.  And so that process of filling out and submitting a form will weed out anyone whose family doesn’t value education enough to fill out the form.  Which means that charter school kids are much more likely to value education and have parents to supplement their in-school experience. It’s a small point but a huge one – public schools are guaranteed to have ALL the kids whose parents don’t fill out the form (and some whose parents do), while ALL charter school students have parents who at least care enough to fill out the paperwork.  Charter schools therefore SHOULD have a higher set of test scores – it’s not surprising that they do, but the masses like the political appeal of “look, they’re doing a better job – just look at the difference in test scores”.

So my rant – which may need to be continued since I’m running out for dinner with  two of my like five loyal readers, the main reason I finally got up and posted again – is essentially this:

Just because a study uses numbers doesn’t make it Nate Silver.  In fact, most conclusions based on numbers are crap – they’re misusing numbers or drawing conclusions that don’t logically follow.  What makes Silver and Billy Beane such visionaries is that they looked behind the numbers, then carefully selected the types of numbers that would lend a high probability of success in prediction.  But the legacy of Nate Silver’s 2012 and the movie Moneyball is quite often used for the opposite purpose – “look, we have numbers…we must be right!”.

Mark Twain is famous for a quote:  “There are three types of lies: lies, damned lies, and statistics”.  If we’re using statistics to improve education we should at least heed the advice of one of the authors that we ALL read in school.  Moneyball and Nate Silver are NOT about “use statistics”, but rather “look behind the numbers to find the most relevant and predictive statistics”.  And if the tests we’re using to measure education lead us to promote and follow leaders who can’t see that ever-important distinction, then we’re absolutely using the wrong statistics.