Probability, miracles, and baseball

It's Opening Day, and it mustn't pass without mention here at QoD, especially since probability, randomness and the supernatural are such central topics around here.

Manny connects, game 2 in Japan. Image from Boston Globe online.

I've already confessed that Stephen Jay Gould is one of my favorite authors, and some of his essays I mark for repeat visits. ("The Most Unkindest Cut of All", in Dinosaur in a Haystack, is worth a trip to the library right now.) Gould was an avid baseball fan, and though he was happy to be a minion of the Evil Empire, his reflections on the National Pastime were always memorable.

Well, Gould's favorite ballplayer was Joe DiMaggio, who holds one of the most extraordinary records in all of sports. Gould wrote about this record in a 1988 book review, reprinted as an essay in Bully for Brontosaurus.

In 1941, as I gestated in my mother's womb, Joe DiMaggio got at least one hit in each of fifty-six successive games. Most records are only incrementally superior to runners-up; Roger Maris hit sixty-one homers in 1961, but Babe Ruth hit sixty in 1927 and fifty-nine in 1921, while Hank Greenberg (1938) and Jimmy Foxx (1932) both hit fifty-eight. But DiMaggio's fifty-six-game hitting streak is ridiculously and almost unreachably far from all challengers (Wee Willie Keeler and Peter Rose, both with forty-four, come second). Among sabremetricians — a contentious lot not known for agreement about anything — we find virtual consensus that DiMaggio's fifty-six–game hitting streak is the greatest accomplishment in the history of baseball, if not all modern sport.

So how should we understand this almost unbelievable feat? Gould claims that the streak is "both the greatest factual achievement in the history of baseball and a principal icon of American mythology." A great achievement because, unlike hitting a lot of home runs, or even hitting for a high average, the streak requires "unfailing consistency every day":

...a streak must be absolutely exceptionless; you are not allowed a single day of subpar play, or even bad luck. You bat only four or five times in an average game. Sometimes two or three of these efforts yield walks, and you get only one or two shots at a hit. Moreover, as tension mounts and notice increases, your life becomes unbearable. Reporters dog your every step; fans are even more intrusive than usual (one stole DiMaggio's favorite bat right in the middle of his streak). You cannot make a single mistake.

Okay, so that's why it's a great achievement; but what about the whole 'mythology' thing? Gould goes on to demonstrate the silliness of believing that DiMaggio's achievement was "better" than other (much shorter) streaks, that DiMaggio's streak was longer because of his greatness. Gould claims that we make this mistake because we just aren't wired to understand randomness and 'clumping' in random patterns.

We believe that long streaks and slumps must have direct causes internal to the sequence itself, and we have no feel for the frequency and length of sequences in random data. Thus, while we understand that DiMaggio's hitting streak was the longest ever, we don't appreciate its truly special character because we view all the others as equally patterned by cause, only a little shorter. We distinguish DiMaggio's feat merely by quantity along a continuum of courage; we should, instead, view his fifty-six-game hitting streak as a unique assault upon the otherwise unblemished record of Dame Probability.

Now, it seems to me that Gould is saying that DiMaggio's streak was well-nigh miraculous, meaning that it is so improbable that it really shouldn't have happened. Consider his (now bittersweet) coda:

DiMaggio's hitting streak is the finest of legitimate legends because it embodies the essence of the battle that truly defines our lives. DiMaggio activated the greatest and most unattainable dream of all humanity, the hope and chimera of all sages and shamans: he cheated death, at least for a while.

That's romantic stuff. And just as we're wiping the tears from our eyes, a couple of smart-ass mathematicians walk up and say, "Pull it together, dreamboat."

Yesterday's New York Times, presumably in celebration of Opening Day in the doomed House That Ruth Built, included a fascinating reexamination of the streak, in true Gouldian fashion. Meaning that the authors ran the kind of thought experiment that Gould made famous – they "replayed the tape," not of evolution but of baseball, and examined the likely outcomes. They re-created virtual baseball worlds, using the actual statistics from the history of baseball. Specifically, they created "parallel baseball universes" – 10,000 of them – and looked to see how improbable a super-streak really is.

And the answer is: not very improbable at all. Here's their punch line:

More than half the time, or in 5,295 baseball universes, the record for the longest hitting streak exceeded 53 games. Two-thirds of the time, the best streak was between 50 and 64 games.

In other words, streaks of 56 games or longer are not at all an unusual occurrence. Forty-two percent of the simulated baseball histories have a streak of DiMaggio’s length or longer. You shouldn’t be too surprised that someone, at some time in the history of the game, accomplished what DiMaggio did.

If you find this outcome disappointing, or if you feel I've robbed baseball of one of its supernatural episodes, take heart: the authors did uncover some eerie facts. First, 1941 was an exceedingly unlikely year for such a streak to occur. The vast majority of virtual streaks occurred in the decades before 1940. And there's this:

And Joe DiMaggio is nowhere near the likeliest player to hold the record for longest hitting streak in baseball history. He is No. 56 on the list. (Fifty-six? Cue “The Twilight Zone” music.)

Now, do you suppose the Discovery Institute fellows have read any of this stuff? For their sakes, I hope they like baseball. Their team, unlike their challenge to evolutionary theory, has a reasonable likelihood of success.