The Plural of My Anecdote is Better Than Your Bad Data: Part 1

I currently envision this as a 2 part series, but may add to it in the future when good examples pop up. In the meantime, part 1 will cover the historical and conceptual background and framework. Part 2 will apply the insights from part 1 to a couple real world examples: one in the strength training realm, and one in realm of The Science™ as seen in the covid era.

Corrupting Quotes

Almost everyone who’s argued online, especially with people who hold the ‘evidence based’ worldview, has encountered the rejoinder: “The plural of anecdote is not data.” In one of modern history’s great ironies, this rejoinder was corrupted within the lifetime of the man who invented it, Cal-Berkley Professor Ray Wolfinger (no relation), who said, “The plural of anecdote is data.” IS.

The story goes as follows:

“I said ‘The plural of anecdote is data’ some time in the 1969-70 academic year while teaching a graduate seminar at Stanford. The occasion was a student’s dismissal of a simple factual statement–by another student or me–as a mere anecdote. The quotation was my rejoinder. Since then I have missed few opportunities to quote myself. The only appearance in print that I can remember is Nelson Polsby’s accurate quotation and attribution in an article in PS: Political Science and Politics in 1993; I believe it was in the first issue of the year.”

Statistician Nate Silver riffed on the corruption of this great quote to its opposite a decade ago:

“You may have heard the phrase the plural of anecdote is not data. It turns out that this is a misquote. The original aphorism, by the political scientist Ray Wolfinger, was just the opposite: The plural of anecdote is data.

Wolfinger’s formulation makes sense: Data does not have a virgin birth. It comes to us from somewhere. Someone set up a procedure to collect and record it. Sometimes this person is a scientist, but she also could be a journalist.”

It could also be a strength coach with his hundreds or thousands of lifters compared to a few n=17 peer reviewed studies, or an interested layman observing what’s going on around him in the real world, when every newscast is interviewing credentialed experts, approved by the establishment, saying the exact opposite.

The snuck premise in “The plural of anecdote is not data”

Why did this pithy quote get turned into its opposite soon after it was first stated in around 1970? My best guess is the rise of the aforementioned evidence based worldview, epitomized by the modern peer review industrial complex. When did the idea of peer review start to become a ubiquitous, necessary precondition for any empirical truth claim, often to the exclusion of all others?

If you said right around 1970, ding ding ding! Though it had begun to percolate in the 40s and 50s, as we shall see momentarily, it really took off around 1970:

Had he said it in 1930, it probably would’ve gone around unaltered for long enough to take root in its original form. Alas Wolfinger, though a well known professor at a prestigious university, was not immune to having his original quote and idea corrupted into its opposite by the ideological zeal of the New Truth, the evidence based paradigm that began surging right around the time he first said his aphorism. And the New Truth is: If you can’t source your claim to a peer reviewed citation, your claim is garbage, and can be more or less dismissed out of hand.

This is the snuck premise behind "the plural of anecdote is not data,” and the central tenet of the modern day worldview that calls itself ‘evidence based,’ but is actually more accurately thought of as gatekeeping evidence to one narrow and unreliable domain, to the exclusion of all others.

What’s wrong with this view?

It’s hard to believe anyone who lives in the real world actually believes this, but it has somehow become the standard operating system of a lot of experts in many fields, and many of their laymen acolytes repeat it endlessly with great zeal:

Thanks Wojack, very cool

Why do I say it’s hard to believe people actually think this way? Let’s take a very brief look at some of the things this “peer reviewed citation or bust” outlook would exclude, and then you tell me whether - empirically speaking of course - it makes sense.

Inside the academy

Albert Einstein only had one anonymous peer review in his career - and the paper was rejected. This happened in 1936. Back in 1905, Einstein had published four incredible papers. One introduced the world to special relativity, another outlined the photoelectric effect — why metals spit out electrons when illuminated with light. He later won a Nobel Prize for this work. A third paper explained that Brownian motion occurs because there are molecules randomly dancing around and bumping into little particles^. In the fourth paper, he supplied the foundation for the relationship between mass and energy summarized by the world’s most familiar equation: E = mc ². Not bad for a single year. Yet, none of these were peer reviewed in the way that term is used today.

Physicist Frank Tippler, in an article titled Refereed Journals: Do They Insure Quality or Enforce Orthodoxy? writes:

All three papers were published in Annalen der Physik, one of the major physics journals in Germany. But none of the papers were sent to referees. Instead the editors—either the editor in chief, Man Planck, or the editor for theoretical physics, Wilhelm Wien—made the decision to publish. It is unlikely that whoever made the decision spent much time on whether to publish. Almost every paper submitted was published. So few people wanted to publish in any physics journal that editors rarely rejected submitted papers. Only papers that were clearly “crackpot” papers—papers that any professional physicist could recognize as written by someone completely unfamiliar with the elementary laws of physics—were rejected.

And if Annalen der Physik rejected a paper, for whatever reason, any professional German physicist had an alternative: Zeitschrift für Physik. This journal would publish any paper submitted by any member of the German Physical Society. This journal published quite a few worthless papers. But it also published quite a few great papers, among them Heisenberg’s first paper on the Uncertainty Principle, a central idea in quantum mechanics. There was no way in which referees or editors could stop an idea from appearing in the professional journals. In illustration of this, the great Danish physicist Niels Bohr said, according to Abraham Pais (The Genius of Science, p. 307), that if a physicist has an idea that seems crazy and he hesitates to publish so that someone else publishes the idea first and gets the credit, he has no one to blame but himself. In other words, it never occurred to Bohr that referees or editors could stop the publication of a new idea

This is in direct opposition to the way the modern peer review process that undergirds the ‘evidence based’ worldview works. Tippler continues:

[Niels] Bohr would not say that today. If one reads memoirs or biographies of physicists who made their great breakthroughs after, say, 1950, one is struck by how often one reads that “the referees rejected for publication the paper that later won me the Nobel Prize.” One example is Rosalyn Yalow, who described how her Nobel-prize-winning paper was received by the journals. “In 1955 we submitted the paper to Science.... The paper was held there for eight months before it was reviewed. It was finally rejected. We submitted it to the Journal of Clinical Investigations, which also rejected it.” (Quoted from The Joys of Research, edited by Walter Shropshire, p. 109). Another example is Günter Blobel, who in a news conference given just after he was awarded the Nobel Prize in Medicine, said that the main problem one encounters in one’s research is “when your grants and papers are rejected because some stupid reviewer rejected them for dogmatic adherence to old ideas.”

Pretty astounding, isn’t it? Einstein’s world-changing ideas, not peer reviewed. Heisenberg’s first paper on the uncertainty principle was published by a journal that literally published anything a member of the German Physical Society would write. Multiple Nobel prize winners had their seminal papers gatekept out of publication by peer review. Though Tippler shows that this had already started by the 50s, it really exploded in the 70s as we can see from the Ngram above.

And these are only a few examples. Much of the science found in the fundamental texts used to teach science majors in university today, comes from before the time that the modern day peer review process even existed.

How could an entire worldview based on such obviously flawed premises come to such prominence and dominate the academy? I submit that this is one reason why the academy is in the dismal state its in today, but regardless, good old human nature rears its head again - established, vested interests are incentivized to protect themselves, and will use all available strategies to do so. Once peer review became a way to gatekeep out ideas and research, whether due to financial or ego/rivalry based considerations, or any other, it started to be used to do exactly that.

Outside the academy

We don’t have to look solely within the walls of academia to see how empirically bankrupt this worldview is, how poorly it matches what we see in the real world. We can look at actual real world discoveries and inventions. If peer review was so absolutely crucial to discovering basic empirical truths that we couldn’t do without it, wouldn’t it play a necessary role in the development of the major technological breakthroughs of our time? To the point that they couldn’t even happen with it, right? Because any claim without a peer reviewed citation is rejected by the evidence based worldview, surely multiple massively world-changing innovations couldn’t happen without it.

Except that, in the 20th century alone, we have the nuclear bomb & nuclear power, rocketry, airplanes, and antibiotics. All discovered via genius level engineering, trial and error, and sometimes collaboration and discussion between scientists, but each and every one successfully invented and changed the world without today’s publishing and peer review gatekeeping process. But hey. Forget all that. You should see my P-values bro.

Summary

The purpose of Part 1 is to show how the modern day ‘evidenced based’ worldview came to prominence and dominates to this day, despite obvious shortcomings and problems to the point of absurdity, and how it displaced the prior view encapsulated pithily by Political Science professor Wolfinger: “The plural of anecdote is data.”

In part 2, I’ll apply this framework to a couple examples, one in strength training and one in The Science™ of covid era tomfoolery.