Byung Kyu Park's Personal Website

xkcd on “revolutionaries”

Yes, scientific process is treacherous. There are crackpots tilting at windmills—I see at least one at every large APS conference I’ve been to; I’ve listened to them and looked at what they have, but it turns out some of them can’t even do simple algebra—and then there are outsiders who have given plenty of time to learning the state of the art and either improving on it or fixing mistakes in it.

That’s why we have peer review. However, once that process has been corrupted either for political reasons or other reasons, we are back to square one: every claim must be examined as if it were serious claim, because we can’t throw out the baby with the bathwater.

We already know peer review isn’t a panacea. For one, it is widely known that peer review simply cannot prevent scientific fraud, because as experimentalists, we tend to trust in the data—we may question the process of data gathering, but once the process itself seems free of error, we trust the data as presented, unless it claims something fantastically impossible, usually involving violation of energy conservation or something on that order.

But, recent global warming scandal, a.k.a. ClimateGate, has uncovered even more problems with peer review: it’s done by “peers”

In response to an article challenging global warming that was published in the journal Climate Research, CRU head Phil Jones complains that the journal needs to “rid themselves of this troublesome editor”-hopefully not through the same means used by Henry II’s knights. Michael Mann replies:

I think we have to stop considering “Climate Research” as a legitimate peer-reviewed journal. Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal.

Note the circular logic employed here. Skepticism about global warming is wrong because it is not supported by scientific articles in “legitimate peer-reviewed journals.” But if a journal actually publishes such an article, then it is by definition not “legitimate.”

In short, if your peers are crackpots, would you want your work reviewed (and judged) by those peers? Peer reviews are used to keep crackpots out of legitimate science journals—but, if the crackpots have already taken root inside that fence, then, well, crackpots can use it to keep legitimate science out of journals.

I wouldn’t call these failures of peer review necessary … failure of science. It represents pitfalls and setbacks of frontiers of research. Eventually (on the time scale of decades or even centuries), scientific theories are judged by their ability to produce predictions that no other theories can. And short of time travel, there is no way to fake that.