Streetwise Professor

The whole CRU disaster has got me thinking about whether given modern information technology, peer review is an anachronism that impedes, rather than advances, scientific knowledge (including social scientific knowledge).  It is quite entertaining–in a perverse, watching a car crash kind of way–to observe the defenders of the climate change consensus repeat “peer review” like a magic spell that will somehow ward off evil (skeptic) spirits.  But if anything, the whole fiasco calls into question the reliability of peer review.

Indeed, the whole display brings to mind the comments of my former colleague, the late Roger Kormendi.  Somebody once mentioned to Roger that he should pay particular attention to a certain piece of research because it had been peer reviewed.  To which Roger replied: “Oh, that means it’s completely arbitrary.”

But as with any institution, it’s not sufficient to point out the flaws in one to justify its replacement with another.  It is necessary to make a comparative analysis with realistic alternatives.  What would be the alternatives to peer review?

In the modern era, the ability to disseminate papers nearly universally and instantaneously, and to make people aware of their existence through things like SSRN makes it possible for myriad scholars to access and evaluate papers, rather than one or two reviewers.  Moreover, the same information technology makes it feasible to provide access to data and code to facilitate replication, examinations of robustness, and the testing of alternative specifications and models on a given set of data.  In this way, it is possible to harness the knowledge of myriad, dispersed individuals with specialized expertise, rather than one or two or even three individuals.  Moreover, the open entry model mitigates the incentive problems associated with peer review, where individuals with weak and often perverse incentives exert incredible influence over what gets published and what doesn’t.

Furthermore, wiki-type mechanisms can be employed to collect and aggregate commentary and critiques about particular papers or groups of papers.  This is another way of harnessing the dispersed knowledge of crowds.  In particular, it would facilitate the exploitation of comparative advantage, allowing, for instance, statisticians to comment on statistical methodologies, computational experts to critique numerical techniques, and so on.

Just think of how things might have evolved differently if climate data and paleoclimate reconstruction had been done under this model, rather than via the peer review mechanism.  The “hockey stick” reconstructions would have been subjected to the critique of expert statisticians who would have uncovered Mann et al’s misuse of principal component methods.  Making raw climate data available would have made it possible to evaluate the sensitivity of results to data selection and methods used to “clean” (quotes definitely needed) data.*  In sum, we wouldn’t be where we are now, not knowing with any confidence just what the climate record tells us, or even what the climate record actually is.

The open kimono approach with code and data would also provide an extremely strong deterrent to fraud.  Moreover, reducing reliance on journals would reduce resources devoted to rent seeking activities (e.g., influencing journal editors, gaming submissions, torpedoing competitors, spending time devising submission strategies).  It would also enhance competition, and reduce the rents that incumbent “gatekeepers” can extract.  Reduced reliance on journals would also mitigate the file drawer effect because journals inevitably condition acceptance on measures of statistical significance.  This leads scholars to abandon research that does not generate such results and encourages selection searches and other “econometric cons,” meaning that published results are likely to present a biased picture of the true state of the evidence.  A more open model would likely reduce these (statistical) size distorting incentives.

One challenge posed by this alternative model relates to the fact that the hiring, tenure, and promotion mechanisms at modern research universities are adapted to the journal publishing mechanism.  Since citations are probably a better metric of quality than whether or not something is published in Journal A or Journal B, or published at all, perhaps a citation-based mechanism could suffice.  (Though if journals faded in importance, this would raise the questions: Cited in what? and How do you compare the quality of citations?  Perhaps some metric where the number of citations of the paper in which a particular work is cited could be used to determine citation quality.)  Also, since participation in wikis, etc., contributes to knowledge, it would be desirable to provide incentives for that kind of activity–which would inevitably require some (inevitably noisy) measurement technology.

These hiring and P&T issues are not immaterial, but to me the first-order issues are reducing the costs of producing knowledge, discovering error, and deterring fraud.  The open source, wisdom of swarms, collaborative, Wiki-based model seems to offer many advantages over the received, hierarchical, journal-based model.  Open access, open source, “swarm,” and wiki models are threatening other information dissemination mechanisms–notably journalism.  So why not journals too?  Why not have reviews by tens or hundreds or thousands of peers who bring to bear comparative advantages (e.g., statisticians critiquing work done by non-statisticians but employing statistical techniques), and who are self-selected for their interest, rather than reviews by less than a handful of inevitably distracted, sometimes conscripted, and often conflicted peers?

The whole journal-based, peer review process is arguably well adapted to a particular technology for producing and disseminating information.  Given the radical changes in information technology, it is at least worth considering whether this received mechanism is still optimal.  I, for one, have serious doubts.

* As a (relevant) aside, one of the most outrageous admissions to come from the Hadley CRU fiasco is that (a) original source data was allegedly destroyed some time ago, and (b) East Anglia University/Hadley have the audacity to claim that only “value added,” processed data was retained.

The arrogance of this claim is beyond belief.  We are supposed to accept that CRU’s methods maximized “value added” for all possible uses of the data?  That every one of the myriad choices that CRU made when processing, filtering, and adjusting the data was the right one for every possible use of the data, and beyond question, let alone reproach?  We should just take this on faith?

How can we test this remarkable assertion?

Oh, we can’t–because they destroyed what would be necessary to do so.

Just think of the hundreds of possible ways of transforming the raw data to deal with problems such as missing observations, or aggregating individual station data to characterize climate over wide areas.  Hadley made a set of choices, and due to their destruction of the original data, we have to live with that, perhaps forever.

Who the hell died and made them the last word?

With open source data and open source code, we would not have to live with the systemic risk inherent in relying on a single set of choices.  Maybe the choices were right–but if they are not, our ability to change adapt is severely constrained.  And maybe they were right at a particular time, but we are now saddled with choices made in light of the techniques available when they were made; it is impossible to bring new techniques to bear on the old data.

Value added my foot.  Who the hell is Hadley to make that assertion?  Just a supercilious, self-serving effort at CYA.

As #1 SWP daughter said in a discussion of these issues: “PAY NO ATTENTION TO THAT LITTLE MAN BEHIND THE CURTAIN!”  That analogy is spot on.