Andy's Brain Blog

Peer review, the cherished academic tradition of having your work criticized by anonymous angry people, is an excellent chance for you to see your prose violated in public. According to one publisher, peer review also helps "increase networking opportunities," in that after having your paper reviewed you will become very, very interested in finding out the names and addresses of all your reviewers.

The reviewing panel consists of two or three reviewers, known by their pseudonyms "Reviewer #1," "Reviewer #2," "Johnny Two-Knuckles," "Icepick Willie," and so on. As everyone knows, the review process tends to be a "good-cop, bad-cop" routine, with Reviewer #1 being nice and lenient - pointing out you shouldn't use Comic Sans font, for instance - while Reviewer #2 is so offended to read your paper that he thinks you should, in so many words, die.

Reviewer #2 is in fact a man named Gary who owns a hardware store in Winnipeg. Although no longer in academia, Gary is still the man who can be counted on, when the chips are down, to write a scathing review of whatever he's reading. Editors scramble to recruit Gary when confronted with a paper they may have to accept, and he is regularly solicited for freelance reviewing. This week we caught up with Gary at his summer home on the banks of Lake Manitoba.

===============

Andy's Brain Blog: How did you become interested in reviewing?

Gary: I took a psychology class in college where we critiqued each other's class projects that were written like scientific articles. Then we anonymously reviewed each other's papers. The professor was impressed that I managed to reject every single paper that I read, and that I also managed to make unnecessary remarks about the author's intelligence and work ethic. At the time I didn't even know what rejecting a paper meant. It just came naturally to me. He put in a good word for me at Elsevier.

ABB: And what happened then?

Gary: Well, I began reviewing everything I read. One time I got so into it that I ended up reviewing the back of a cereal box. It was an accident, but the review was accepted anyway. Two employees at General Mills got fired because of it.

ABB: Wow.

Gary: Yeah. There were grammatical mistakes on there like you wouldn't believe. I couldn't follow the logic of how solving a word game would help Buzz escape from a bank vault full of honey. And the figures were atrocious.

ABB: What was the most memorable review you ever did?

Gary: It's funny you ask, because just last week I returned from the annual Reviewer's Gala in Manhattan. It's a private party for those who have the highest rate of rejecting manuscripts, with awards given for achievements like Most Papers Rejected, Most Brutal Review, Most Irrelevant Comment, and so on. This year I won the prize for Most Hurtful Comment, which went something like: "Writing this paper didn't make you a terrible scientist - you were born one." When the emcee read that line, the audience went wild.

ABB: What is the most ridiculous comment you've ever gotten someone to address?

Gary: I'm not that good at making crazy requests, but one of my fellow reviewers - Carl - can get people to do almost anything. One of his comments was, and I quote: "This is a strong paper, but I think it would be even stronger if, for some reason, all of the authors did the gallon challenge, and uploaded a video of it to YouTube. Now obviously you don't have to do this, but you should, because I am a reviewer."

ABB: They actually did that?

Gary: Yeah. One of them had to go to the hospital. Carl felt pretty bad about that one.

ABB: What advice would you give to a first-time reviewer?

Gary: Rejecting a paper takes a tremendous amount of courage. We've all had the temptation to accept a paper because the science was "solid," or because the logic was "air-tight," or because one of the authors secretly gave us "money." Be firm! I find that I write my best reviews when I'm pissed off about something that has nothing to do with the paper, such as getting something in the mail about taxes.

ABB: You owe a lot of taxes?

Gary: No, I just found out about them, as a concept. They're ridiculous. That's the kind of thing I'm talking about that will get you in the right mood to review a paper.

ABB: Have you ever accepted a paper?

Gary: No.

ABB: Never?

Gary: Never-ever.

ABB: Never even come close?

Gary: Well, there have been a few times. Maybe if one of the authors had the same last name as a celebrity I like, such as Barry Manilow or Kenny G. But other than that, no.

ABB: How long does it take you to write a review?

Gary: Not long. I have a template that I follow, which is a lot like Mad-Libs. For example, "This is an interesting [study / review / prophecy], but I find the [results / figures / theology] unconvincing because I am [an expert / a skeptic / a nun]." Things like that.

ABB: So, how long does it take to get back to the authors? A couple of days? A week?

Gary: No, no, nothing like that. The review takes a couple of days at the most, but you can't let the editor think that you're just blowing through it. I sit on it for at least a few months.

ABB: What are your strategies for writing a review? Is it to always go negative, or what?

Gary: Well, you have to be careful about that. Writing only negative comments raises suspicions that you're taking out your own frustrations and lack of success on the authors instead of addressing their arguments. I aim for a mix of negative comments, nitpicking, and vague sentences. Vague sentences are great, because the authors aren't going to admit that they don't understand what you're saying. Asking an academic to be clear is like asking him to take his clothes off - it's a rude request, almost obscene. So instead they reply as though they understood perfectly what you were saying. It's amazing to see how they try to interpret what is in fact nonsense.

ABB: Can you give an example?

Gary: Sure. Let me see - here's one: "Among the considerations that arise at this stage are the likelihood that the manuscript would seem of considerable interest to those working in the same area of science and the degree to which the results will stimulate new thinking in the field, although we cannot be persuaded of the justifiability, synergy, or translatability of how these results integrate with the conclusions and narrative of Fensterwhacker et al, 2009. Are you professional. Also, you spelled 'their' wrong (should be 'they're': p. 19)."

ABB: I have no idea what that means.

Gary: Exactly.

ABB: How do they respond?

Gary: Usually they begin with something like "We thank the reviewer for their insightful comment," or "We are just thrilled by this excellent suggestion," or "I simply cannot wait to meet this reviewer in person and show him how incredibly, insanely grateful I am, which in no way would include kidnapping his dog." It's interesting how far someone will bend over backwards to address a comment that could've been written by a complete space loon.

ABB: Why do you keep doing this? You're not in academia anymore.

Gary: I try to focus on the big picture. I think that by irritating so many people, everybody will have something in common to talk about. Then they can bond over their shared frustrations and challenges. It makes academia more like a family, except in the sense of being related to or liking or caring about one another.

ABB: Gary, thanks for your time.

Gary: You spelled "your" wrong.

"Everybody talks about the weather," Mark Twain once famously declared, "But nobody ever does anything about it." He then ate a knife to show the seriousness of his statement.

The same sentiment applies to reviewers in academia. Young researchers are frequently traumatized by their first exposure to the peer-review process, in which a paper is scrutinized and criticized by one's colleagues, ostensibly to judge whether the paper is acceptable for publication and to suggest improvements. However, as every one of us scientists intuitively understands, in reality it serves as nothing more than a vicious catharsis for the shattered dreams and failed career and (probably) frustrated sex life of the reviewer, unable to distinguish between his personal neuroses and legitimate criticism, by ruthlessly excoriating an otherwise brilliant exposition of scientific reasoning and experimental methodology. This trauma is then propagated along to other reviews in a never-ending cycle of misery and despond, with the abused becoming the abusers. Conrad summed up this mentality in his book The Secret Agent, in which he characterized a particularly violent group of terrorists plotting against a repressive government as not necessarily opposed to tyranny per se; they were merely unhappy that they were not the ones doing the tyrannizing.

The whole review process has been thoroughly criticized by many, but one of the first systematic exposés of peer-review was done back in the 1980s by two young researchers. To better understand their motivations, let's turn the clock back a few decades to the United States of the '80's: Top Gun, one of the supreme achievements of world cinema, had just been released in theaters to universal acclaim; cocaine flowed like crack through the streets of America; and the nation was beginning to feel its oats after successfully invading and destroying the hated island of Grenada. Life was so good, you could taste it in your spit.

However, even in the midst of all this success and prosperity, problems with the peer-review process still loomed large in the minds of the public. This is where our two young heroes, a couple of spitfire hoydens named Peters & Ceci, entered the picture, intent on producing concrete evidence of systematic bias among reviewer. Although subsequently tortured and murdered for their heresy by the peer-review mafia, their publication still circled clandestinely among groups dedicated to changing how the review process works.

Before their ghastly deaths, Peters & Ceci had taken papers previously accepted for publication and changed a few trivial details on the manuscript - for example, changing the author affiliation from a prestigious university, such as Yale, to a less prestigious-sounding, totally fictitious institution, such as the Nelson Center for the Study of Bodacious Ta-Ta's. Everything else, including materials and methods, remained exactly the same. After resubmitting these manuscripts, surprisingly few were reaccepted; in fact, out of the nine journals sampled, only one accepted the same papers it had accepted earlier! Keep in mind that these were all papers that had been accepted at respectable journals; virtually nothing had been changed, except for trivial details that had no bearing on the quality of the reported results. Reviewer comments were focused on methodological details ("Analysis of variance was improperly used in a post-hoc fashion"), writing style ("The theoretical focus of the introduction could be tightened up considerably"), and even, incredibly, features of the individual authors ("Just where can I meet this Nelson fellow?").

On the face of it, these results may seem to be cause for suffering and despair, as we find ourselves at the mercy of only one more facet of an openly hostile universe. One may well ask whether anything can be done, or whether it would instead make more sense to curse God, and die.

I offer a few suggestions. First, make sure that you are at a prestigious university; and if you are not, begin to associate with people who are at prestigious universities. Offer to make them authors on your paper, even if they haven't done anything. Ingratiate yourself with well-known figures in your field, even going so far as to openly debase yourself by performing menial tasks for them, or by humiliating yourself by offering to eat one of their used tissues. By stripping away any vestige of self-respect, you will increase your chances of their agreeing to be a co-author on your paper, as long as they don't have to do any work and never have to see you again. This, in turn, will increase your chances at publication.

The word sycophant has had negative connotations, long associated with spineless, weak, craven poltroons who would not hesitate to pawn their souls for the most trivial of earthly gains. Our job is to make it a positive word, associated with success, advancement, and - above all - publication.

EPILOGUE

Ceci and Peters laid on the grimy floor of their cell, wasted, defeated, only a putrid dish of water sitting between them. Their eyes were hollow and sunken and their frames emaciated, their faces dirty and wizened, their hair wild and bedraggled. Ceci's eyes roved around the room and he gibbered in an incomprehensible argot, thin strands of drool running down his cheeks in viscous rivulets. They had not eaten in five days and they awaited the hour of their execution with neither apprehension nor resolve but exhaustion.

There was the sound of a key turning in the door and an enormous Swede entered the room, holding a maul in his right hand. He seemed to fill up the entire room with his frame and he had two little pig's eyes set into the sockets of his skull and a cheesy rictus pasted on his face like a doll's. When he moved he gave the impression of a boulder set into motion. He gestured with the maul for Peters to get up but Peters merely glared at him and would not move. The Swede seemed to expect this and he bent down and picked him up by the meatless bone of his arm and dragged him to his feet.

Peters spit in the face of the Swede but it seemed as though it was all part of his day for he did not blink or wince but merely smiled, his right hand raising up slowly and methodically and then snapping down like the lid of a box and the maul made a sound like hitting a pumpkin with a club. Peters' eyes bulged as he collapsed to the floor like his bones had turned to jelly and the blood began to stream out of his ears.

The Swede bent down to look at him and watched as the capillaries in his eyes began to break up. Ceci's eyes darted zigzag across the room, his mouth open and his lower jaw flapping up and down uncontrollably. The Swede looked over at him and smiled.

* * *

The next day their lifeless bodies were dumped in a shallow arroyo along with everyone else who had dared to challenge the might of the peer-review process. For the next two days the air was filled with the whine of flies and the wolves and buzzards wallowed in the carrion, and when the feast was over there was nothing left save for their bleached skulls and the bones of their briskets curving in upon themselves like grotesque half-closed traps.

FIN

Link to the paper can be found here.

Thanks once again to Keith Bartley, who is quickly becoming an Andy's Brain Blog VIP. VIP status includes a free year-long subscription to the blog, shampoo samples, Nutella aroma therapy, and a summer vacation to North Dakota, including a tour of the legendary sugarbeet processing plants, a pitchfork steak fondue, and tickets to the one-hour acting monologue Bully! The Teddy Roosevelt Story.