The Subtle Art of the Truly Vicious Review

a guide for Referees of Journal Articles

 from the Vaults of Wilford C. Terris, Prof. emeritus (At Large)

 

We all know the situation: your name has gotten onto a journal’s list of reviewers, and one day they stick it to you by sending you an article on a topic that is dear to your heart (or would be if you’d thought of it first). It’s close enough to the work of your lab that you should have thought of it. And you surely would have, if you hadn’t been running around like a headless chicken in search of external funding. That, of course, is your job: to be a headless chicken, with no time to think about anything besides funding all those technicians, predocs, and postdocs, not to mention the Christmas party. It’s what you have the postdocs for, to do the thinking, and they are the ones who should have thought of doing the experiments described in the article the journal has just sent you, but they failed to do so, in spite of having heads and not being chickens. Inexcusable. Now you’ve got to single out someone to punish, to set an example, to emphasise to the others how important it is to think about things you aren’t thinking about. But should be.

The real dilemma, however, is more serious: somehow you’ve got to convince the journal to reject the damned paper. Better yet, not to reject it outright, but to send it back for major revisions, asking for experiments that are at least very difficult, and ideally, impossible. If you’re lucky, this will send the manuscript spiraling off into Revision Limbo, where it needs to remain just long enough for your lab to catch up. You’ll have to do some things differently, of course. Maybe you’ll use extra crispy mice rather than regular mice – which you can, thanks to all of that funding you have been running after.
[Editor’s note from Russ: I think he means CRISPR mice; Wilford has been emeritus for quite some time now.]

If the paper is a first submission, there’s bound to be something inherently wrong with it. These days scientists almost always submit a story before they’ve nailed down every last detail; they’re too worried about being scooped. Which they ought to be; that’s how the game is played. The only question is who will scoop them, and since somebody’s going to try, why shouldn’t it be you? So you’ve got to slow things down by thinking up more experiments for them to do, asking them to validate the results in another model system, using different statistical methods… All of this is standard procedure for a review; just try to be in a really bad mood when you read the paper and write your report for the journal.

Sometimes, however, more drastic measures are called for. Maybe you’re dealing with a third or fourth revision, or one of those rare papers that is truly excellent and so thorough that only a fool would disagree with its conclusions. That’s when the Artistry is called for. You’ve only got one chance to derail this thing, so you’ll have to aim for strategic targets in a way that has a devastating impact on the paper, while seemingly going about the referee business as usual. To pull this off, no one may suspect that you have a personal stake in the outcome.

Now before anyone jumps up and accuses me of perverting the sense of the review the  process, and its lofty goals of being fair and impartial, I will only say that I have spent many decades on the receiving end of the peer review system. Fair and impartial? What planet have you been living on? If you get back three reviews of your paper, there’s always one joker in the deck who seems determined to muck you up. Usually their comments indicate they haven’t read the paper, or if they did they entirely missed the point – even though it’s right there in the abstract, as plain as day to any sane person with a reasonable comprehension of the English language. Based on comments offered up in the past, I’m not convinced that this third reviewer meets either of these criteria. I have noticed, however, a method to the madness, and I’ll lay it out here for future reference, if you ever find yourself in need.

The following list provides some strategies that have proven effective in bringing the publication process to a grinding halt. They can be safely used in almost any situation, providing you follow two guidelines:

Don’t ever use them all, or use the same subset in successive reviews for the same journal, because eventually the editor will get wise to you.

Each comment should be just vague enough that you don’t get caught outright in an easily refutable lie. If the journal editor does come back to you on some point, apologize and say the comment probably referred to a point on a different page. Unfortunately you’re currently on the road and don’t have access to the manuscript, but once you get back you’ll check into it and contact him. (Which, of course, you won’t)

Each criticism needs to be adapted, of course, to suit the paper at hand. Here the examples are oriented toward biomedical research, but they can be easily tweaked to fit any other field.

 

1. Advise the journal simply to reject the paper out-of-hand, claiming that “while it may have some merit, it is clearly of interest to only a small group of specialists devoted to a highly arcane field that is definitely on the wane and is likely to disappear entirely in a few years.”
2. If the paper is based on studies of a particular molecule/cell type/ organism/species, claim it has no relevance beyond delivering a trivial detail about the specific system considered in the study.
3. If the paper makes some large, global claim about a basic issue in science, state, “It’s not transparent to me how the authors conceptually move from the specific system they are working with to the grand claim they make about this process/mechanism/theory as a whole. Several other possible interpretations of the data come to mind; it would be interesting to get a deeper look into the process by which they selected a conclusion which, primae facae, does not seem to be the most likely one.”
4. Claim that “while the paper claims to present original results, I seem to remember having read the same basic study in an obscure journal a few years ago, and I’m certain that I have heard other groups present on practically the same topic at various conferences. If I come across the reference, I will send it along.”
5. If the paper does not use –omics technologies, include the statement, “One must wonder why the authors didn’t approach the question using high-throughput methods across the whole genomes of entire species.”
6. If it does make use of –omics technologies, write, “It probably would have been better to focus this study on a specific system, cell type, or organism. The enormous breadth of the study created a pile of supplemental data so huge that we simply have to take the authors’ word that it means what they claim. Personally I do not quite understand how the authors could discern significant effects from the noise, particularly given the statistical model they used (rather than much better ones which have developed in the meantime). Not to mention the cut-off points, which seem rather arbitrary.”
7. If no industry or private affiliations are listed, state, “I find this strange, because I personally saw one of the authors having lunch with a vice president of a major pharmaceutical company/defense contractor/psychiatrist” (depending on which best applies to the situation; don’t name the specific author).
8. If industry affiliations are listed, write, “As can be clearly seen from the list of affiliations, the results may well be biased by funding from the pharmaceutical/ defense/psychiatry industry, even if only a little money changed hands, and the effects are quite subtle.”
9. Include the statement, “It is unfortunate that the principles of double-blind studies were not applied to the experiments, which clearly reflect the influence of unconscious choices and bias among the author(s).” This is usually safe because unless the paper is specifically medical, you’ll almost never find double-blind experiments. If by chance you’ve received one that does, then write, “But were the scientists truly blind? and if so, truly double-blind?? Someone had to be able to keep track of which was the experimental group and which was the control… If this was managed by computer, were appropriate firewalls in place? Could someone have hacked in after hours?”
10. Point out that the list of authors reflects a fundamental bias against women in science (if there are more male authors), men in science (if there is a preponderance of females), authors of Italian/Spanish/French/German/ Japanese/Chinese/Indian, etc. etc. ancestry (depending on which ones are missing).
11. If there are any authors from non-academic organizations, criticize their presence as a potential source of bias and dig up some dirt on their organizations. If you can’t find any dirt, write, “And have you heard how many retractions there have been of papers from this place?”, regardless of whether there have actually been any retractions.
12. If all the authors are academics, criticize the absence of industrial partners with a statement such as, “Some of the methods used in the work have been developed to a much higher state of precision by industry; the authors would have been safer adopting more standard technologies and methods.”
13. Write, “This paper was clearly written by someone with a limited familiarity with English and it could use a good work-over by a native speaker,” even if the language is flawless. If the paper is very good, this will send the authors into a tizzy of sentence-by-sentence editing, usually producing something worse than the original – in any case it will delay publication by at least 6 weeks.
14. Write, “I find the usage of commas rather bizarre, for example on page…” pointing out that your pagination may be different than that seen by the authors or editor. This comment is always safe because English speakers never entirely agree on all aspects of comma usage. If you’re lucky you’ll get another 6-week delay out of just the frenzied search for a misplaced comma.
15. Write “In some cases the logic is sloppy and there are gaps. As just one example, see paragraph… on page…” (picking out any paragraph that fails to begin with “Thus”, “therefore”, “however”, or some other logical connector anyplace that you could squeeze one in.
16. Write, “While the experiments seem to indicate a trend which tends to support the conclusions, I am not entirely convinced; the argument would have been stronger if the authors had studied the issue at the level of basic mechanisms/cellular level/tissue/organism as a whole” (whichever is lacking).
17. Write, “Our attempts to reproduce the experiments completely failed, or led to entirely different results.” (Although you haven’t tried to reproduce any of them, this will almost always be true; in the case of a miracle where the experiment can be reproduced, you can always find a way to sabotage it.) 
18. Write, “Although at the moment I don’t have access to all of the images, I seem to remember two that seemed strangely similar – has the author (presumably by accident) used part of the same image twice, in a different orientation or color scheme? Although the image I’m thinking of may, in fact, come from a previous paper by the group.”
19. Pick one of the technologies or methods used in the paper and mention something like this: “Our experience with instrument X demonstrates that the manufacturer’s protocol occasionally produces inconsistent results. Ideally the results should be validated using instrument Y,” where Y is something so fantastically expensive or unique that no one else can acquire it.
20. Conclude with a statement such as, “In these days of endless retractions of even seemingly exceptional work, caution is advised, particularly in cases where there is even the vaguest scent of scientific malpractice.”
21. Append a global, vague generalisation such as, “I find the repeated use of specific adverbs extraordinarily tedious,” or, “Why don’t the authors ever use adjectives?” or, “Check for inconsistencies in the style of references.”
22. If in spite of your efforts, the journal decides to accept the paper, well, you’ve done your best and it’s out of your hands. At this point you should switch sides and send a note to the editor saying, “I am greatly pleased to hear that your decision went this way. I regretted feeling obliged to offer a few small points of criticism, but am encouraged that the writers took them in the spirit they were intended and have produced a final draft of the ms. that is greatly improved.”

As should be obvious, the overall strategy is to engage the enemy simultaneously at all levels from all directions (raising issues about the scientific question, methodology, writing style, comma usage, etc.). I believe a similar strategy is described in a passage in The Art of War, the ancient treatise by the Chinese general Sun Tzu, but I don’t have access to my copy right at the moment. When I find it, I’ll get back to you.

 

If you liked this piece, you will probably enjoy:

Even God’s first paper got rejected, and

The Devil’s Dictionary of Scientific Words and Phrases