The Great Escape – new cartoons

somewhere inside….

all cartoons free for use by citing
copyright 2018 by Russ Hodge
http://www.goodsciencewriting.wordpress.com

Advertisements

The Subtle Art of the Truly Vicious Review

a guide for Referees of Journal Articles

 from the Vaults of Wilford C. Terris, Prof. emeritus (At Large)

 

We all know the situation: your name has gotten onto a journal’s list of reviewers, and one day they stick it to you by sending you an article on a topic that is dear to your heart (or would be if you’d thought of it first). It’s close enough to the work of your lab that you should have thought of it. And you surely would have, if you hadn’t been running around like a headless chicken in search of external funding. That, of course, is your job: to be a headless chicken, with no time to think about anything besides funding all those technicians, predocs, and postdocs, not to mention the Christmas party. It’s what you have the postdocs for, to do the thinking, and they are the ones who should have thought of doing the experiments described in the article the journal has just sent you, but they failed to do so, in spite of having heads and not being chickens. Inexcusable. Now you’ve got to single out someone to punish, to set an example, to emphasise to the others how important it is to think about things you aren’t thinking about. But should be.

The real dilemma, however, is more serious: somehow you’ve got to convince the journal to reject the damned paper. Better yet, not to reject it outright, but to send it back for major revisions, asking for experiments that are at least very difficult, and ideally, impossible. If you’re lucky, this will send the manuscript spiraling off into Revision Limbo, where it needs to remain just long enough for your lab to catch up. You’ll have to do some things differently, of course. Maybe you’ll use extra crispy mice rather than regular mice – which you can, thanks to all of that funding you have been running after.
[Editor’s note from Russ: I think he means CRISPR mice; Wilford has been emeritus for quite some time now.]

If the paper is a first submission, there’s bound to be something inherently wrong with it. These days scientists almost always submit a story before they’ve nailed down every last detail; they’re too worried about being scooped. Which they ought to be; that’s how the game is played. The only question is who will scoop them, and since somebody’s going to try, why shouldn’t it be you? So you’ve got to slow things down by thinking up more experiments for them to do, asking them to validate the results in another model system, using different statistical methods… All of this is standard procedure for a review; just try to be in a really bad mood when you read the paper and write your report for the journal.

Sometimes, however, more drastic measures are called for. Maybe you’re dealing with a third or fourth revision, or one of those rare papers that is truly excellent and so thorough that only a fool would disagree with its conclusions. That’s when the Artistry is called for. You’ve only got one chance to derail this thing, so you’ll have to aim for strategic targets in a way that has a devastating impact on the paper, while seemingly going about the referee business as usual. To pull this off, no one may suspect that you have a personal stake in the outcome.

Now before anyone jumps up and accuses me of perverting the sense of the review the  process, and its lofty goals of being fair and impartial, I will only say that I have spent many decades on the receiving end of the peer review system. Fair and impartial? What planet have you been living on? If you get back three reviews of your paper, there’s always one joker in the deck who seems determined to muck you up. Usually their comments indicate they haven’t read the paper, or if they did they entirely missed the point – even though it’s right there in the abstract, as plain as day to any sane person with a reasonable comprehension of the English language. Based on comments offered up in the past, I’m not convinced that this third reviewer meets either of these criteria. I have noticed, however, a method to the madness, and I’ll lay it out here for future reference, if you ever find yourself in need.

The following list provides some strategies that have proven effective in bringing the publication process to a grinding halt. They can be safely used in almost any situation, providing you follow two guidelines:

Don’t ever use them all, or use the same subset in successive reviews for the same journal, because eventually the editor will get wise to you.

Each comment should be just vague enough that you don’t get caught outright in an easily refutable lie. If the journal editor does come back to you on some point, apologize and say the comment probably referred to a point on a different page. Unfortunately you’re currently on the road and don’t have access to the manuscript, but once you get back you’ll check into it and contact him. (Which, of course, you won’t)

Each criticism needs to be adapted, of course, to suit the paper at hand. Here the examples are oriented toward biomedical research, but they can be easily tweaked to fit any other field.

 

  1. Advise the journal simply to reject the paper out-of-hand, claiming that “while it may have some merit, it is clearly of interest to only a small group of specialists devoted to a highly arcane field that is definitely on the wane and is likely to disappear entirely in a few years.”
  2. If the paper is based on studies of a particular molecule/cell type/ organism/species, claim it has no relevance beyond delivering a trivial detail about the specific system considered in the study.
  3. If the paper makes some large, global claim about a basic issue in science, state, “It’s not transparent to me how the authors conceptually move from the specific system they are working with to the grand claim they make about this process/mechanism/theory as a whole. Several other possible interpretations of the data come to mind; it would be interesting to get a deeper look into the process by which they selected a conclusion which, primae facae, does not seem to be the most likely one.”
  4. Claim that “while the paper claims to present original results, I seem to remember having read the same basic study in an obscure journal a few years ago, and I’m certain that I have heard other groups present on practically the same topic at various conferences. If I come across the reference, I will send it along.”
  5. If the paper does not use –omics technologies, include the statement, “One must wonder why the authors didn’t approach the question using high-throughput methods across the whole genomes of entire species.”
  6. If it does make use of –omics technologies, write, “It probably would have been better to focus this study on a specific system, cell type, or organism. The enormous breadth of the study created a pile of supplemental data so huge that we simply have to take the authors’ word that it means what they claim. Personally I do not quite understand how the authors could discern significant effects from the noise, particularly given the statistical model they used (rather than much better ones which have developed in the meantime). Not to mention the cut-off points, which seem rather arbitrary.”
  7. If no industry or private affiliations are listed, state, “I find this strange, because I personally saw one of the authors having lunch with a vice president of a major pharmaceutical company/defense contractor/psychiatrist” (depending on which best applies to the situation; don’t name the specific author).
  8. If industry affiliations are listed, write, “As can be clearly seen from the list of affiliations, the results may well be biased by funding from the pharmaceutical/ defense/psychiatry industry, even if only a little money changed hands, and the effects are quite subtle.”
  9. Include the statement, “It is unfortunate that the principles of double-blind studies were not applied to the experiments, which clearly reflect the influence of unconscious choices and bias among the author(s).” This is usually safe because unless the paper is specifically medical, you’ll almost never find double-blind experiments. If by chance you’ve received one that does, then write, “But were the scientists truly blind? and if so, truly double-blind?? Someone had to be able to keep track of which was the experimental group and which was the control… If this was managed by computer, were appropriate firewalls in place? Could someone have hacked in after hours?”
  10. Point out that the list of authors reflects a fundamental bias against women in science (if there are more male authors), men in science (if there is a preponderance of females), authors of Italian/Spanish/French/German/ Japanese/Chinese/Indian, etc. etc. ancestry (depending on which ones are missing).
  11. If there are any authors from non-academic organizations, criticize their presence as a potential source of bias and dig up some dirt on their organizations. If you can’t find any dirt, write, “And have you heard how many retractions there have been of papers from this place?”, regardless of whether there have actually been any retractions.
  12. If all the authors are academics, criticize the absence of industrial partners with a statement such as, “Some of the methods used in the work have been developed to a much higher state of precision by industry; the authors would have been safer adopting more standard technologies and methods.”
  13. Write, “This paper was clearly written by someone with a limited familiarity with English and it could use a good work-over by a native speaker,” even if the language is flawless. If the paper is very good, this will send the authors into a tizzy of sentence-by-sentence editing, usually producing something worse than the original – in any case it will delay publication by at least 6 weeks.
  14. Write, “I find the usage of commas rather bizarre, for example on page…” pointing out that your pagination may be different than that seen by the authors or editor. This comment is always safe because English speakers never entirely agree on all aspects of comma usage. If you’re lucky you’ll get another 6-week delay out of just the frenzied search for a misplaced comma.
  15. Write “In some cases the logic is sloppy and there are gaps. As just one example, see paragraph… on page…” (picking out any paragraph that fails to begin with “Thus”, “therefore”, “however”, or some other logical connector anyplace that you could squeeze one in.
  16. Write, “While the experiments seem to indicate a trend which tends to support the conclusions, I am not entirely convinced; the argument would have been stronger if the authors had studied the issue at the level of basic mechanisms/cellular level/tissue/organism as a whole” (whichever is lacking).
  17. Write, “Our attempts to reproduce the experiments completely failed, or led to entirely different results.” (Although you haven’t tried to reproduce any of them, this will almost always be true; in the case of a miracle where the experiment can be reproduced, you can always find a way to sabotage it.) 
  18. Write, “Although at the moment I don’t have access to all of the images, I seem to remember two that seemed strangely similar – has the author (presumably by accident) used part of the same image twice, in a different orientation or color scheme? Although the image I’m thinking of may, in fact, come from a previous paper by the group.”
  19. Pick one of the technologies or methods used in the paper and mention something like this: “Our experience with instrument X demonstrates that the manufacturer’s protocol occasionally produces inconsistent results. Ideally the results should be validated using instrument Y,” where Y is something so fantastically expensive or unique that no one else can acquire it.
  20. Conclude with a statement such as, “In these days of endless retractions of even seemingly exceptional work, caution is advised, particularly in cases where there is even the vaguest scent of scientific malpractice.”
  21. Append a global, vague generalisation such as, “I find the repeated use of specific adverbs extraordinarily tedious,” or, “Why don’t the authors ever use adjectives?” or, “Check for inconsistencies in the style of references.”
  22. If in spite of your efforts, the journal decides to accept the paper, well, you’ve done your best and it’s out of your hands. At this point you should switch sides and send a note to the editor saying, “I am greatly pleased to hear that your decision went this way. I regretted feeling obliged to offer a few small points of criticism, but am encouraged that the writers took them in the spirit they were intended and have produced a final draft of the ms. that is greatly improved.”

As should be obvious, the overall strategy is to engage the enemy simultaneously at all levels from all directions (raising issues about the scientific question, methodology, writing style, comma usage, etc.). I believe a similar strategy is described in a passage in The Art of War, the ancient treatise by the Chinese general Sun Tzu, but I don’t have access to my copy right at the moment. When I find it, I’ll get back to you.

 

If you liked this piece, you will probably enjoy:

Even God’s first paper got rejected, and

The Devil’s Dictionary of Scientific Words and Phrases

The Bible of Elazığ (5)

Part five

(For the beginning of this story, see the earlier posts at part 1, part 2, part 3, and part 4.)

Carbon 14 dating is based on the priniciple that an animal’s tissue contains carbon dioxide from the plants it has eaten. CO2 is radioactive because the carbon arises through an interaction of cosmic rays with atoms in the high Earth atmosphere; it then settles to the surface and is absorbed by plants during the process of photosynthesis. When an animal dies, it stops eating and absorbing CO2, and its radioactivity begins a process of decay. About half of the value is lost approximately every 5,730 years. So measuring the amount of CO2 and its level of radioactivity in a sample permits scientists to date a tissue – providing the animal that absorbed it died within about the past 50,000 years.

Modern dating has to take another factor into account. Until the 1960s, when the practice was banned, a number of nuclear weapon tests were carried out above ground. This nearly doubled the amount of radioactive carbon in the atmosphere, and the tissues of every animal that has died since bear the signature of this event.

* * * * *

I hadn’t been able to stop thinking about the Bible during my family’s vacation – maybe the results of the dating would be waiting when we returned. But there was nothing from the Center for Archeometry in the mailbox.

I took a couple of calls from the Turkish family and met once more with the group in Berlin, but had nothing to report. “They said in the best case it would be a few weeks,” I said, and promised to check in as soon as I had the results. The nerves of everyone involved were frazzled, which I could well understand.

Finally, two weeks later, the letter arrived in Berlin. I wanted to tear open the envelope, but peeled open the flap carefully and removed several printed pages. The Center had included a cover letter which didn’t summarize the results. For that I’d have to look at the data, which was included. Instead the letter merely stated that the smaller sample – the one I’d taken from the inner pages – was too small for reliable measurements, so they’d had to use the fragment I’d obtained from the cover.

I opened the report and scanned the data, which consisted of a number of tables; at first I couldn’t make any sense of it. Finally I found a sentence in the text that resolved all of the numbers and figures: “Noted is the clear presence of a radioactive peak that definitively places the date of the material after the 1960s; the best estimate that can be made from the data suggests that it stems from the period between 1996 and 2003.”

In other words, as ancient as it looked to the untrained eye, the Bible was a modern forgery.

* * * * *

I had been thinking about the fragmentary sample during our vacation. What if that bit hadn’t really come from the Bible? The cloth that the old man had spread across the table had been clean, I was fairly sure of that, but from the way they’d handled the book – could something have been transferred to it? Could the fragment that had been lying there when they lifted it, which I carefully packed and took abroad, have come from somewhere besides the cover? Had it gotten stuck to the outside by accident?

Out of concern for preserving an object that was probably a forgery, I’d eliminated any chance of obtaining a definitive answer. This was nothing more than idle speculation, but the possibility would haunt me for years. Another trip to Turkey was plainly out of the question.

Especially after I called the family in Berlin, and after I placed one more call to Abdullah in Turkey, to deliver the dismal news.

“Are you sure?” he asked, at least three times.

There was no question, I told him; the sample that had been tested was modern.

“Okay,” he said, in a tone of voice that very clearly indicated his displeasure.

That was the last I would hear from any of them.

* * * * *

A few years have gone by. From time to time I have opened the files of the photographs of the book, and I have kept my eye on the news for reports of discoveries of any ancient manuscripts in Turkey. A year or so after all of this happened, a friend whom I had told the story sent me a clipping of a report of a Bible that had been found there. It purportedly dated from the year 1000 BCE or so, and was in good condition; its value was estimated in the tens of millions of Euros.

I had a story to tell, but I remained haunted by questions – not only about my carelessness in taking a sample about the book. Some of what had happened didn’t make any sense. If the family had known from the beginning that they were dealing with a forgery, why would they ever have let us come to take samples in the first place? They surely would have known that scientific testing would have exposed the fraud.

Perhaps they didn’t know – maybe someone else had made the Bible and the other documents they’d seen. Maybe they’d acted in good faith, having found the manuscripts in more or less the way they had described. But if so, what forger would have spent the months that were surely necessary to create an object that was a work of art in its own right – complete with a fastidious but completely fabricated, ancient script, possibly telling some sort of story in some ancient language – unless he’d been sure of selling the thing for a profit? Why such a long, laborious effort? Surely 50 pages of material and script would have sufficed?

From what I have learned in the meantime, such objects normally appear on the black market and are initially sold for a meager price to a gullible buyer who is willing to take the risk that they are forgeries. Then they work their way up the food chain of the underground antiquities market, until they reach a price where someone insists on authentication. At that point the game is up – but at least the transactions have introduced layers between the artist and the buyer.

But coming to any satisfying answer requires an assumption that those involved were acting on good faith, and the slipperiness of some of the stories we were told indicates otherwise.

Once in a while, when I’ve told this story to friends, they’ve suggested it would make great material for a novel. But recently there have been too many fictional tales of ancient Biblical manuscripts; I remember being terribly impressed by Irving Wallace’s The Word, when I was in high school, and of course there is the entire loopy (while highly crafted) Dan Brown genre that has passed through today’s culture like an infection.

No, the interesting thing about this story was that it was true. And that there was more to it: somewhere, most likely in Turkey, is or was an artist who is making these objects. But that was a story that would clearly be dangerous to pursue – at this stage in my life, no thanks, not me. Maybe there will come an ambitious young journalist ready to take it on someday.

Not the last of my thoughts have centered on my own involvement in this: did I act responsibly and ethically? I’m not sure. Given that the object turned out to be a forgery, no harm was done; I can’t say how things would have developed if the dating had turned out differently, and assigned a date to the Bible that more closely reflected the family’s claim.

I still believe that if such an artifact ever does appear, and it proves to be authentic, its contents belong to the world. That requires the sort of protection that can only be assured if academics get involved, but it also requires that governments and institutions handle ancient documents in good faith.

If I had to do it over again, I would have insisted on an expert in ancient manuscripts making the trip to Turkey with us. Those I had previously contacted were too skeptical to get involved, which probably should have told me something. I certainly would have planned a few more days time for the photographs, and would have obtained a larger sample that would have provided an unambiguous date for the manuscript.

If, if, if.

* * * * *

I don’t know why now seemed the right time to publish this story. But a few days after the third installment appeared, I received the following message from a biologist in Turkey, which I am reprinting with his permission:

Dear Dr. Hodge,

I am a Turkish biologist from Inonu Univ, Malatya, the neighboring city of Elazig. Same pictures were shown to myself about 4-5 months ago. The guy told me that it is 105 pages old book written on gazelle leather. The pictures were looking as old as the ones you have posted here. There was a non symmetrical cross with dots around more like a four legged sea star. I sent pictures to Sotheby’s where I sold a piece of rug when I was doing my PhD in IIT(Chicago). Their experts found that it is a product of forgery and has no value whatsoever. Later I learned that these pictures are everywhere in the smartphones of spooky people (so I deleted them. wish I did not and have some here for you). The book is mass produced and put under harsh conditions (such as acid bath and exposing to intense radiation) so that its has a wear out look.

Now sipping my çay (turkish hot tea)!, I hope you have not been swindled. Best wishes,

Hikmet Geçkil, PhD

Thank you, Hikmet, for bringing this story to an end after all.