Tuesday, February 14, 2012

Gaming the system

Two semi-related topics have come to mind lately.  First, this post by the FSP caught my attention, regarding "citation circles", where a sub-community within a scientific discipline agree to cite each others' work.  I've heard of such things, and there's nothing inherently wrong there as long as the citations are relevant and don't consciously omit other equally relevant papers.  Still, I never considered this practice to have too much impact.  Back when I was in grad school, I'd heard of something that is equally fine ethically, and perhaps more important to progress in the long term:  the timely reviewing circle - a group of scientists who agree to respond promptly to invitations to review one anothers' work.  This is not a matter of conspiring to give positive reviews, but an agreement to get manuscripts through the review process quickly.  Imagine if certain "Letters" journals were actually speedy!

The second topic is the idea of trying to influence the selection of referees.  Of course, under many circumstances you as an author can suggest possible referees for scientific papers or grant proposals.  Let's call that a first-order influence.  It's a way of making sure that the editors can get the paper out to technically knowledgeable people in a specialty.  (I've been told by multiple editors that authors who try to suggest "friendly" referees often do themselves more harm than good, because those suggested reviewers are often more harsh than randomly selected peers.)  Recently I learned about a "higher-order" approach:  avoiding citing the work of potentially hostile reviewers, under the assumption that the editor/program officer often gets referee ideas from the reference lists.  I am very skeptical that this approach could matter in a statistically significant way.

1 comment:

some guy said...

Recently I learned about a "higher-order" approach: avoiding citing the work of potentially hostile reviewers, under the assumption that the editor/program officer often gets referee ideas from the reference lists. I am very skeptical that this approach could matter in a statistically significant way.

I'm not so sure. Don't some journals run software that auto-suggests referees based on the manuscript bibliography? I would imagine in such cases this practice could easily be statistically significant.

I've heard it postulated that adding potentially problematic referees to the acknowledgments should stop the editors from sending the manuscript to them. Passing ethical judgement on such practice is left as an exercise for the reader.

(I've been told by multiple editors that authors who try to suggest "friendly" referees often do themselves more harm than good, because those suggested reviewers are often more harsh than randomly selected peers.)

Aside from that issue, even, it could be hard to argue credibly (in the editors' eyes) that the referee is clueless, after you have explicitly suggested them as someone qualified to judge your work -- regardless of whether you expected them to be friendly. (However, I can imagine that this effect is mitigated somewhat by the probability that exactly this occurs all the time.)