Maybe this is how it happens: You see an interesting (seemingly innocuous) paper and decide to read it. Upon finding it very information-dense, you decide to take a look at the supporting information (SI) and notice that the SI has a word count greater in size than an average PhD thesis. Or maybe it’s when you decide to print the SI and realize something unusual is going on when your printer is still spitting out paper after half an hour.
However you have become aware it, scientific practice has been changing in the last few years. If I remember correctly, supporting information packages started becoming the norm for papers (at least in some journals) a only few years ago and the average SI length has been growing steadily ever since.
Now something interesting has happened. From November 1st and onwards, The Journal of Neuroscience (JNS), a leading Journal in that field, will no longer allow authors to include supplemental material when submitting new manuscripts (JNS agrees to link to non-peer reviewed supporting material on the author’s own site). The decision is explained in detail by Editor-In-Chief John Maunsell, who presents a lucid and interesting argument. He explains that on one hand, the decision was made to make the task of peer reviewing a paper more manageable, i.e. to help the referees:
Although [JNS], like most journals, currently peer reviews supplemental material, the depth of that review is questionable. Most well qualified reviewers are overburdened with requests to review manuscripts, and many feel that it is too much to ask them to also evaluate supplemental material that can be as extensive as the article itself. It is obvious to editors that most reviewers put far less effort (often no effort) into examining supplemental material. Nevertheless, we certify the supplemental material as having passed peer review.
This surely is an accurate description of the situation many referees find themselves in. Going over every equation and argument in a 100 page SI takes several days, an amount of time that most academics simply don’t have available. (In fact the current state of peer review, even without mammoth SI’s, has been argued to be suffering from serious problems.)
On the other hand the decision is also intended to protect the authors.
Another troubling problem associated with supplemental material is that it encourages excessive demands from reviewers. Increasingly, reviewers insist that authors add further analyses or experiments “in the supplemental material.” These additions are invariably subordinate or tangential, but they represent real work for authors and they delay publication. Such requests can be an unjustified burden on authors. In principle, editors can overrule these requests, but this represents additional work for the editors, who may fail to adequately referee this aspect of the review.
Reviewer demands in turn have encouraged authors to respond in a supplemental material arms race. Many authors feel that reviewers have become so demanding they cannot afford to pass up the opportunity to insert any supplemental material that might help immunize them against reviewers’ concerns.
The “supplemental material arms race” described eloquently above is another element that I, as an author, can relate to—and suspect that many others feel the same.
With no room for peer reviewed SI, each manuscript must be self contained and convincing on its own merits:
A change is needed if we are to maintain the integrity and value of peer-reviewed articles. We believe that this is best accomplished by removing the supplemental material from the peer review process and requiring that each submission be evaluated and approved as a complete, self-contained scientific report […] With this change, the review process will focus on whether each manuscript presents important and compelling results.
I think most scientists can agree that large SI’s present a challenge to the scientific method as we know it. As is argued by JNS, large SI’s present a challenge to referees and authors alike and contain the potential for a potentially harmful “SI arms race”.
But let’s consider the suggested solution. In my interpretation, the proposed solution is to introduce more trust into the process. By eliminating the peer reviewed SI, the Editor-In-Chief is effectively stating that referees should trust that the authors have done their legwork (data preprocessing, programming, statistical analysis, and other “boring” elements underlying the main results) properly.
Of course, the entire foundation of peer review is trust. As referees we begin our task trusting that authors have done their work properly and presented their results honestly. Even a good referee can only be expected to catch mistakes and problems in the material presented to him. So why not a little additional trust?
Personally, I am unsure what to think. On one side, I wholeheartedly agree that there are important problems with the current state of affairs. But, on the other side, I think that there are important arguments against allowing too much of the ‘legwork’ to left out of the peer review process. Firstly, examples of scientific misconduct are many and the elimination of peer reviewed SI will make sloppy or dishonest science easier. Secondly, and more importantly, as John Timmer at Ars Technica has recently pointed out, the increasing use of computers could potentially put an end to the entire concept of scientific reproducibility (precisely because of extensive preprocessing of data, etc). Without peer reviewed SI, this problem will even more difficult to counter.
Regardless of the pros and cons, this is an interesting move by JNS. Since JNS allows fairly long articles (typically over ten pages), getting rid of the SI might be easier for JNS and other journals aimed at specific scientific disciplines, than for highly cited interdisciplinary journals – say Science or Nature – where word-count restrictions for main text are taken very seriously.
It will be interesting to see if this policy of “no supporting material” catches on.
sune lehmann 