Journals of Irreproducible Research – downgrading reproducibility and fact checking

As far as I am concerned, good science is about communication and reproducibility, or, as Stephen Heard argues, at least being able to believe that it is reproducible.  I would argue a bit more strongly than Stephen, in that I think you should, at the very least, be able to be confident that you could reproduce the experiment without having to contact the author(s) and that you can also easily check the cited literature.   In this context, there are two things that really annoy me about some of the so-called ’high impact’ established print journals and their on-line would be rivals.  First, the way in which the methods and materials section is relegated to the end of the paper, often in smaller font, and in some cases to the supplementary material section  In other journals e.g. Nature, the methods section is also very minimal and I defy anyone to repeat those experiments!  My second bugbear is the habit that some journals have, possibly to reduce space, in making you use numbers to denote references, placing them either in parentheses or superscript in the main text.

Perhaps I am alone in this, but I do like to know whose work is being cited without having to constantly refer to the references section.  What  particularly annoys me, are those journals that not only insist on numbered references but then list them in number order and not in alphabetical order!  I once wrote a review paper for Annual Review of Entomology, which has the numbering system, but subverted it by listing my references alphabetically – the editor never noticed 😉

You may say that what all these journals are doing is merely structuring the paper in the order that people tend to read them which is, I admit, a valid point. To me however, they are saying to the scientific community, perhaps not overtly, but certainly subliminally, that methods and materials are something you don’t really need to bother about, somewhat akin to those things that you store in an attic or basement, just in case you might want them at some time in the future, but probably not often, if at all.

Hidden methods

This sends a strong and erroneous message to authors that despite the methodology being the most important part of how we do our science, as long as they report the general gist of how they did things it is fine.  To referees the subversion of the methods section sends an equally strong signal; you don’t really need to spend a lot of time reading about the methodology as long as the rationale for the work is justified and that the results are significant and well presented.

As someone who works on insect-plant interactions I constantly come across inadequate methods and materials sections both as a referee and as a reader of published work.  The thing that perhaps causes me the most annoyance are descriptions of plant phenology.   Herbivorous insects have a very intimate relationship with their host plants and the growth stage of their host plant or the age of the plant tissue that they are feeding on can have very marked effects on their development, survival and fecundity (Awmack & Leather, 2002).  I so often came across methods descriptions along the lines of “10 day-old cabbage seedling” “ 3 week old pepper plant”,  “2 week-old wheat plant”, that in desperation I wrote an editorial (Leather, 2010) explaining how important it was to use a measure that didn’t depend on the temperature,  photoperiod, nutrient or water status that the plants were grown at i.e. the BCCH scale.  I also compiled a virtual issue of Annals of Applied Biology, with relevant examples drawn from the journal which has a long and distinguished history in publishing such articles.  If you can’t find your host plant in past issues of the Annals you will find that most plants have a published version somewhere, even if only on Wikipedia.  Despite my efforts however, I still often have to remind authors to describe the phenological stage of their host plants accurately and precisely.

Methods and materials, please come back, we need you!



Awmack, C. S. & Leather, S. R. (2002). Host plant quality and fecundity in herbivorous insects. Annual Review of Entomology 47, 817-844.

Leather, S. R. (2010). Precise knowledge of plant growth stages enhances applied and pure research. Annals of Applied Biology 157, 159-161.




Filed under Bugbears, Science writing, Uncategorized

6 responses to “Journals of Irreproducible Research – downgrading reproducibility and fact checking

  1. Oh my goodness, I was just whingeing about this today! I absolutely detest the numbered citation system, period. But the Mystery of the Missing Methods is so distressing! For early career researchers/PhD students, it can be really disheartening when you are attempting a new analysis/methodology. I was looking at a Nature paper today about evenness of pest/natural enemy communities in ag fields. The postcript ‘Methods’ contains 2 paragraphs explaining the basics, but not the analysis of evenness indices (which I was wanting to read about), with a footnote saying ‘Full Methods available in the online version at the Nature website’. But, lo & behold, the details weren’t at the online version either. As an undergrad student some years ago, I was told numerous times that the best way to learn a new analysis/sampling approach was to ‘read the methods’ of similar papers – doesn’t seem to apply anymore.

    Liked by 1 person

    • yes indeed – really annoying when you can’t find the detail you want. Also annoying is when the author gives rough outline and says for full methods see Bloggs (2000) and you get there and it bears no resemblance to what they said in first paper so you contact author and they say ‘oh yes, we don’t do it that way anymore’ True story – happened to a PhD student of mine recently!!

      Liked by 1 person

  2. “What particularly annoys me, are those journals that not only insist on numbered references but then list them in number order and not in alphabetical order! I once wrote a review paper for Annual Review of Entomology, which has the numbering system, but subverted it by listing my references alphabetically – the editor never noticed ;-)”

    Hi Simon – I have to say that I’ve never had a problem with numbered reference lists; the role of any reference list is to allow readers to see what you are citing and to follow the paper trail back if they wish. For me, numbers work as well as letters (though I take your point about visibility of authors in the text).

    With that in mind I don’t understand how a reference list can be both numeric and alphabetical, and still allow the reader to trace the citation.

    Have I misunderstood?


    • Hi Jeff

      arranged alphabetically but with the number next to them so 12. Adams et al (2011), 5. Axelrod et al., rather than 1. Zalucki et al (2014) 2. Nobbs et al. So much easier if you are scanning a reference list looking for a particular author after you have read the paper.

      Liked by 1 person

      • OK, I get you now. But then you have the problem that tracking down reference 1, 2, or whatever becomes problematical, especially for very long reference lists.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s