Review for "Preprinting Microbiology"

Completed on 31 Mar 2017 by Jonathan A. Eisen . Sourced from http://biorxiv.org/content/early/2017/03/15/110858.

Login to endorse this review.


Comments to author

OK am not sure if this is the right thing to do but it seems right so am doing it.

I just reviewed this paper for a journal. The version submitted is the same as this latest version here. And so I am going to post my review here. I guess this is very meta - posting at the preprint site reviews I made for a preprint about preprints for its publication in a journal

Review by Jonathan Eisen, UC Davis

ORCID ID 0000-0002-0159-2197

First a disclosure. I am on the Advisory Board of BioRXiv and am an editor at PeerJ.

Overall and general comments

1. This is overall an excellent and very useful paper on preprints in one general area (microbiology)

2. I believe the overall analysis is sound and the claims are generally supported by the evidence.

3. There are a few areas of the manuscript that could use modification – mostly just additional detail. I provide line by line comments for any part of the manuscript that I believe could use some modification below.

4. It would be good if all references to web sites include a data when the web site was accessed and in many cases it would be good to include some information from the site as supplemental material. After all, web sites can change and referencing them is tricky. More detail in line by line comments below.

5. The posting of a reproducible version of the analysis in Github is great and I would recommend highlighting this in the text when the analysis is first mentioned.

Line by Line Comments Below

---------------------------------------

Lines 18 and 19.

“Preprints were initially adopted among physicists and biologists in the 1960s as a method of sharing interesting research amongst colleagues (5)”

I think it would be good to discuss a little bit more the history of preprints and of sharing manuscripts prior to publication. The text here makes it seem like preprint sharing started in the 1960s when I believe this is untrue. The reference cited here appears to focus on organized efforts to share preprints in the 60s (e.g. an attempt by NIH and efforts by the physics community). While important these were not the beginning of preprints and thus the text is a bit inaccurate

In addition, I think it is not quite accurate to say this was a means of "sharing interesting research among colleagues". The efforts in the 60s were attempts to make preprints more broadly available and with less of a bias where some people saw preprints but others did not.

Furthermore, I think there were probably many reasons why people did this including establishing priority and staking out territory (i.e., it was not just about sharing interesting research among colleagues).

---------------------------------------

Line 36

“preprints are typically publicly available in about 24 hours”

Is there data supporting this claim

---------------------------------------

Lines 42-45

“This difference can be meaningful to authors since some journals, including the American Society for Microbiology (ASM) Journals, will only accept submissions that have been posted on preprint servers hosted by non-profit organizations”

Is there a reference for this claim?

---------------------------------------

Lines 48-49

“their work to a journal as journals will not consider manuscripts posted as preprints under a CC-BY license (e.g. Proceedings of the National Academy of Sciences). “

Should this read “to a journal as SOME journals”

---------------------------------------

Lines 53-55.

“Compared to the bioRxiv site, the PeerJ Preprint site is more fluid, gives readers the ability to “follow” a preprint, and provides better access to article keywords and the ability to search preprints.”

This should include a date of accessing the sites and I would recommend perhaps submit screenshots as supplemental material.

---------------------------------------

Lines 55-57

“With broader acceptance of preprints by traditional journals, many journals, including all of the ASM journals, have established mechanisms to directly submit manuscripts that are posted as preprints on bioRxiv.”

and

Lines 57-58.

“The only direct submission mechanism for manuscripts submitted at PeerJ Preprint is to the PeerJ journal”

A reference / web link would be useful here. Plus a date when these policies were examined would be helpful. I would recommend including the text of the policies as supplemental material if possible.

---------------------------------------

Lines 68-71.

“A relatively new example of what this might look like is PrePubMed (http://www.prepubmed.org/), which seeks to index preprints from numerous sources. A more organized effort is being initiated with funding through ASAPbio to create a “Central Service” that would aggregate preprints in the life sciences (http://asapbio.org).”

It would be good to include the date of access of these sites and if possible some additional information in supplemental material such as text from the site or screen captures to support the text.

---------------------------------------

Lines 72-73

There is also hope that the National Institutes of Health (NIH) will renew their interest in indexing preprints as separate research products than peer-reviewed publications.

Please clarify. Hope from whom?

---------------------------------------

Lines 81-83.

“First, a significant amount of attention has to be given to the potential dual use research of concern (DURC) since posted results in microbiology research could offer insights to individuals seeking to engage in terrorist activities”

DURC is not just about terrorist activities – it is about any possible misuse of research information.

---------------------------------------

Lines 83-85

“Second, for researchers engaging in research that involves human subjects it is critical that assurances be made that institutional review boards have been consulted and have approved of the research.”

I would suggest adding a line here about animal research too and how it is critical that appropriate reviews were done for such research.

---------------------------------------

Lines 89-90

“Again, while hoping to maintain the efficiency of the preprint format, traditional microbiology journals have policies for these issues in place that should be easy to implement by preprint servers”

Could you clarify or provide examples of what you mean by policies.

---------------------------------------

Lines 100-101.

“Each take a generally permissive stance towards posting of preprints prior to submission.”

Given the following sentences about journal policies changing it would be good to mention how the journal policy was inferred and if possible include references and or web links with dates and such.

---------------------------------------

Line 145-146.

“Considering the preprint is a citable work with a DOI, it would, in fact, be the preprint author that scooped the second.”

I personally agree with this statement but many in the community do not. This only works if people view preprints as valid publications. There are many examples where peer reviewed papers have been published claiming priority on some novel finding when preprints existed on the topic and where the authors of the peer reviewed paper and sometimes the editors of the journal have stated something akin to “we do not consider preprints valid papers”.

---------------------------------------

Lines 160-162.

“Some fear that the use of preprints will allow scientists to circumvent page limits by posting preliminary manuscripts.”

It would be helpful to reference examples of people expressing such fear if they are available online anywhere.

---------------------------------------

Lines 173-176

“In fact, several funding agencies including the Wellcome Trust and the UK Medical Research Council encouraging fellowship applicants to include preprints in their materials; meanwhile, the NIH is in the process of soliciting input from the scientific community on their role in grant applications.”

References / links would be very useful here.

---------------------------------------

Lines 193-195

“Any manuscript that was published went through several month delays in releasing information to health care workers, the public, and scientists needing to learn new methods to study a previously obscure virus.”

I think the wording here and in a few other places in the paper is a bit confusing or awkward. Preprints are published too. So I would suggest using some alternative wording to “published” for papers that have gone through peer review vs. preprints for those that have not.

---------------------------------------

Lines 207-214

Discussion of NY Subway metagenomics.

I think an important part of the story has been left out. On February 17, 2015 Chris Mason wrote a long blog post (at the request of this reviewer) for microBEnet about their paper and about some of the challenges in inferring the presence of anthrax in the subway. See

https://www.microbe.net/201...

---------------------------------------

Lines 265-267

“Although the hosted commenting is only one mechanism for peer review, this result was somewhat disturbing since the preprint model implicitly depends on people’s willingness to offer others feedback.”

Any information you could provide on the relative balance of commenting in other systems (e.g., Twitter, Facebook, Blogs) would be very helpful here. Or perhaps at least refer to the discussion later in the article

---------------------------------------

Lines 301-303

“Although it is impossible to quantify the quality or impact of research with individual metrics, it is clear that preprints and the publications that result from them are broadly accepted by the microbiology community”

I would love for this to be true. But I am not sure I completely follow the logic of this claim here. Could it not be that the citations for preprints are mostly coming from the authors of those preprints? I know we in my lab cite our preprints quite a bit. Or could it be that a subset of the community cites preprints a lot but others do not?