Review for "BIDS Apps: Improving ease of use, accessibility and reproducibility of neuroimaging data analysis methods"

Completed on 25 Jan 2017 by Titus Brown .

License: CC-BY

Login to endorse this review.

Endorsed by 1 person.


Comments to author

Author response is in blue.

This is a well written and seemingly comprehensive paper about the idea of using containerization technology (Docker & Singularity) and a slightly custom framework to distribute/provide apps for neuroimaging.

The writeup is well done, and with two exceptions, we have no major comments.

The major exception is that the Singularity discussion needs to be revisited. Right now it is somewhat too aspirational and does not clearly articulate the (major) drawbacks of singularity; we had to read between the lines and do some digging on our own to figure out where the failure points were.

Thank you for your feedback. We have improved the description of drawbacks concerning Singularity in various places across the manuscript (see below).

A few critical aspects to Singularity are either not mentioned or glossed over:

* it seems you still need root access on HPCs to *install* singularity containers.

This statement is not correct and must come from some misunderstanding. We have added a sentence to make this explicit:
“It is worth noting that copying and running Singularity images does not require elevated permissions.”

* the Docker-to-Singularity conversion approach is very inconvenient looking, and we feel it is a major drawback.

We acknowledged this shortcoming in the Discussion and mentioned an upcoming feature which will make the conversion easier:

“The need to convert Docker container images outside using docker2singularity tool is less than ideal. In the future, this will be a new feature of Singularity (currently in development) that will allow HPC users to directly import container images from Docker Hub (without the requirement of elevated privileges or access to Docker software).”

* the imposition of "read-only" mode on container execution is understandable but again should be highlighted as inconvenient.

We have added this point to the Discussion:

“Finally Singularity container images require root permission to create or modify (this is why conversion from Docker needs to be performed outside of the target HCP). On one side this can be perceived as inconvenience preventing users from performing quick modifications of container images. However, on the other side, the side effect of this limitation is that users are forced to create a new container image file each time an app is updated which promotes versioning and reproducibility.”

* Singularity *may* be installable *in theory* on many HPCs, but we don't have any idea of its adoption in practice. This could be addressed by an explicit comment that it's still early days but that at least the situation is likely to be better than it is with Docker.

We added a comment in the Discussion listing describing existing Singularity compatible systems:

“As mentioned above Singularity due to its minimal system requirements is easy to install. We are currently aware of 51 HPCs spread around the world that support Singularity (https://docs.google.com/spreadsheets/d/1Vc_1prq_1WHGf0LWtpUBY-tfKdLLM_TErjnCe1mY5m0/pub?gid=1407658660). This list includes systems located at prestigious institutions such as National Institutes of Health, Massachusetts Institute of Technology, and Stanford University as well as the 17th most powerful HPC in the world - Stampede, located at Texas Advanced Computing Center (https://www.top500.org/system/177931). Nonetheless, Singularity is still fairly new and it is fair to assume that it is not installed on most existing HPCs.”

We think these points need to be made more clearly in the paper.

Our second major concern - depending on the Docker Hub for archiving and versioning is not a good idea, and (at the very least) some sort of caveat should be applied and some longer-term directions suggested.

We thank the reviewer for bringing this point to our attention. We have addressed it with the following paragraph in the discussion:

“The proposed way of archiving BIDS Apps - using Docker Hub - even though convenient and cost effective (using Docker Hub is free for public images) might pose challenges in the long term. Docker Hub does not make any commitments about the availability of images. To avoid potential loss of images uploaded to Docker Hub we plan to periodically backup all of the BIDS Apps in a repository dedicated for long term academic storage such as the Stanford Digital Library.”

Minor correction -- 'particiapant_label' is misspelled.

Fixed.

Many thanks for your feedback!

Sincerely,

C. Titus Brown

Luiz Irber