Review for "Chiron: Translating nanopore raw signal directly into nucleotide sequence using deep learning"

Completed on 9 Jan 2018

Login to endorse this review.

Comments to author

Author response.

The paper proposes and implements a new base caller for ONT MinION data called Chiron. The main claim of the paper is that by working from raw signal directly, one can avoid potential errors in event detection steps.

However, this claim is not very well supported by the results. In particular, it seems that the performance of Chiron is very similar to other available tools, and in many cases they seem to be very similar to e.g. Albacore-1.1 that uses the event segmentation. So it does not seem convincingly shown that substantial increase in accuracy can gained by removing the event segmentation.

This is not correct. Table 1 shows that Chiron-BS is consistently better than Albacore v1.1 on bacterial and viral genomes at the read level. Moreover, following the suggestion of reviewer one, we have investigated the assembly-level accuracy (described above). We show that Chiron is superior to Albacore (v1 but also superior to v2) in generating highly accurate assemblies. We have added the following sentence in the discussion to reflect these new results:

“Bacterial and viral genome assemblies generated from Chiron basecalled reads all had less than 0.5\% error rate, whereas those generated by Albacore had up to 0.8\% error rate. This marked reduction in error rate is essential for generating accurate SNP genotypes, a pre-requisite for many applications such as outbreak tracking. “

These results conclusively demonstrate the benefits from removing the event segmentation step in base-calling.

Moreover, design of the deep neural network underlying Chiron is much more complex than the one used in other currently available tools. In consequence, the tool is very slow and on CPU (even if parallelized) it would be very difficult to use. When using a high-end GPU card, Chiron can process ~1600bp per second. By a conservative estimate, a MinION run produces over 30000bp per second, so one would need approx. 19 of these GPU cards to keep up with the speed of sequencing (ONT Albacore would need about 10 CPU cores to process such run on-line according to the authors' measurements, which is a much more realistic setting). Consequently, Chiron cannot be considered a practical tool.

As indicated above, we have removed the statement that indicated Chiron could be used as a real-time base-caller. However, we reject the characterization the Chiron is not a practical tool. In certain settings, obtaining the most accurate base-calls possible is extremely important. One such example is in SNP calling, e.g. accurate identification of SNPs conferring drug resistance. The fact that Chiron leads to up to a 50% reduction in base-calling error rate makes it a valuable tool.

Moreover, there are approaches to accelerating neural networks which may be used to accelerate Chiron. We have indicated this in the discussion as follows:

“Also there are several existing methods which can be used to accelerate NN-based basecallers such as Chiron. One such example is Quantization, which reformats 32-bit float weights as 8-bit integers by binning the weight into a 256 linear set. As neural networks are robust to noise this will likely have negligible impact of the performance. Weight Pruning is another method used to compress and accelerate NN, which prunes the weights whose absolute value is under a certain threshold and then retrains the NN\cite{han2015deep}.”

One interesting point of the paper is that they only used a limited amount of data for training and the network seems to generalize well. It would be interesting to explore this issue. Would using significantly more data lead to a significantly better accuracy? Is the use of training data more efficient than in the case of other available tools?

We agree that the fact that the Chiron Neural Network generalises well is an interesting feature. However, exploring this issue in depth is beyond the scope of this paper. Moreover, it would be extremely difficult to compare the generalisability of Chiron and Albacore precisely because it is impossible to 're-train' Albacore on less data, as it is a proprietary basecaller.