Review for "Predicting functional networks from region connectivity profiles in task-based versus resting-state fMRI data"

Completed on 20 Feb 2018 by Krzysztof Jacek Gorgolewski .

Login to endorse this review.

Endorsed by 1 person.


Significance

Functional connectivity as measured by resting state fMRI could one day be an important clinical biomarker. This paper attempts to push forward our understanding of intrinsic brain connectivity during rest and while performing a cognitive task.


Comments to author

Author response.

"Predicting functional networks from region connectivity profiles in task-based versus resting-state fMRI data" by Rosaro et al. has the potential to contribute to our understanding of resting state connectivity meaningfully, but is held back by a confusing and unclear presentation.

First of all, all the authors really appreciate the comments and suggestions made and are really sure that they will help improve and clarify the message and methodology of the paper. As a response to this revision, we have modified the manuscript in the way described below.

After reading the abstract, introduction and the methods section, I was not clear what the authors attempted to predict from connectivity measures. My best guess is that the task was to predict which brain network a brain region belongs to given a vector of its connectivity measures with all other brain regions. A task formulated this way is, however, straightforward if we assume correspondence of connectivity measures across all the input samples. This assumption means that the first value of the vector always corresponds to connectivity with region A, second with region B, etc. for all input samples. The consequence of such encoding is that the connectivity vector for region A will have a correlation value of 1 at first value of the connectivity vector. In other words, the identity of brain regions is represented as noisy one-hot-encoding. All the network or the classifier has to do is to figure out which brain regions correspond to which networks - something that could be done without any knowledge of brain connectivity.

Thanks a lot for raising such an important point. We were aware of this issue from the beginning and that is why we set self-correlations zero. This was mentioned already in a footnote, so in order to provide it with more visibility, we have decided to incorporate it the main body text through the following sentence at the end of section 2.2:

“Moreover, since self correlations, {\it i.e.} values of 1 in the Pearson connectivity matrices, would straightforwardly identify the ICN in a one-hot encoding fashion, we set these features to zero so that we guarantee that regions interactions to the whole brain are indeed the responsible for determining the ICN label.”

We can't just skip self-correlation component in the feature vector since we would be changing the feature vector each time, but the algorithm's architecture is fixed. One could also argue that setting self-correlation zero is still biased since for each node, we are focusing on the hypersurface orthogonal to the self-correlation vector (1 for self-correlation component and zeros in the rest) and therefore it is unique for each node. However, we saw in the beginning of the project that setting self correlations zero made the performance get much much worse when compared to having one situation. That's because "noisy" components take on and and now they determine the belonging of each node. That is, the pattern connectivity to the rest of the brain determines the belonging of a node to a specific intrinsic connectivity network. Anyway, we have also added a paragraph in the discussion sections talking about the possible limitations that this could trigger.

This is just speculation since I was not able to grasp the details of the analysis to confirm what was being predicted and how connectivity was encoded.

To improve clarity in the future revision of the manuscript, I recommend adding a conceptual figure presenting the prediction task in terms of dependent and independent variables (features and labels).

Thanks a lot for the recommendation. We totally agree with it and that is why we added a conceptual figure ( Figure 1) and a subsection in materials called “workflow”, in which we describe how data was manipulated to obtain the matrix of instances and features and how data was used in the subsequent machine learning analysis.

More specific comments:

- The abstract is confusing. "In this study we use a large cohort of publicly available data to test to which extent one can associate a brain region to one of these Intrinsic Connectivity Networks looking only at its connectivity pattern, and examine at how the correspondence between resting and task-based patterns can be mapped in this context." This sentence too long and convoluted.

Thanks for the comment. The sentence noted has been replaced in a shorter and clearer way, which reads as follows:

“We study the reconfiguration of the brain networks between task and resting-state conditions by a machine learning approach, to highlight the Intrinsic Connectivity Networks (ICNs) which are more affected by the change of network configurations in task vs. rest. We use a large cohort of publicly available data in both resting and task-based fMRI paradigms;...”

- Page 3: "we will explore..." -> "We will explore..."

Thanks for bringing up this typo, corrected now

- Page 4: missing citation for the HCP project

Thanks for pointing us to this. We have consequently added the following citation:

“Essen, D. C. V., Smith, S. M., Barch, D. M., Behrens, T. E., Yacoub, E., and
Ugurbil, K. (2013). The wu-minn human connectome project: An overview. NeuroImage, 80:62
– 79. Mapping the Connectome.”

and added to the the acknowledgements section the information required by the HCP initiative that reads as follows:

“Data were provided [in part] by the Human Connectome Project, WU-Minn Consortium (Principal Investigators: David Van Essen and Kamil Ugurbil; 1U54MH091657) funded by the 16 NIH Institutes and Centers that support the NIH Blueprint for Neuroscience Research; and by the McDonnell Center for Systems Neuroscience at Washington University.”

- Page 4: "has been proved to increase the quality of the original data" citation needed.

Thanks for the comment. Such an information was already in the text but in a wrong place and it corresponded to the following cite:

“Salimi-Khorshidi, G., Douaud, G., Beckmann, C. F., Glasser, M. F., Griffanti, L., and Smith, S. M. (2014b). Automatic denoising of functional mri data: Combining independent component analysis and hierarchical fusion of classifiers. NeuroImage, 90:449 – 468. “

As a consequence, citation was then moved to the appropriate place in the text

- Page 4: "connectivity map" might be a better term than "correlation image"

Thanks for the suggestion. We agree that it might be a better term, so we have changed it throughout the text

- Page 4: How was the assignment of each brain region to a brain network performed? Shen and Yeo's parcellations differ in region definitions.

Thanks for the comment. Such an information was indeed missing in the text, something that can cause misinterpretation. We added the following sentences at almost the end of section 2.2 to clarify this:

“ICN assignment to each of the 268 brain regions was performed by overlapping both Shen and Yeo atlas with a minimal 80\% threshold. As a consequence, if the number of voxels within a Shen parcel exceeding this threshold belongs to one of Yeo's ICNs, such a Shen parcel is identified to that particular ICN”

Also, we added a paragraph discussing limitations of this approach in the discussion section, which reads as follows:

“The interpretation of the results of the present study should be seen at the light of the following limitations. These are mainly related to the construction of the matrix of features and the definition of the labels to be classified.

First, the use of the correlations pattern to the whole brain of a given node as features that determine the ICN of that node should be treated carefully. In particular, setting all the terms but the self-correlation in the feature vector to zero, the resulting vector would directly provide the ICN label of the node with a 100 \% of accuracy since each node vector would be unique and orthogonal to the rest and therefore classification would not be needed whatsoever. However, our hypothesis is that ICN assignment should be predicted by the whole pattern connectivity of the node. As a consequence, and given that we are limited by the fixed dimensions of the feature vectors (these can not change across subjects), we set self-correlations values to zero so as to diminish their effect and allow the rest terms to truly determine the label of each example.

Second, the labels used to fit the model are not fully and uniquely defined since they are threshold dependent when matching Shen with Yeo's atlas. As a result, this spatial variability might somewhat blur the connectivity maps and therefore reduce the performance. ”

- Page 5: "Finally, the 282 resulting individual FC matrices were concatenated together" it's unclear if this was done separately for task and rest or the data was combined first. What dimension were the matrices concatenated along?

Thanks for the comment. In order to clarify this, we replaced the sentence by the following:


“Finally, for both task and resting data separately, the 282 resulting individual $268 \times 268$ Pearson correlation matrices were bound together row-wise to obtain ${\cal X}_{\rm task}$ and ${\cal X}_{\rm rest}$ super-matrices of $282\times \ 268$ instances and 268
features each. Both these matrices are then later used in the subsequent machine learning analysis..”

- Page 5: was the cross-validation performed across participants or nodes? Or both? If so why?

Cross-validation was at first performed across the instances, represented by the connectivity maps. For each node and patient we have a connectivity map whose label is known and belongs to a specific resting state network. Connectivity maps are the nodes correlation pattern to the whole brain obtained from the connectivity matrix, i.e., a row (or column) in this matrix. Since we have 282 subjects and 268 nodes per subject, we have 268*282 total instances whose label is known and we try to predict. However, we were warned by one of our collaborators that this would make instances from the same subject ( for each subject we have 268 instances) fall in the training and test datasets at the same time. This would lead to the so-called “double-dipping effect”, yielding inflated results. As a consequence, we changed the cross-validation to be performed across participants so that we ensure that examples from the same subjects are only used either in the training or in the test dataset. Likewise, now instead of using 5-Fold Cross-validation, we use a 5 times 10-Fold Cross-Validation in order to take, on one hand, a more appropriate choice of K (citation of this included), and, on the other hand, to assess results stability by repeating the cross-validation 5 times.

- Page 6: Table 1 is missing the MLP results

Thanks again for the comment. We originally showed the accuracy results from fitting the different models in two different objects: Figure 1 for results from Neural Network models and Table 1 for Random Forest and Support Vector Machines. In order to avoid misunderstanding, we have removed Table 1 and put all the results together in figure 1.

- Prediction accuracy on another dataset (with different acquisition parameters) would be good evidence of the robustness of your findings.

Thanks a lot for the suggestion. This is something we plan to do.