Completed on 26 Feb 2018 by David Pascucci and Mattia Pagnotta.
Login to endorse this review.
Endorsed by 1 person.
While the view of large-scale brain networks as the biological basis of human cognition is taking deep root in contemporary neuroscience, a careful evaluation of the main functional and effective connectivity methods is becoming increasingly needed.
In the present work, Anzolin and colleagues accomplish a critical assessment of some of the mainstream methods for Electrical Source Imaging (ESI) and connectivity, which may become a key guideline for future work. In a set of simulations based on a simple three-node network, the Authors evaluated and compared the performance of two state-of-the-art algorithms for inverse solutions (‘Exact’ Low Resolution Tomography, eLORETA; Linearly Constrained Minimum Variance, LCMV) crossed with three connectivity metrics (Multivariate Granger Causality, MVGC; Time-Reversed Granger Causality, TRGC; and Partial Directed Coherence, PDC). The performance of each combination of methods was assessed as a function of the level of noise (Signal-to-Noise Ratio, SNR) and of the location of the three sources, which was defined by their relative distance and their depth in the brain.
The results of their simulation framework indicate that sources location may strongly influence connectivity estimation accuracy. In particular, a significant increase in false positive connections is generally observed when the sources are close to each other and deep in the brain, regardless of the type of connectivity estimator and inverse algorithm. Differently, sources that are far and superficial represent the most favourable condition.
In addition, this study further shows that TRGC provides better estimation than MVGC and PDC, thanks to reduced false positive connections under the same simulated conditions.
Finally, while eLORETA achieves the best performance when dipoles are far and is more robust against variation in SNR, LCMV results less sensitive to the distance between sources and the combination LCMV/TRGC helps to mitigate effects of volume conductions when sources are close and deep.
Altogether the results of this study provide a clear demonstration of the negative effect of volume conduction on the accuracy of directed connectivity estimates even at the source level.
Overall, the MS is very well written and organized; the main results are rather informative and provide a useful reference for the choice of different approaches in future work. The methods are clear and the ESI/connectivity metrics are well reviewed and described. The approach of using three nodes with fixed/moving constraints appears quite efficient in testing the effect of distance and topographical distribution of brain sources within a simple network.
It may be suggested to include more details about the simulation settings and to clarify some passages in the text which may improve the impact of the MS at first reading.
The following are some minor comments/questions along with suggestions to include more details in the MS:
Thank you all for the suggestions and for your ideas that will surely enrich our paper and will inspire our future work.
1) Would the general findings hold when more complex networks are investigated (e.g., distributed networks, with more nodes and more imposed interactions)?
We could generate more than three temporal series with imposed connectivity pattern (up to about 60), thus in the future of course we could validate such results on more complex conditions and variable number of dipoles on which reconstruct the simulated brain activity. What we currently think is that the accuracy in the source reconstruction and in the following estimates could be affected from all the factors you mentioned (more nodes and more imposed interactions) but also that the results of our comparative study will continue to be valid. Moreover, the amount of spurious links, that strongly depends on the position on the dipoles included in the model, can be limited on model of any size following the guidelines provided by this paper.
2) Is the overall pattern of results specific to networks without reciprocal interactions?
Different benchmark systems for Granger Causality, like the one used by Barnett and Seth, have reciprocal connections. In theory these measures are completely separable and independent by direction, but we will investigate the effect of their presence. For example, we evaluated the presence of an inversion among all the other false positives for each iteration as indirect measure of the phenomenon. We will mention this issue in the next version, possibly adding some supplementary material.
3) Do you think that the main results are invariant to the number of solution points used in the forward model (e.g., the resolution of the leadfield)?
I don’t think that in this kind of simulation framework the number of points used in the forward model could have an influence on the inverse-problem solution and for this reason we did not include this variable as a factor of the analysis. Could be interesting to investigate how performances change in extreme situations, such as projecting the source activity on a really low number of “electrodes”.
4) Were the dipoles simulated with free or fixed orientations? In the latter case, how was the main orientation determined during inverse modelling? Please report this information in the MS.
We assumed perpendicular source orientation following the simulation framework proposed by Haufe and Ewald in the Brain Topography paper . However, we obtained virtually identical results simulating freely oriented dipoles and determining their main orientation during the inverse modelling using the Principal Component Analysis (PCA). For this study, with fixed orientation, we used the 2D leadfield due to the significantly lower computational time.
 S. Haufe and A. Ewald, “A Simulation Framework for Benchmarking EEG-Based Brain Connectivity Estimation Methodologies,” Brain Topogr., pp. 1–18, Jun. 2016.
5) Since the PDC is a spectral measure, how were the FPR and FNR determined? The authors reported the use of the asymptotic statistic to determine significant connections in the frequency space. How was the presence/absence of an overall connection determined?
PDC validation was performed by means of a statistical test called “Asymptotic Statistics” . PDC in the null case tends to a χ2 distribution and its null-case distribution was built by using a Monte Carlo Method. The MATLAB function for PDC estimation provides for each frequency point a matrix (nodes x nodes) whose values are the connections, and a matrix with the estimated thresholds to filter the first one. After that, we just averaged all the 30 frequency points.
 D. Y. Takahashi, L. A. Baccalà, and K. Sameshima, “Connectivity Inference between Neural Structures via Partial Directed Coherence,” J. Appl. Stat., vol. 34, no. 10, pp. 1259–1273, Dec. 2007.
6) We were wondering whether, in addition to the index considered, having a measure of the fit of each connectivity model (e.g., the residuals or RMSE against the ground truth) would also aid to characterize and differentiate the performance of each method.
Of course, it could be an interesting parameter to observe. We did not include it in the current version of the paper just to remain focused on the problem related with the presence of spurious links because of the volume conduction/demixing effect. The accuracy of different Granger-causal connectivity estimators could be evaluated also independently of the type of linear-inverse algorithm employed for the source signals reconstruction.
7) We found no information about the length (samples) of each simulated dataset.
Each simulated dataset is composed by N=1000 samples. You can see all the details in the code we shared on GitHub but we will insert this information also in the text of the updated version of the MS.
8) How was the time window for covariance estimation in LCMV selected?
On the basis of the sampling rate. In other words we selected a 1-second time window.
9) Maybe you can consider changing the colour or shape of sender/receiver/neutral nodes in figures 6-8, the black-purple-red were hard to distinguish when the figures were projected on the slides.
Thank you for the suggestion.
10) In figures 7-9 a) there is an unexpected peak at 14cm, please elucidate more the nature of this rise in the text, which seems surprising at first sight.
While surprising at first sight, it is related to the presence of the second interactive dipole (receiver), located 14 cm away from the sender.
Please see this figure for clarification http://users.ugent.be/~dmarinaz/anzolin_reply_review.png
We will explain this better in the revised version.
11) Please report the optimal regularization parameter chosen after cross-validation.
The regulation parameter for eLORETA was imposed equal to 0.01.
12) In the captions of figures 6 and 8, panel b) and c) are inverted.
13) In the caption of figure 8, sender and receiver are mentioned instead of the sender and non-interacting dipole.
14) Eq.(25): since you referred to the original PDC definition, the non-squared version should be reported in the first definition.
15) Eq.(28): the second term should be weighted by 1-α (now is multiplied by α-1).
16) In eq. (24) it should be useful to specify the net score in the apexes as in Winkler et al. (2016). The same in the following paragraph.
17) It might be useful to introduce more about the main findings in the abstract.
I. Section 2.1, last paragraph, location should be plural
II. Section 3.1, in the subsection title about FPR, positives should be singular
III. Page 5, repetition: “it is not it is not”
IV. Page 11, second paragraph, third person for change
Thank you very much. We will correct all the figures, equations and typos in the updated version of the paper.