Preprint reviews by Lasse Kliemann

The Reproducibility Of Research And The Misinterpretation Of P Values

David Colquhoun

Review posted on 09th August 2017

I have a couple of questions and comments on this article. Let us start with one particular issue in this post. Maybe the others will become clear for me during the course of the discussion.

The way the theoretical computations are conducted in Section 5, it appears to me that it is important to have exactly *one* distribution under H1. This is the situation given in Figure 1: one density for H0 and one density for H1. This way, the likelihood ratio immediately makes sense and is easy to compute.

Thus we not only need to know that the difference between the two means is 1 under H1, but also we need to know which of the two groups has the higher mean under H1. Only with this knowledge, we can, by an appropriate ordering of the sample means in the test statistic, ensure that, under H1, this test statistic has the Student's t distribution with the positive non-centrality parameter as shown in Figure 1. So if x and y are the two samples and we know that, under H1, x stems from the group with mean 1, we would use as test statistic:

t(x,y) = (mean(x) - mean(y)) / sqrt(var(x) + var(y)) * sqrt(n)

Here, n = length(x) = length(y). A one-sided rejection region would then be appropriate, so the p-value would be p.val = 1 - pt(t(x,y), df=2*(n-1)).

However, a two-sided test is used in the article. Since the two-sidedness strongly influences the likelihood ratio (factor 2), this is an important detail.

Most certainly I am missing something. Could you help me shed some light on this? Thanks.

show less

See response