Quantitative Issues in Cancer Research Working Group Seminar

Sajia Darwish, PhD Student, Department of Biostatistics, Harvard T.H. Chan School of Public Health

Discussion of “What is the probability of replicating a statistically significant effect?” (Miller 2009)

Abstract: If an initial experiment produces a statistically significant effect, what is the probability that this effect will be replicated in a follow-up experiment? [This paper] argues that this seemingly fundamental question can be interpreted in two very different ways and that its answer is, in practice, virtually unknowable under either interpretation. Although the data from an initial experiment can be used to estimate one type of replication probability, this estimate will rarely be precise enough to be of any use. The other type of replication probability is also unknowable, because it depends on unknown aspects of the research context. Thus, although it would be nice to know the probability of replicating a significant effect, researchers must accept the fact that they generally cannot determine this information, whichever type of replication probability they seek.