Is it possible to get a handle on the magnitude of this additional uncertainty?
Is this uncertainty reduced by the referee process?
Abstract: In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We will present our results in the seminar and release the paper on the very same day. It will be the paper’s world première.