But are they similar enough? Accounting for between-study heterogeneity when specifying informative prior distributions in small-sample situations – Christoph Koenig

Bayesian methods have repeatedly shown to be advantageous for small-sample situations. To benefit from these advantages, researchers are required to quantify existing background information in informative prior distributions, which are currently used only scarcely. A prominent reason for this may be the distinct heterogeneity of studies in psychological and educational research. Studies are being conducted under different circumstances, with different samples and varying instruments, and not all available background information can and should be used to specify informative prior distributions for a study. Consequently, to specify adequate informative prior distributions for a study, it is necessary to assess its similarity to previously conducted studies first and weigh them accordingly. In this talk, I will present a quantitative method to assess the similarity of studies in terms of sample and variable characteristics, unifying existing approaches to quantify between-study heterogeneity and the comparability of samples, and to weigh prior information in Bayesian analyses. Moreover, I will illustrate its use and its performance in terms of parameter accuracy with results of a simulation study, comparing the performance of the novel weighted informative prior with the classic power-prior by Ibrahim and Chen (2015) and the meta-analytic predictive prior by Neuenschwander et al. (2010). These results will be discussed in the context of the utility of the novel weighted informative prior for psychological and educational research as well as possible generalizations beyond simpler regression problems.

Christoph Koenig
Goethe University Frankfurt