Evidence synthesis
An important theme of research in our department is evidence synthesis, that is, combining knowledge from different sources. These sources can be multiple studies on the same topic, for instance, resulting from replication studies. But also other sources of information can be synthesized, e.g., expert knowledge can be incorporated in the analysis of empirical data. We distinguish three lines of research where methodology for evidence synthesis is developed.
Aggregation of results from replication studies
One application of evidence synthesis is the aggregation of results from replication studies. In social and behavioral science, the gold standard for scientific evidence is replication: finding results that are consistent across multiple independent studies. There are two types of replication studies: exact or direct replications and conceptual or indirect replications. Each type requires different methods for aggregation.
The aggregation of results from exact replications can be on the level of the estimates or effect sizes, as in meta-analysis or in Bayesian sequential updating. Both methods require that studies are highly similar, for instance, similar in design, variables, and statistical models. This is usually the case in direct or exact replication studies.
Alternatively, we may wish to aggregate results from conceptual replications. These studies will often be too diverse to aggregate on the level of parameter estimates or effect sizes. However, for each study a Bayes factor can be computed that expresses the amount of relative evidence in that study for two or more hypotheses of interest. The underlying idea is that, although the study specific statistical hypotheses may differ between studies, due to variations in designs and statistical models, they all represent the same underlying common research hypotheses. Resulting Bayes factors per study are combined by an updating procedure at the level of model probabilities. This type of evidence synthesis, on the level of Bayes factors, has been developed in the context of Informative Hypotheses (the linked page provides more information).
Aggregation of diverse sources of information
Another application of evidence synthesis is the aggregation of diverse sources of information in a broader sense, for instance, by combining qualitative and quantitative data. Qualitative data can be used to inform prior parameter distributions that are subsequently used for the analysis of quantitative data. Alternatively, qualitative data can be used to formulate informative hypotheses that, again, can be evaluated with a quantitative study. A third example is the use of text mining in a systematic review, allowing quantitatively aided qualitative synthesis. Another way to obtain informative prior parameter distributions is by using expert knowledge. Several procedures for expert elicitation have been investigated and results have been applied in different contexts. See also the page Bayesian Statistics.