Back in the 1980s [1], I distinguished three contexts in which one might wish to combine expert judgements of uncertainty:

• The Expert Problem: In this a group of experts are consulted by a decision maker who faces a specific real choice and is not a member of the group. She asks the group for their assessment of the probability of an event or for their uncertainty about some quantity or forecast. She must then assimilate their advice into her probability model and feed this into the decision analysis. She alone is responsible and accountable for the decision. In this context the emphasis is on the decision maker learning from the experts.
• The Group Decision Problem: Here the group itself is jointly responsible and accountable for the decision; and they are their own experts. They wish that to the outside world their decision appears rational and, possibly, also fair and democratic. Thus they may wish to combine their judgements in some formal structured way. In this context, the emphasis is on supporting rational, democratic deliberations.
• The Textbook Problem: The group is simply required to give their judgements for others to use in future, as yet undefined circumstances. The emphasis here is on reporting their judgements in a manner that offers the greatest potential for future use.

A few years ago back I surveyed the current state of the art in these contexts [2].

Over the years much has been written on the first two, which have the focus of a single decision context, but little on the third. The textbook problem arises when one needs to draw together expert judgements into a decision analysis when their judgements were made originally in a context free manner or perhaps for other decision contexts. The advent of more public and stakeholder participation in complex societal decisions along with the ready accessibility to reports provided by the web have led to an increased importance of the textbook problem, which in many ways parallels that of performing a meta-analysis of empirical studies. However, there are differences; and, as yet, there are no established methodologically sound approaches to its solution. Recently I published a paper on these issues [3].

[1] S. French (1985) Group consensus probability distributions. In J. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith (Eds) Bayesian Statistics II, North Holland, 182-203

[2] S. French (2011) ‘Aggregating Expert Judgement’. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. 105(1),181–206. Link.

[3] S.French (2012) ‘Expert Judgement, Meta-Analysis and Participatory Risk Analysis’ Decision Analysis 9(2): 119-127. Link.