Monday 22nd January 2018, 12.00 to 17.30
Zeeman Building, Room B3.02

Organiser: Simon French

Although we live in a data-rich age, it is not true that we have or ever will have sufficient data to evaluate all potential future events, risks or opportunities. Some will have novel or unexplored characteristics which require that we need look to experts for guidance. Implicit use of expert judgement has always been present in any risk or decision analysis, if only in selecting the models and analytic methods to use. But it is now common to use panels of experts to provide judgement of key uncertainties when there are few relevant data.

The seminar will explore theoretical advances and practical applications of expert judgement in many domains. Judgements are subject to behavioural biases and thus it may be necessary to calibrate and debias the judgements. That is not easy to do theoretically and in some circumstances it may be unethical to do so.

Please note that you need to register for the meeting: see panel to the right.

Schedule

12.00	Registration in Room B1.37 and Lunch in the Mathematics Common Room
13.00	Simon French (University of Warwick) Introduction to the Seminar and Overview of Structured Expert Judgement
13.30	Martine Barons (University of Warwick) Expert Judgement in Food Security and a Quantitative Approach to Policy Design
14.05	Robert MacKay (University of Warwick) Calibrated panel assessments
14.40	Tea in the Mathematics Common Room
15.10	Laura Bojke (University of York) Eliciting subjective priors for cost-effectiveness modelling
15.45	Mark Burgman (Imperial College London) Don't trust me, I'm an expert: getting the best out of experts
16.20	David Hartley (University of Warwick) A Bayesian Model for Calibration and Aggregation of Expert Judgement
16.55	Reception in the Mathematics Common Room

Abstracts

Expert Judgement in Food Security and a Quantitative Approach to Policy Design

Martine Barons (Applied Statistics and Risk Unit, University of Warwick)

Calibrated panel assessments

Robert Mackay (Department of Mathematics, University of Warwick)

When a panel has to assess a set of objects, it may be impractical for each panellist to assess each object, or they may have varying degrees of expertise for the different objects. They may also use the marking scale in different ways. How to combine the scores? We propose a method called Calibrate with Confidence. For each object that a panellist assesses, they provide a score and an uncertainty. The method combines all these to produce a “true” value for each object, a “bias” for each assessor, and posterior uncertainties. The results of some tests are reported. There are many contexts in which the method could be helpful (e.g. REF, grants, exam boards, appointments)

Eliciting subjective priors for cost-effectiveness modelling

Laura Bojke, (Centre for Health Economics, University of York)

Health care decision problems are generally complex. Not least because they typically involve a number of alternative courses of action, but also because a range of health outcomes and cost implications may need to be considered. Such decision are often informed using cost-effectiveness modelling, however evidence to inform such models is often uncertain. Subjective priors in the form of formal expert opinion can provide valuable information to inform such assessments, particularly where evidence is missing, less developed or less appropriate. This presentation attempts to summarise the available guidance on expert elicitation for health care decision making. The choices available and complexities arising are here illustrated using two examples: a medical device (negative wound pressure) and a diagnostic test (Photo Acoustic Mammography).

Don't trust me, I'm an expert: getting the best out of experts

Mark Burgman (Centre for Environmental Policy, Imperial College London)

You may have heard the provocation that “Experts are about as good at predicting the future as dart throwing monkeys”. However, research has shown that expertise in prediction is not related to being an expert (or SME, as we would call them) in the traditional sense, but to the way individuals think about problems and (importantly) these skills can be learnt.

A Bayesian Model for Calibration and Aggregation of Expert Judgement

David Hartley and Simon French (Department of Statistics, University of Warwick)

A Bayesian hierarchical model to update opinion based on the aggregation of the judgements of multiple experts, elicited in the form used by Cooke's Classical Model, has been developed. This model utilises clustering techniques to deal with correlations between experts and looks to recalibrate their judgements using the elicited seed variables, subsequently forming an aggregated distribution over the target variables. Initial results suggest that the method can be effective, although there are challenges when seed variables have been assessed on very disparate scales. Further developments of the model are discussed.

See also:
Mathematics Research Centre
Mathematical Interdisciplinary Research at Warwick (MIR@W)
Past Events 
Past Symposia