Skip to main content Skip to navigation

IP306 Impact Evaluation: Frameworks for Implementing and Assessing Transdisciplinary Interventions

The Embarkation of the Queen of Sheba, an oil painting by Claude Lorrain (born Claude Gellée, traditionally known as Claude), signed and dated 1648. The composition draws the eye to a group of people on the steps to the right, at the intersection of a line of perspective.
THIS MODULE IS NO LONGER RUNNING IN THE 2020-21 ACADEMIC YEAR
Dr Lauren Bird
Optional module
15 CATS
Term 2
10 weeks

Moodle Platform »

Prerequisites

Students with other backgrounds will be admitted on a case-by-case basis, though it is expected that they will have an equivalent level knowledge as students who have studied the conventional prerequisites. Please contact the module leader for further information.


Principal Aims

This module is for you if you're interested in understanding frameworks for the design and evaluation of policies, big or small. It aims to provide you with the skills and knowledge required to make informed decisions concerning the design of policy interventions, and the evaluation of their impact in a range of settings.

To achieve its aims, the module balances the necessary technicality of rigorous statistical and econometric analysis against the need for an intuitive operational understanding of the required steps when confronted with a policy problem.

Through extensive hands-on use of a range real-world examples and datasets, you will gain conceptual knowledge such as frameworks for effective policy design, core statistical knowledge, and a transferrable toolkit of practical analytical skills required for effective impact evaluation and other statistical analysis. You will then be able to apply this skill set to any situation where an intervention is required – either very locally within a business or organisation, or in terms of national- or international-level issues.

The module is taught via a combination of lecturer-led classroom discussions, problem-based learning, and practical computer lab sessions


Principal Learning Outcomes

Upon completion of this module, you will be able to:

  • Demonstrate an understanding of frameworks for policy design, be able to apply such frameworks to interpret existing policy, and use established frameworks as a tool for designing new policy interventions;
  • Demonstrate an understanding of the application of policy design frameworks to both localised interventions, and also broader applications;
  • Demonstrate an understanding of the importance of appropriate data collection and choice of intervention subjects, random sampling, and how data collection impacts on the ability to use sample data to make inferences about the broader population;
  • Interpret, produce, appropriately present, and explain (verbally and in writing) a range of descriptive statistics relevant to policy interventions;
  • Demonstrate (verbally and in writing) an understanding of the need for rigorous policy or intervention evaluation and be able to provide examples of good practice in this subject area; and
  • Identify and implement appropriate statistical methods for the evaluation of policy intervention based upon the characteristics of the intervention and the available data, and to be able to use these approaches to overcome challenges of identification of causality.

Syllabus

In order to facilitate the acquisition of knowledge and competency, the course is structured and taught via a number of core problems in the development and evaluation of policy which must be understood and overcome. Where possible, problems will be addressed through a case-study approach which focuses on the design and impact evaluation of real-world policy – including the study, replication, and extension of analysis carried out in reports and/or academic papers.

 I. How and why do we consider evaluation when designing policy?

    • What frameworks can we use to design effective policy?
    • Why do we evaluate?
    • What are the links between good policy design and effective impact evaluation?
    • The power of data: Descriptive Statistics, Indicators, Indices

 II. How should we collect data, what can it tell us, and what can it not?

    • Data collection, sampling methods, distributions, sample v population
    • Revision of correlation, statistical significance, spurious relationships, causality, formation of hypotheses

 III. Are we observing correlation or causation?

    • Revision of multivariate analysis: Principles, assumptions, methods, interpretation
    • Difference in Difference (DiD) approaches and the importance of a control group
    • Regression discontinuity

 IV. How do we overcome issues of problematic data?

    • The frequent issue of endogeneity, Instrumental Variables (IV) and the art of finding a good instrument
    • Overcoming imperfect control groups, non-compliance, and Regression Discontinuity Design (RDD)

 V. How can we maximise the impacts of our findings?


Illustrative Reading List

The main texts for the course are:

Gertler, P. J., Martines, S., Premand, P., Rawlings, L.B., and Vermeersch, C.M.J. (2016) Impact Evaluation in Practice (2e). The World Bank, Washington, US (freely available from the World Bank website)

Khandker, S.R., Guyatri, B.K., and Samad, H.A. (2010) Handbook on Impact Evaluation: Quantitative Methods and Practices. The World Bank, Washington, US (freely available from the World Bank website)

This will be supplemented by accessible technical sources such as:

Gujarati, D. (2015) Econometrics by Example (2e). Palgrave , UK

Other technical texts will be employed where necessary to ensure full coverage of the statistical techniques featured in the course

The course will also make extensive use of individual reports and journal articles relevant to the case studies used in the course.


Assessment

Coursework:

1500-word individual assignment (30%)

4000-word group project (1000-word individual contribution) (15%)

Practical:

20-minute in-class group presentation (10%)

Examination:

8 x weekly online quiz (5%)

90-minute computer-based exam (40%)