Evaluation of the NHS Partnership with Virginia Mason Institute*
End of First Year Report
Nicola Burgess† and John Richmond
* This Evaluation of the NHS partnership with Virginia Mason Institute is funded by The Health Foundation. The Health Foundation is an independent charity committed to bringing about better health and health care for people in the UK.
† Corresponding author: firstname.lastname@example.org
This executive summary synthesizes preliminary findings from the evaluation of the NHS-VMI partnership led by researchers at Warwick Business School. Across the first twelve months of a three year evaluation the research team have worked in collaboration with NHS Improvement, Virginia Mason Institute (VMI), the CEOs of the five NHS partner trusts and the KPO teams at each of the five trusts. The generosity of time and openness of access to all partnership organisations and their employees has been exceptional, testament to the desire of the partnership to facilitate transparency and a genuine desire to evaluate and reflect on the effectiveness of the work so far. Subsequently the evaluation has collected a substantial volume of qualitative data in the form of interviews and observations of meetings and events. At the end of year 1 the evaluation team has conducted:
- 98 interviews at senior levels of each partner organisation, recorded and transcribed verbatim;
- 5 hours of observation, including monthly observations of the Transformation Guiding Board (Partnership level) and Transformation Guiding Teams (Trust level);
- A survey of Lean for Leaders for which we have achieved a response rate of 42%;
- Social Network Analysis at senior and middle levels of the partnership (349 respondents).
In addition, the research team has collected:
- Metric and performance data pertaining to 35 value streams and 80 RPIWs across five partner NHS Trusts.
The five partner trusts are:
- University Hospitals Coventry and Warwickshire NHS Trust (UHCW)
- The Shrewsbury and Telford Hospital NHS Trust (SATH)
- Barking, Havering and Redbridge University Hospitals NHS Trust (BHRUT)
- The Leeds Teaching Hospitals NHS Trust (LTHT)
- Surrey and Sussex Healthcare NHS Trust (SASH)
Each NHS partner organisation is hereafter referred to by their abbreviated name: UHCW, SATH, BHRUT, LTHT, SASH.
Summary of key partnership aims and progress towards them
The partnership aims were set out in the tender for evaluation as follows:
- For each trust to work in partnership with VMI and NHSI to develop localized versions of the Virginia Mason Production System (VMPS);
- To build a sustainable culture of continuous improvement capability across each organisation.
But how will we know if we’ve achieved these aims?
We asked representatives of NHSI, VMI and the five CEOs “What does success look like?”
A lively discussion ensued with each CEO reflecting aspirations based on their varied operating contexts and progress to date. The conversation concluded that success is best summarized in terms of the capability of each partner organisation to sustain a culture of continuous improvement:
“By the end of five years we expect each trust to have sufficient capacity in their organisation to build on this journey themselves without necessarily getting support externally. They have a sustainable culture of continuous improvement. The journey – they can carry on themselves.”
(NHSI respondent A)
However, much caution was expressed about the use of specific metrics as measures of success, with a reminder that the performance of these five NHS trusts is measured in the same way as every other NHS trust (emphasis added):
We concluded we couldn’t set at board level an artificial set of extra objectives, - careful not to set a separate set of objectives from what other trusts are facing. All NHS trusts are measured within an inch of their lives. We had a long debate about this – but we agree – we hope to see improvements across three areas [quality of care, staff morale and financial position] but we aren’t going to say you have to be at a certain point by the end of the partnership.
[NHSI Respondent A]
Overview of Partnership Progress:
For our five partner trusts, year three of the partnership seems pivotal. At time of writing, four out of five trusts had made substantial progress with regards to developing localized versions of VMPS, with many examples of value streams completed and a significant number of employees -including a growing number of senior leaders - having completed the formal Lean for Leaders training. However, at this juncture the partnership presents a very mixed picture of progress and success.
Some early success
In addition to the many improvements that each of the five partner trusts have achieved through the partnership, three of our trusts achieved widescale improvements that improved their financial position, quality of care and staff morale. Most salient among these achievements we highlight the following:
- Largest ever financial surplus for LTHT;
- Outstanding CQC rating for SASH;
- Prestigious HSJ Patient Safety Award for UHCW.
LTHT reported its first financial surplus for four years during 2018 and their largest surplus ever. SASH have also reported a financial surplus for 2018, and boasts the lowest reference cost in England, an achievement the CEO unequivocally attributes to the organisation’s local adaption of the Virginia Mason Production System, known as SASH+. Further SASH has received an outstanding rating from the CQC, one of only a small number of trusts nationally to achieve this accolade. The CQC report mentions SASH+ a number of times, declaring “an exceptional culture of data-driven continuous improvement and transformation at the Trust, supported by a comprehensive meeting structure and detailed performance reporting processes.” (p.3). Further, the report also highlights the trust’s performance in the annual staff survey placing the organisation in the top 20% for the past 3 years (with some scores in the top four nationally).
At UHCW, the patient safety team received the HSJ Patient Safety Award for their innovative work to redesign the incident reporting process, dramatically improving the time to respond to an incident report, as well as the efficacy of that response. This work was reported in an article for The Economist: ‘Hospitals are learning from industry how to cut medical error’, published 28th June 2018.
- Two of our partner trusts were placed into special measures during 2018.
BHRUT was placed in financial special measures in February and SATH were placed in quality special measures in November.
At BHRUT, development of a local adaption of the VMPS had gained much less traction compared to the other four partner trusts. Progress with VMI methods had been stymied by a range of contextual complexities that have plagued the trust since the before the partnership had begun. Such complexities included: high turnover of executive level staff including Finance Director in 2017, the Medical Director and Chief Executive Officer during 2018‡; turnover of staff trained by VMI to lead the KPO§; and ongoing performance issues that have resulted in the trust being placed into ‘quality special measures’ between December 2013 and March 2017, followed by ‘financial special measures’ in February 2018 (less than a year after leaving quality special measures).
Despite this challenging context, the Interim CEO of BHRUT indicated in December 2018 that he intends to ‘reboot’ work in relation to the partnership, introducing a number of measures to relaunch the partnership work during 2019:
- 10 leaders to receive training from VMI in Seattle.
- Plans to run a 3P event – focused on outpatient services – facilitated by VMI in conjunction with the KPO during the first week of March 2019.
- Plan to train the ‘top 100 leaders’ over next 12 months via Lean for Leaders training (with a commitment to ‘backfill’ if required**).
¨ Year three has also proven a challenging year for Shrewsbury and Telford Hospitals (SATH) despite substantial progress with implementing VMI methods.
‡ We also note that the CEO of BHRUT was recipient of a kidney transplant during 2017, requiring approximately six months of sick leave. Unfortunately, this period of absence contributed to a perception of leadership instability in the Trust.
- KPO refers to the Kaizen Promotion Office. This is a central team of improvement facilitators, comprising staff who have successfully completed Advanced Lean Training (ALT). The team undertake all activity related to value stream improvements as well as educating cohorts of staff via the Lean for Leaders training programme. ** A primary reason for poor completion rates of Lean for Leaders training at BHRUT was known to be a lack of capacity, with some managers refusing to release staff for training.
The evaluation finds that SATH has made the most progress in relation to the technical aspects of implementing the methods in accordance with the VMI approach. SATH have invested considerable time, effort and resource towards building a sustainable continuous capability across the Trust. For example, SATH’s KPO is the best resourced in terms of number of trained facilitators per employee; aligned to this, they had the highest number of value streams active (joint with LTHT) and the most RPIWs†† completed; they also trained the most staff (see table 2 of main report for details‡‡).
Despite a coordinated and sustained effort aligned to the partnership, the CQC report finds some core services have deteriorated at SATH and as such they were placed into quality special measures during November 2018. During this time the Trust received intense media attention and scrutiny, however as at December 2018 the CEO has continuously affirmed confidence in the VMI method as a key mechanism for helping the Trust move forward.
The evaluation team are interested to understand why, despite adherence to the technical aspects of the VMI method, SATH has struggled to deliver improvements in financial position, quality of care and morale to the same degree as SASH, LTHT and UHCW.
We consider this apparent paradox to be related to issues of ‘culture’, and in particular the absence of ‘cultural work’ prior to the partnership, necessary to create a receptive context for change. We highlight the importance of cultural work pre-intervention in our preliminary findings below, and in greater detail in the body of the main report.
†† RPIW stands for Rapid Process Improvement Workshop, where a pre-selected team of employees come together for five days to understand a process, examine which process steps are value adding and which are not value adding and can potentially be removed.
Preliminary lessons for NHS leaders
Reflecting on the findings of year one we offer three ‘lessons’ for NHS leaders in relation to how best to build change capacity to bring about sustainable culture of continuous improvement at an organisational level in the NHS.
- The value of partnership and informal accountability
Observations of ‘partnership level’ monthly meetings of the Transformation Guiding Board (TGB) have identified the important role of partnership, where relationships are mediated by a shared goal and a sense of ‘relational authority’.
Senior members of NHSI, VMI and the five Chief Executives come together for one day every month to reflect upon their work, the challenges they face, the successes they’ve achieved, and to respectfully challenge and support one another in pursuit of a shared goal. That shared goal is best summarized as ‘developing a sustainable culture of continuous improvement capability’ within each of our partner trusts and to transfer learning more broadly across the NHS.
‘Relational authority’ is a term the evaluation team have employed to convey a different type of relationship observed between the CEOs of the five NHS trusts, NHSI and VMI. The concept of relational authority is perhaps best explained in contrast to ‘positional’ or ‘regulatory’ authority, where one partner has power and influence over others. Conventionally, a healthcare regulator exhibits ‘positional authority’ where power and status creates a relationship that relies upon adherence to policy, targets, and performance levers. Conversely, relational authority produces ‘informal accountability’ where chief executives discuss progress, highlight areas of challenge and all partners (including NHSI and VMI) share learning and experience to benefit the partnership as a whole.
- A new style of leadership (system leaders take note)
We reflect through our observations that the TGB produces an almost continuous and cumulative process of sharing and reflexivity that fosters inter-organisational learning about how system leaders can better support providers to nurture and spread a culture of continuous improvement across the NHS. Of note, considerable discussion over the course of 2018 has centred on the need for a different type of leadership, both from within NHS Trusts themselves but crucially also, from the system.
“As Chief Executives we’ve learnt that doing this properly requires a different style of leadership. It means adopting a coaching style that empowers staff to find solutions, that creates the time and space for them to do improvement and where our role focuses on removing barriers. More broadly it’s about demonstrating those core behaviours of ‘go see, ask why, show respect’...creating that leadership culture locally can be made more challenging at times
given the traditional ‘system oversight’ culture in the NHS.... [we believe] similar changes in leadership style and behaviours between the centre and providers [are required] as we, and others like us, are adopting with our staff.”
- Culture comes first. The importance of cultural work in creating a receptive context for engagement with the improvement method
Our preliminary analysis suggests that CEO’s leading ‘cultural work’ within the five-year period preceding the VMI intervention achieved better traction with clinicians leading to improvements of greater scale and pace.
We identify ‘cultural work’ in relation to the collective setting of strategy, goals and behaviours. For example, LTHT had used a crowdsourcing platform to enable its 18000 staff to share their views and opinions about the challenges and issues that the organisation faced, following which the organisation was able to collectively express a set of organisational values and priorities alongside the expected/desired behaviours of all staff. Similarly, SASH had worked to improve leadership culture and medical engagement in the years prior to the partnership while also setting a clear and concise set of priorities across the organisation. Prior to joining the NHS-VMI partnership the CEO at SASH had supported the organisation to move from ‘requires improvement’ in 2010 to ‘good’ in 2014 and subsequently ‘outstanding’ in 2019.
Conversely, SATH’s CEO was new in post when the partnership began in 2015, thus he was not afforded time pre-partnership to engage in cultural work and we are not aware of any ‘cultural work’ taking place pre-partnership at BHRUT. We note that SATH has the highest quantity of staff trained in lean methods and had completed the highest number of RPIWs across eight value streams during 2018. It is curious therefore that SATH hasn’t seen the same level of improvement as SASH and LTHT. Reflecting on the challenges of developing SATH’s transforming care system the CEO notes that the method had ‘nothing to plug into’, in other words the context was not receptive to change.
Since hospitals comprise multiple sub-cultures or ‘micro-cultures’, the evaluation finds the same principle applies also at the micro-level; where culture is receptive to change, greater traction will be achieved at faster pace. Thereby, we propose that ‘cultural work’ aimed at fostering respectful behaviours and establishing shared goals and values, should precede lean based interventions.
In summary, the five NHS trusts have each embarked upon a journey in partnership with NHSI and VMI towards developing a culture of sustainable continuous improvement capability. The partnership is just over three years into a 5-year contract, after which support from VMI will no longer be funded. Development of
a management system takes many years. Through illustrating the early successes and stark challenges faced by the five NHS partners we begin to unpick the reality of trying to develop a management system in the NHS and how the NHS as a system can better support our providers to develop a sustainable culture of continuous improvement capability.
As evaluators of the partnership, we have observed candid and intelligent reflections, reflexive learning and the creation of new knowledge via the meeting of the TGB, supported by the presence of senior members of NHSI and senior members of VMI. Insights gained here has benefits for the wider system as a whole.
We conclude our executive summary with a recommendation to system leaders to consider how best to harness and exploit the learning from the partnership more effectively so that both the system and NHS providers can continue to learn and reflect on how to best build a sustainable culture of continue improvement capability in the NHS.