HMICFRS inspections: evaluation of remote inspection methods

Summary

Her Majesty’s Inspectorate of Constabulary and Fire & Rescue Services (HMICFRS) independently assesses and reports on the effectiveness and efficiency of police forces and fire and rescue services with the aim of encouraging improvement.

The pandemic required us to move to remote inspections, rather than on-site and face-to-face.

We evaluated how the move to remote methods went to:

• identify where improvements were required;
• understand what happens when we change the way we inspect; and
• consider whether inspecting remotely is something we could continue to do in the future.

We found that most people throughout HMICFRS supported using a blend of on-site and remote inspection activities. And police forces and fire and rescue services generally reported positive experiences of remote inspections.

Remote methods that we found to be effective were interviews, remote case file reviews, staff surveys, self-assessment, team debriefs and meeting observations. But large focus groups were harder than small focus groups to carry out remotely. We noted that all these methods rely on efficient IT capabilities (both at HMICFRS and in police forces and fire and rescue services).

Carrying out reality testing remotely was found to be challenging, particularly for police force inspections. This affected some inspection teams’ ability to triangulate evidence. Staff surveys were particularly useful where reality testing wasn’t possible. They helped with triangulation and allowed a wide range of force and service personnel to take part in the inspection.

The effect of working remotely on staff wellbeing was mixed, with some staff adapting well and others finding it more difficult. Difficulties included the intensity of working online all day without breaks (for example, to travel to different sites), the lack of team camaraderie compared to inspecting on-site, and, the challenge of balancing home and work life.

Implications for the inspectorate (summary section)

Using remote methods has allowed us to successfully continue with our inspections during the pandemic. Our evaluation gave us confidence in the evidence we collected remotely. It also allowed us to establish where problems existed and to address these promptly. Work has been carried out across the inspectorate to resolve some of the issues raised. For example, we clarified the process for accessing force laptops for remote case file reviews and began using staff surveys more often to mitigate any limits on reality testing. The findings have informed the guidance HMICFRS gives to teams using remote inspection methods, such as on the effective use of IT for focus groups and interviews.

The evaluation findings have also underpinned more permanent revisions to our inspection methodology, to include remote inspection methods where appropriate. HMICFRS is now using a blend of on-site and remote inspection methods.

List of abbreviations and definitions

Abbreviations

Abbreviation	Full term
FLO	force liaison officer
FRS	fire and rescue service
HMICFRS	Her Majesty’s Inspectorate of Constabulary and Fire & Rescue Services
HMI	Her Majesty’s Inspector
SLO	service liaison officer

Definitions

In this report, the words, phrases and expressions in the left-hand column have the meanings in the right-hand column assigned to them. Sometimes, there will be a fuller explanation after the definition, with references to sources and other material which may be helpful.

Term	Definition
case file review	a way to collect evidence that involves examining case files to gather information on the way cases have been dealt with; for example, assessing ways of working across the force/service, establishing compliance with local and national requirements, understanding the victim, witness, or offender ‘journey’
chiefs of staff	for each region, the HMIs have a chief of staff who oversees all inspections in their region; chiefs of staff are also responsible for managing the inspectorate’s relationships with the forces and services in their region
force liaison officer	police force staff who HMICFRS work with to co-ordinate the inspection activities
force debrief	a meeting where the inspection team verbally inform the force of the main findings from the inspection; this allows forces to start addressing the findings before their inspection report is published
reality testing	where HMICFRS tests what a police force or fire and rescue service claims is taking place in relation to a particular activity or function; this may involve an audit of file entries, observation, or conversations with staff
remote methods	inspection methods used to collect evidence without attending a police force or fire and rescue service in person
self-assessment	a tool used to collect a core set of evidence from all police forces and fire and rescue services that were inspected; the self-assessment tool included a mix of open and closed questions and was given to forces and services to complete before fieldwork activity began
service debrief	a meeting where the inspection team verbally inform a fire and rescue service of the main findings from the inspection; this allows services to start addressing the findings before their inspection report is published
service liaison officer	fire and rescue service staff who HMICFRS work with to co-ordinate the inspection activities
team debrief	an HMICFRS inspection team meeting, held during the inspection fieldwork, to share findings among inspection team members and identify gaps in evidence
triangulation	where evidence from one source is supported by evidence from at least two other sources

Background

HMICFRS independently assesses and reports on the effectiveness and efficiency of police forces and fire and rescue services with the aim of encouraging improvement. We inspect police forces across England and Wales and fire and rescue services in England.

How we inspect police forces and fire and rescue services

Evidence collection for our inspections involves a range of activities:

• On-site inspection fieldwork, where inspectorate staff visit the force or service and obtain evidence. Evidence is gathered by examining documents, observing policing and fire and rescue service activity (also known as reality testing), interviewing force and service personnel, and holding focus groups with force and service personnel.
• Document requests and self-assessments, where forces and services provide the inspectorate with evidence of their performance in relation to inspection questions.
• Data requests, which are carried out before inspections, to inform pre-inspection briefings and fieldwork. Data is submitted by forces and services and analysed to help identify areas of interest for inspection teams.
• On-site case file reviews, examining case files to gather information on the way cases have been dealt with. For example, assessing ways of working across the force/service, establishing compliance with local and national requirements, and understanding the victim, witness, or offender ‘journey’.

Evidence collected as part of our inspections is usually triangulated. This means there are at least two other sources that support the first source of evidence.

At the end of evidence collection, there is a force or service debrief – a meeting with the senior leadership of the force or service where the inspectorate gives initial feedback on what has been found.

The findings of the inspection are written into a report, which is quality assured and moderated, before being published on the HMICFRS website.

More information about our inspection methodology can be found here:

• Police force inspections: Inspection of the performance of Home Office police forces.
• Fire and rescue service inspections: How we inspect fire and rescue services.

Evaluating the move to remote inspection methods

In September 2020, due to the pandemic, HMICFRS moved to conducting inspections fully remotely, rather than carrying them out on-site in police forces and fire and rescue services. This followed a short suspension of all inspection work requiring appreciable contributions from police forces and fire and rescue services (from March 2020), so that they could focus on responding to the pandemic.

We evaluated this change from collecting evidence on-site to collecting it remotely.

The remote methods used for our inspections were:

• online interviews and focus groups;
• remote case file reviews;
• online self-assessments;
• observing meetings remotely; and
• online staff surveys (for our fire and rescue service inspections and our police custody inspections).

Not all of these methods were used for every inspection: methods varied depending on the inspection aims and needs.

The aims of the remote inspection evaluation were to:

• identify what was and wasn’t working well, to help improve the way we carry out remote inspections during the pandemic;
• understand what happens when HMICFRS changes the way we inspect to include more remote inspection methods; and
• consider whether this is something that we could continue with in the future.

Evaluation methodology

The evaluation was carried out internally, by HMICFRS analysts.

We gathered evidence from HMICFRS staff, police force personnel and fire and rescue service personnel who were involved in inspections carried out either fully or partly remotely between September 2020 and January 2021.

The inspections covered in the evaluation included:

• COVID-19 inspections of fire and rescue services.
• The policing response to COVID-19.
• Follow-up inspections of Greater Manchester Police and Cleveland Police.
• The National Child Protection Inspection.
• Custody Services in a COVID-19 environment.
• London Fire Brigade – progress against the Grenfell Tower Inquiry Phase 1 recommendations.
• Regional Organised Crime Units.

We took a mixed-methods approach to the evaluation, using surveys, interviews and focus groups. The evaluation was carried out remotely.

We collected evidence after inspections had ended, to make sure that the feedback we received couldn’t influence our inspection findings in any way. But we did this as soon after the end of inspection activities as possible so it was easier for people to recall them. We reported the findings without revealing the names of people or organisations that had contributed information.

Our research tools included surveys, and topic guides for interviews and focus groups. These were designed to collect evidence on what was going well and where problems had arisen, and about the quality and quantity of evidence collected using remote methods, compared to on-site inspection. Questions covered the following topics:

• wellbeing and work-life balance for HMICFRS staff;
• IT platforms;
• designing the inspections;
• experience of using each remote method;
• perceptions of the evidence gathered; and
• ability for HMICFRS to make judgments and recommendations.

The methods used are outlined below.

HMICFRS staff

Online survey

We sent an online survey to all HMICFRS staff who were working on each inspection, as soon as they had completed their work on the inspection. The survey asked questions about staff’s general experiences as well as detailed questions about how each inspection method went. We received 64 responses: 40 from staff working on policing inspections and 24 from staff working on fire and rescue service inspections.

Focus groups and interviews

We held focus groups and interviews to collect more detailed information about using remote methods, how an inspection was carried out and the inspection evidence collected.

In total we held eight focus groups, to cover each inspection included in the evaluation. Each focus group had between four and five participants. Participants in the groups included staff leading inspections and those carrying out inspection activities.

These focused on:

• confidence carrying out remote inspections;
• how remote inspections compared to on-site inspections;
• how evidence collected remotely compared to that gathered from on-site inspections;
• any lessons learned; and
• wellbeing during remote inspections.

Due to smaller numbers and differences between regions and roles it wasn’t appropriate to run focus groups with regional chiefs of staff (who oversee all inspections in their region and manage the inspectorate’s relationships with forces and services in their region) and Her Majesty’s Inspectors. Instead, we carried out six one-to-one interviews with people in these roles.

The interviews focused on: their role in the inspection, and how this had changed; the quality of evidence; their relationship with the force/service; how they worked with their team during the pandemic; and their own and their team’s wellbeing.

IT platform survey

We asked HMICFRS staff and inspection participants from forces and services to complete a short survey about IT platforms following their inspection interview or focus group. This was to identify how conference call platforms were being used, how well they worked, and any problems people had faced. We received 27 responses: 15 from HMICFRS staff and 12 from force or service inspection participants.

Police force and fire and rescue service staff

Online survey – inspection participants

We sent a survey to force liaison officers (FLOs) and service liaison officers (SLOs) for them to distribute to all force/service staff who took part in any remote inspection activity. This survey asked questions about participants’ general experiences and opinions relating to HMICFRS remote inspections, as well as more detailed questions about inspection methods.

The survey was sent to participants straight after they had completed their inspection activity or their contact with HMICFRS inspectors.

We received 478 responses: 281 from police force participants and 197 from fire and rescue service participants.

FLO and SLO feedback

We asked FLOs and SLOs for feedback on how they found organising remote inspections with HMICFRS, both generally and in relation to specific inspection methods.

We received feedback from FLOs through a mixture of live online questions and polls (using anonymous responses) and breakout groups at the quarterly HMICFRS FLO meeting in December 2020. We sent SLOs questions by email, which we received 26 responses to.

Analysis of evaluation evidence

We carried out and analysed the surveys discussed above using SmartSurvey.

A note-taker made a record of the interviews and focus groups. An analyst then identified themes from these notes, which were reviewed by a second analyst. We then collated these analyses to identify common themes.

Findings from all parts of the evaluation were quality assured by a separate, independent analyst.

We reported our findings without revealing the names of people or organisations that had contributed information. We also reported the findings in stages. This allowed us to learn continuously and decide whether changes were required as the inspection programme proceeded.

At the end of the evaluation, we pulled together findings from all elements of the evaluation. And we made recommendations on how to carry out inspections in the future while COVID-19 restrictions remained in place and for when such restrictions were lifted.

Main findings

The main findings from the evaluation are presented below.

HMICFRS staff wellbeing

We examined the effect of inspecting remotely (rather than on-site) on staff wellbeing.

When we started inspecting remotely, many staff reported that their work-life balance was better than when inspecting on-site. But many people also felt uncertain about how their work-life balance would be affected if remote inspecting continued in the long term, and views on this subject were mixed.

Positives included the lack of travelling, more choice over food and the timing of food breaks, and reduced stress about being exposed to COVID-19.

Negatives included inadequate IT set up, the difficulty of switching focus between work and home or family commitments, and the lack of team camaraderie compared to inspecting on-site.

It was noted that it can be exhausting carrying out back-to-back remote inspection activities, which were sometimes scheduled this way due to the removal of travelling time. So although travelling time was saved by working remotely, the need for breaks means the time required to complete inspection fieldwork may not be shorter than working on-site if remote inspection work continues.

Inspection design

Inspection teams reported that there were very few changes to the design of the inspection, in terms of questions asked and inspection participants. The main consideration was whether the activity could be effectively carried out remotely.

Inspection methods

Inspection teams reported that generally the remote methods they used for their inspections were effective.

Inspection participants from forces and services also generally reported positive experiences of remote inspections.

For FLO/SLOs, organisational logistics were in many ways easier without the need to consider issues such as room bookings and travel arrangements for inspection teams. But organising calls got complicated where there were IT problems, such as forces and services using different conference call platforms to HMICFRS, or where firewalls needed to be navigated.

Interviews

We received positive feedback from HMICFRS staff and from force and service staff regarding online interviews. Our inspection teams reported that they were able to engage effectively with interviewees and collect similar evidence as they would from an on-site inspection – as long as there weren’t any IT problems. Scheduling interviews was easier without having to schedule travelling time. But rooms still needed to be booked (to ensure privacy), and IT problems (such as poor connections, navigating firewalls or using different call platforms) could be time consuming to resolve.

Some HMICFRS staff raised concerns about not being able to make sure that an interviewee was in a suitable environment for the interview (for example, where they couldn’t be overheard) and about this being harder to manage remotely than when inspecting on-site.

Focus groups

Small focus groups (with two to four participants) were easier to conduct remotely than larger ones (involving five or more participants). Focus groups were difficult to hold without video, and if all participants could not be seen on screen at once. Holding the groups remotely made them easier to schedule. It also allowed staff from a force or service working at different sites to participate in the same group easily.

Self-assessment tool

We designed a self-assessment tool for each inspection, and used these to collect a core set of evidence from forces and services. The tools included a mix of open and closed questions. We gave them to forces and services to complete before fieldwork activity began – either as an online survey or as a Microsoft Excel or Word document. We required FLOs and SLOs to seek the relevant information from staff and then submit a final version to us using SmartSurvey.

The tool collected valuable evidence. Analysing this evidence helped to give context to evidence from later inspection activity and in some cases provided evidence used for triangulation. Answers to closed questions in the self-assessments for FRS inspections helped to structure later inspection activity. Drawing out conclusions from the open questions across forces and services required a lot of HMICFRS staff time.

Meeting observations

We found that we were able to effectively observe meetings at forces and services remotely, particularly where all meeting participants were attending remotely. Where attendees were on-site and our staff remote, there were sometimes problems due to poor sound quality.

HMICFRS staff said that as long as the call technology functioned adequately, their experience was fairly similar to when observing a meeting in person, with the benefit of not having to travel there. Our inspection teams noted that meeting participants sometimes forgot that HMICFRS were observing. This allowed us to be less intrusive, but mitigations were introduced to make sure that meeting participants were aware of our presence to help maintain the transparency of our inspections. Mitigations used by our inspection teams included keeping their videos on throughout the meeting, and making sure they were introduced at the beginning of the meeting and to any latecomers.

Inspection team debriefs

Team debriefs are held during inspection fieldwork so the inspection team members can share their findings and identify any gaps in their evidence. These debriefs worked relatively well remotely, especially when video was used. Staff believed that the meetings would have been more informal if they were attended in person, and told us there was less opportunity to build team camaraderie remotely. But they said the remote debriefs were easier to arrange, as large meeting rooms didn’t need to be found and booked and people didn’t have to travel to one site.

Case file reviews

We received positive feedback from HMICFRS staff and force/service staff where it was possible to carry out case file reviews remotely – for example, by accessing files on police force IT. It was noted that forces and services use different IT systems, which can present a challenge for inspection teams to navigate.

Some FLOs raised concerns about needing as much notice as possible to lend force laptops to HMICFRS inspection staff – which is often required for remote case file reviews. The process for moving laptops securely between the force and inspectors needed to be clarified for HMICFRS staff and forces. A satisfactory process has now been agreed.

It was noted that case file reviews couldn’t always be done remotely, for example, if they involved particularly sensitive information.

Reality testing

We trialled using screen sharing and video calls with force and service personnel as a substitute for on-site reality testing. Inspectors would call a force or service and be put through to staff who were on shift at that time. Our inspection teams (particularly for policing inspections) reported that these methods weren’t as effective as carrying out reality testing on-site. This affected some inspection teams’ ability to triangulate evidence. Staff surveys were found to be useful to mitigate this problem.

Staff surveys (FRS and police custody inspections)

Staff surveys are an established part of our inspection methodology and were carried out as part of the FRS and police custody inspections. These worked well remotely and were particularly useful where it wasn’t possible to carry out reality testing. The surveys helped us triangulate evidence and allowed a wide range of force and service personnel to take part in the inspection.

Force and service debriefs

Force and service staff reported that having a remote debrief after the inspection fieldwork was complete was very useful. It allowed a larger number of staff to take part than was possible for on-site debriefs and allowed staff from different sites to participate without travelling.

IT platforms and tools

Initially, IT problems affected every part of the remote inspections, and were mentioned very often by participants throughout the evaluation. There were problems navigating firewalls (belonging both to HMICFRS and forces and services). And forces and services used a variety of different software for conference calls, which wasn’t always accessible to HMICFRS staff. Poor sound quality, poor video, and freezing were also reported.

In a survey, HMICFRS staff said the inability to predict IT performance was the biggest factor causing them to lack confidence in our ability to carry out inspections remotely.

Testing connections before inspection activities were carried out was reported to be crucial. The move at HMICFRS to using Microsoft Teams, which happened early in the inspection programme, largely resolved the IT problems faced and continues to do so.

Working with other inspection team members

Inspection teams told us they missed the camaraderie of on-site inspection team working, and that initially there were fewer informal discussions of evidence among team members. But arranging meetings online to discuss evidence and inspection findings was easier without having to find rooms or travel to meeting locations. And inspection teams adapted quickly to the new ways of working, particularly following the move to Microsoft Teams early in the inspection programme.

In particular, FRS inspection teams reported holding effective meetings to review evidence throughout the inspection, which were easier to organise remotely and helped streamline the report writing process.

Perceptions of evidence gathered using remote methods

Our staff noted that the evidence collected remotely sometimes lacked depth, particularly where reality testing wasn’t possible. This made it more difficult to triangulate evidence.

Inspection teams reported that there was a lack of ‘soft’ evidence that you can only get from physically visiting a site – for example, from notice boards and informal ‘corridor conversations’. This lack was particularly noted when inspecting culture in forces and services. The online staff surveys were found to be useful in helping to mitigate this problem.

Interview evidence was generally felt to be more reliable than focus group evidence, because focus groups were harder to conduct remotely than interviews.

The self-assessment analyses were useful for adding context to findings and helped with triangulation.

Overall, FLOs and SLOs reported being confident in the evidence we collected, and inspection participants said they were able to tell us everything they wanted.

Ability to make judgments

We didn’t make graded judgments for any of the inspections we included in this evaluation. But we asked our staff about their attitudes to making them in the future based on remotely collected evidence.

Many of our inspection design leads and chiefs of staff expressed reservations about the ability to make robust judgments based purely on remotely collected evidence. The biggest factor causing staff to lack confidence in their ability to make judgments in remote inspections was when there was a lack of on-site reality testing. Some staff felt that for some inspections they would need on-site inspection evidence to make robust judgments (for example, police custody inspections).

FRS inspection teams were generally more positive than policing inspection teams about the feasibility of making judgments based on remotely collected evidence.

All our staff said they wouldn’t make a judgment unless they were confident in the evidence.

Views on continuing on-site inspection activity

We found that most people throughout HMICFRS supported using a blend of on-site and remote inspection activities in the future.

Many staff told us that they would like the ability to continue carrying out inspection activities on-site in some instances. They felt this would be particularly important if the inspection subject was sensitive, or if there were IT problems.

Conclusions

We found that most people throughout HMICFRS supported using a blend of on-site and remote inspection activities.

Police forces and fire and rescue services generally reported positive experiences of remote inspections.

Remote methods that we found to be effective were interviews, remote case file reviews, staff surveys, self-assessment, team debriefs and meeting observations.

But large focus groups were harder than small focus groups to carry out remotely.

We noted that all of these methods rely on efficient IT capabilities (both at HMICFRS and in police forces and fire and rescue services).

Carrying out reality testing remotely was found to be challenging, particularly for police force inspections. This affected some inspection teams’ ability to triangulate evidence. Staff surveys were particularly useful where reality testing wasn’t possible as they helped with triangulation and allowed a wide range of force and service personnel to take part in the inspection.

The effect of working remotely on staff wellbeing was mixed, with some staff adapting well and others finding it more difficult. Difficulties included the intensity of working online all day without a break (due to lack of travelling), the lack of team camaraderie compared to inspecting on-site and the challenge of balancing home and work life.

Implications for the inspectorate (conclusions section)

Using remote methods has allowed us to successfully continue with our inspections during the pandemic. Our evaluation gave us confidence in the evidence we collected remotely. It allowed us to identify where problems existed and to address these promptly. Work has been carried out across the inspectorate to resolve some of the issues raised. For example, we clarified the process for accessing force laptops for remote case file reviews and began using staff surveys more often to mitigate any limits on reality testing. The findings have informed the guidance HMICFRS gives to teams using remote inspection methods, such as on the effective use of IT for focus groups and interviews.

The evaluation findings have also underpinned more permanent revisions to our inspection methodology, to include remote inspection methods where appropriate. HMICFRS is now using a blend of on-site and remote inspection methods.

Back to publication

HMICFRS inspections: evaluation of remote inspection methods