Did you impact your knife crime problem, and how?
Why assess when problem solving knife crime?
There are two main purposes of assessment in problem solving.
The first purpose deals with the here and now. It helps you determine whether a knife crime problem persists following the implementation of responses.
Knowing this can help you decide whether further problem-solving efforts to address the selected problem are needed. The second purpose of assessment is to learn lessons for the future – to understand how your efforts to reduce knife crime might inform your work going forward and to contribute to the wider evidence base about what is, and what is not, effective in tackling knife crime.
For the first purpose of assessment, it may be enough to know whether your local knife crime problem remains, regardless of whether your problem-solving work was responsible for any observed reductions. However, this won’t help you know whether to use similar responses if your problem comes back in the future.
For the second purpose of assessment, we need to know much more, including whether it was what you did that led to a decrease in knife crime and whether there were any side effects because of your activities. This approach will help you and others know whether your responses are worth trying when tackling new knife crime problems.
Meeting the first purpose of assessment is relatively straightforward. The second is more challenging, and will vary in its level of complexity depending on the scale of your local knife crime problem, the nature of the responses implemented, and the skills and resources available.
It is important to decide early in the problem-solving process what purpose is to be served by the assessment. Deciding this has implications for what you do in other parts of the SARA process.
For example, if you are aiming to learn lessons for the future, you will need to start planning your measurements before any responses are put in place. If you find that numbers of knife-enabled robberies have declined and then look for evidence that what you did was responsible for those falls, this risks producing biased findings.
This section of the guide provides practical guidance for conducting both kinds of problem-solving assessment.
First purpose of assessment – are continued efforts needed?
As part of scanning, you will have quantified the specific type of knife crime problem you are addressing. Simply comparing the levels of the selected problem before and after your response will help you decide whether the problem has fallen enough for you no longer to need to devote resources to trying to reduce it. If the selected problem has fallen sufficiently, you may elect to close the current problem-solving project and move on to addressing the next problem. However, in relation to knife crime, there are some challenges for before-and-after measurement. The obvious starting point for this type of assessment is to repeat the measurements used in scanning to see whether the problem has dropped or disappeared. But data used to measure knife crime needs to be handled with care. There are several issues to be mindful of, which are given below.
Changes due to knife crime recording and reporting practices
As indicated in the scanning section, knife crime data generally derives from records kept by the police, hospitals and ambulance services. These records depend on decisions to report incidents or to seek medical attention, and then on how these incidents are classified. Changes in reporting and recording practices can therefore make comparisons of knife offences over time hazardous. For example, simply by selecting a particular knife crime problem as the focus of problem-solving work – such as domestic abuse – may mean that agencies become more diligent in adding relevant ‘flags’ to indicate that a knife has been used. This could potentially suggest an increase in the problem when in fact the observed increase may be a result of changes in data recording. Similarly, police services may become more attentive in looking out for incidents where knives have been used in crimes and classifying them accordingly. This means, for example, that an intervention may have had a positive impact, but this is masked by improvements in recording. Data cleaning may be needed to check that the information you plan to use in assessment makes sense and to remove any anomalous records. It may also make sense to clean data before embarking on an intervention, to make sure that knife crime has been recorded as accurately as possible. Where possible, comparison of numbers where the data is most likely to be robust – notably, homicides involving knives – will help you ensure that more discretionary reporting and recording practices do not explain apparent changes in the numbers of incidents in your local area.
Changes due to normal fluctuations in knife crime
The numbers of knife crime incidents at the local level tend to be small. Knife crime rates are therefore liable to fluctuate widely month by month, regardless of any problem-solving responses that are put in place. This fluctuation comprises a kind of ‘noise’ in the data. Obtaining a ‘signal’ relating to real change against this background noise is challenging. To illustrate this, see Figure 3, which shows the quarterly recorded knife-enabled robberies for Lancashire Constabulary from April 2012 to June 2020. It can be seen that even across a whole police service area and using quarters rather than months, the numbers of police-recorded knife-enabled robberies bounce around a lot, making identification of real short-term changes difficult. A consequence is that longer-time trends are useful for increasing confidence that a knife crime problem has actually fallen. It is important to take account of possible seasonal variations in levels of knife-related crimes when making before-and-after comparisons.
Second purpose of assessment – lessons for the future
Knowing that your local knife crime problem has reduced is different from knowing whether it was what you did that was responsible for that reduction. Knowing the latter is crucial for working out what can usefully be learned for tackling future problems.
But assessment for lesson learning has long been one of the weakest elements in the SARA process (Read and Tilley, 2000). This is partly because it is challenging to do well, especially with small-scale, local projects.
However, when attempting this form of evaluation, it is important to try to produce assessments that will be useful for others who hope to learn from your experience, both within and outside your organisation. It will also be useful for you should a similar knife crime problem emerge again.
Assessment for lesson learning means collecting and analysing the kind of evidence that others can usefully draw on. It also means making sure that assessments are honest. They must not claim more than can be justified from the data at hand. This too can be challenging.
It is natural to think that what we are doing is helping to resolve a pressing problem. Our confirmation biases tend to make us look for information that supports our hopes for impact and to disregard information that might dash them. Confirmation biases also mean that we are liable to draw false conclusions without any intention to do so. This can easily happen.
In relation to knife crime, in particular, there are often differing metrics that can be used as indicators of effectiveness (for example, the number of recorded offences, number of people hospitalised for stab wounds, number of people found with blades in stop and search, number of people charged with carrying knives as offensive weapons, and so on). It will almost always be possible to find one metric that does (or does not) indicate success.
Good problem solving involves calling it as it is. Failure on some occasions is inevitable. Moreover, failure is an important source of learning, and a stimulus for taking corrective action in the interests of continual improvement.
Knife crime projects – measuring the effects
Of the five EMMIE components (see Distinguishing outputs from outcomes, and why it matters), effects tend to receive most attention. This is understandable. Practitioners want to know whether an intervention has worked previously and hence whether it is worth trialling in the future. As discussed previously, this cannot confidently be known by simply observing whether a knife crime problem has changed following our problem-solving activities.
This is because there are many possible explanations for why crime goes up and down, including – but not limited to – the following:
- changes in economic conditions
- the movement of key offenders
- the increased availability of legitimate local opportunities
- the arrival of a pandemic
- freak weather conditions
- new housing developments
- a changed road layout
Few of these possibilities can be eliminated simply by observing that there has been a reduction in a problem following intervention. To identify what caused a change in crime, this requires methods of estimating the effects of your selected responses as rigorously as possible. Those methods are discussed here.
It is crucial to know how your response works when you are estimating the effects of that response. Put differently, to assess the impact of your response, it helps to have a clear idea of the outcomes you want to achieve and how your selected response might plausibly generate those outcomes. Logic models like those presented in the Response section serve this purpose – they depict a ‘theory of change’ showing the processes through which your intervention is expected to have an impact your local knife crime problem. We have already seen in the Response section how a theory of change can help you work out whether a selected response is appropriate for your local context. But it can also help you work out how to assess whether the intervention put in place is having its expected impact, and what you can most easily measure to check whether the response is working out as intended. The more specific your intervention, the easier it will be to find measurements to determine effectiveness.
Let us take the specific problem of knife-enabled robbery of school children travelling home from schools on buses as an example. We might introduce high-visibility police patrols on those bus routes at those specific times to increase the perceived likelihood of offenders being caught, with the aim of deterring robberies. If falls are occurring in ways that are inconsistent with our intervention – for example, knife-enabled robbery is falling at different times of the day – then that counts strongly against attributing the cause of the fall to our problem-solving endeavours. However, if the falls in robbery closely follow what we would expect to see if the intervention is working as intended, then that counts in favour of attributing the cause to the response we have put in place. Of course, we may still be mistaken, but the more closely we can specify expected chains of events that would have to occur if our response were responsible for the observed changes, the more it becomes up to others to suggest alternative accounts of what would explain the observed changes.
Specificity
Specificity is again important. Just as good problem solving calls for responses to be tailored to local conditions, good problem-solving assessment requires that you focus your evaluation on specific aspects of your local knife crime problem that might plausibly be affected by your choice of responses. We would not expect that police patrols deployed to high-risk bus routes would cause any change in the levels of, for example, knife-related violence associated with the night-time economy or domestic disputes involving knives. If knife crime is falling across the board, then it is likely that something else is going on to explain these patterns.
Counterfactual comparison
In trying to estimate whether a problem-solving intervention has been effective, we normally try to find some benchmark against which we can compare trends. In doing this, we are looking for a socalled ‘counterfactual’ – an estimate of what would have happened to your local knife problem had you not put in place your selected responses.
You can’t directly observe the counterfactual. It can only be approximated through comparing the observed knife crime patterns with, for example
an estimate based on past trends,
what is going on in the wider area beyond the reach of the intervention
through selecting people or places that are similar to the people or places where you have implemented your response (such as another similar urban area in your force).
There are many approaches for estimating the impact of an intervention. [Methods for measuring the effects of an intervention] summarises the different options for assessing effects, indicating what is involved, what can be learned and what to consider in deciding on their use. None of these methods is perfect and some are only possible in special circumstances. Advice on which methods are most suitable for your own problem-solving initiative can often be sought from research partners or the College of Policing.
Whichever approach you adopt, you are strongly advised to use statistical process charts that keep track of what is done and of trends in indicators of the knife crime problem you are addressing, with built-in estimates of statistical significance. The National Health Service has pioneered these and you can download a user-friendly version in Excel (NHS, 2021). The site also has a brief explanatory video that illustrates the application of the tool. The tool can easily be adapted for use in a knife crime problem-solving initiative. It will help you see whether you are achieving success or whether things are going awry, in which case you may need to consider modifications.
Before describing the practicalities of conducting assessments for future lesson learning, it is important to make clear that in describing the ‘effect’ element of EMMIE, we refer to negative effects as well as positive ones. As discussed above, all crime prevention measures have the potential to backfire, as was highlighted in the logic models outlined in the Response section of this guide. Sometimes well-intentioned and well-executed responses make things worse. Good assessments in problem solving are attentive to unintended consequences and make provision for their measurement.
Methods for measuring the effects of a knife crime intervention
| Research design | What’s involved? | What can be learned? | Considerations |
|---|---|---|---|
| Time series | Key expected outcomes are tracked over a sustained period of time to determine whether there has been a discernible change at the points predicted by the theory of change. | Whether the intervention was causally associated with the intended changes. | Depends on the consistency of data collection over the period covered. Reporting and recording practices are apt to change, which may invalidate long-term comparisons. |
| Shift share | Track change in the proportion of incidents in target area compared to those in a wider area. For example, if knife-enabled robberies had consistently made up 10% of all knife crimes in the previous five years, but this reduced to 5% following the response, this would indicate success. | Whether the distinctive changes within targeted groups accord with expectations. | Consistency of share trends needed before the intervention is put in place. Record keeping needs to be consistent over time both for the targeted incidents and wider population. |
| Randomised controlled trial | Interventions are randomly allocated to treatment and non-treatment groups (whether people or places). | Provides strong evidence that measured change was associated with the intervention and not some other unknown factor. | Best suited to single, simple measures where a population is well defined. Randomised controlled trials do not work so well where a number of interventions are implemented at the same time, as is common in problem-solving projects. |
| Comparison areas | Comparisons are made between intervention areas and areas with similar social, economic and/or demographic attributes but which do not receive the intervention. | Fairly strong evidence for estimating the effect of an intervention on intended outcomes in the target area. |
Generally used where the unit of intervention is a geographical area rather than individuals. Areas may not be similar enough and it can be difficult to compare areas where other activities are going on in the two areas. |
| Before and after | Simple before-and-after measurements of intended changes. |
Can sometimes provide plausible evidence of impact and is probably the most common research design when problem solving (see Box 14). Indicates whether continued problem-solving efforts are needed. |
It avoids the problem of finding and making comparisons with other areas or randomly selecting. The problem is that you cannot be sure that any positive outcome was the result of the initiative rather than some other factor. |
Before-and-after evaluation of knife crime prevention initiative
Operation Blade began in February 1993 following a 15-year increasing trend in violent crime in the Strathclyde region of Scotland.
The Operation included:
- a knife amnesty
- an intensified stop-and-search campaign
- safety measures, such as:
- CCTV at public entertainment venues
- metal detectors
- improved lighting
- training of stewards
- talks to knife retailers and to secondary-school pupils
- a change in licensing hours (earlier closing and prohibited re-entry)
- a high-profile media campaign
Before-and-after data from the Accident and Emergency Department at Glasgow Royal Infirmary were analysed to assess the effectiveness of Operation Blade. The main comparison was of cases in January 1993 and in January 1994. Total numbers of assault victims, of those with penetrating injuries, and of those directed to the chest and abdomen directed to the resuscitation room were compared.
The results found no statistically significant change. In January 1993, 282 victims of assault attended A&E, compared to 290 in January 1994. Of these victims, 60 presented with penetrating injuries in January 1993 compared to 53 in January 1994. In January 1993, 10 of 40 knife assault victims were directed to the resuscitation room, compared to 14 of 36 in January 1994.
The assessment also noted that following the start of Operation Blade, there was a short-term drop in serious stabbings, but this fully recovered within 10 months, when numbers of previous equivalent months were surpassed.
The authors cite police figures suggesting that there had been a 19% reduction in violent offences in 1993, compared to 1992, and a 33% reduction in violent crimes involving the use of a knife, but add that, ‘It is well-recognised in published reports that less than half of violent crime is in fact reported to the police’.
The authors take the view that the initial fall they found in knife crimes reflected ‘increased police presence in the city centre, particularly at pubs and clubs.’ They also conclude that: ‘In order to maintain decreased levels of violent crime, this type of operation would have to be repeated at regular intervals, as with the annual drink-drive campaigns, so that cultural attitudes may be changed in the longer term’. However, the authors provide no evidence to support this contention.
Bleetman and others (1997).
The practice of assessment: SARA meets EMMIE
In this final section of the guide, we bring together the different elements of the SARA model to provide a detailed step-by-step guide for how to carry out an EMMIE-compliant assessment when problem solving. These steps are further illustrated with two case studies presented below, which describe ideal assessments of knife arches and knife sweeps, respectively.
Building on scanning and analysis
1. The groundwork for assessment begins with scanning. You need to specify the particular knife crime problem you are trying to address and assemble quantitative data relating to it, such as the number of incidents, trend over time and patterns of concentration. These figures will provide you with the benchmark against which you will later assess the impact of the selected responses.
2. Problem analysis will also feed into assessment for lesson learning. As described previously, good problem analysis is focused and specific, and both identifies and measures the key causes and conditions that enable your selected knife problem to persist. In particular, problem analysis helps work out which of those causes and conditions you will focus on in your response. You need to decide on ways to measure whether the targeted causes and conditions are changing, in accordance with your logic model for the intervention.
3. What you find through scanning and analysis will help map out the interventions you plan to implement as part of your response strategy, which in turn (according to our logic model) will lead to the reduction or elimination of the specific problem you are focusing on. This will allow you to identify barriers to implementation that may be encountered during your assessment, so that these can be reported for anyone thinking about emulating what you have done (the ‘implementation’ part of EMMIE). In addition, you need to track implementation as the response is being delivered, to inform adjustments to your strategy where necessary.
Deciding the scope of your assessment
4. Following steps one to three, decisions can now be made about the scope of your assessment. Your decision needs to be based on answers to the following questions.
a. Within the scope of your problem-solving project, are the starting numbers of targeted knife crimes high enough and is the expected change in them large enough to make meaningful measurement of change a realistic prospect? If not, then including quantitative impact in your assessment is not a viable option.
b. Are there viable means for quantitatively estimating the counterfactual? For example, how many relevant knife crimes would there have been without the intervention, compared to how many there were with it (to find out the effect element of EMMIE)?
Key issues to cover in your assessment
5. In addition to determining effect, you will ideally want to know whether the response is working as expected (the mechanism and moderator elements of EMMIE). Using the logic model that you developed for your response, you can check on this. To do so, you need to check that your planned response is actually being put in place. This can be achieved by observing interventions directly or by checking administrative records, or by dip-sampling a sub-set of them. For example, are weapons sweeps happening when they are supposed to? Are police officers routinely stopping those who double back from installed knife arches? To better understand the implementation process, you can do the following.
a. Track intermediate steps along the expected causal chain – are the expected outputs observed ahead of the sought-after outcomes? For example, are knives being found in knife sweeps? If they are not, then it is clearly less likely that knife sweeps are responsible for any changes in knife crime.
b. Interview those delivering the intervention or targeted by the intervention to find out whether they are delivering and experiencing the intervention and the immediate response to it. For example, are officers and citizens delivering and experiencing stop and search as expected?
c. Examine data signatures, which are the patterns of events that you would expect to observe if your response is working as expected. Such data signatures could take many forms depending on your response and your theory of change. For example, if knife arches are only used on selected days and at selected venues, do the observed changes in knife offences correspond to the targeted days and venues, as compared to other days and other venues? The more precisely your theory of change specifies how intended effects should be brought about, the less scope there is for alternative causes to be at work in producing observed changes, such as other police activities or other local changes that might affect levels of knife crime.
6. In addition to testing your theory of intended change, you should also devise and test plausible theories of unintended change. Sadly, we know that some well-meaning crime prevention interventions inadvertently cause harm. There are some unintended harms that should routinely be checked. These include crime displacement by place, time, type of crime, offender and MO. It is never possible conclusively to rule all of these out. Instead, you need to decide which forms of displacement you consider to be most likely in the case of your response, then put measurements in place that can best capture them.
7. Most problem solving comprises a form of action research. This means that we start with the best strategy we can based on our analysis and formulate the most plausible theory of change we can. However, we also want to learn as we go and make adjustments to what we do when things are not working out quite as expected. You therefore need to build in feedback loops to use when fine tuning – or even making more radical changes to – your response plan if you find the intervention is going off-track. This is tricky. Many interventions take a long time to implement and, when implemented, there are often teething troubles before the final response becomes fully operative. The primary reason for problem solving is to deal effectively with problems, so making adjustments based on feedback makes sense, even if it makes impact assessment more difficult. What is crucial is that you log any feedback received and adjust and revisit your theory of change if necessary. For future users of your assessment, this will be useful. However, it may require adjustment to some of the measurements you make as part of your assessment.
Data collection for your assessment
8. At this point, you need to design your data collection instruments. What are you going to measure to determine the impact of your response plan, and how are you going to measure it? As detailed in the Scanning section, the main sources of data in the case of knife crime will be administrative records, within which you would expect to see change if the response were working as expected. Knife crime: A problem solving guide college.police.uk 106 To determine the resources devoted to your initiative, ideally you will need an account of all that went into delivering it. This includes costs that could have been used for different purposes and potentially produced different benefits.
a. The total costs of a knife crime response will cover such things as:
- police and other staff time
- transportation (for example, cars used to go to hotspots)
- hardware (for example, knife bins or knife arches)
- office space
- volunteers
Ideally, all need to be monetised (estimated in cash terms).
b. You also need to be able to estimate the net effects, both direct intended effects on knife crime and also side effects. These benefits then also have to be monetised, which can be done using standard Home Office estimates of the overall costs of crime (differentiating between offence types affected)(Heeks and others, 2018). Comparing costs and benefits allows one to say that for every pound spent on the response, a given monetary return was achieved.
c. Making robust estimates of the costs and benefits of a problem-solving initiative focused on knife crime will be technically very difficult. This is reflected in the very poor track record of economic evaluation in crime prevention more generally (Tompson and others, 2020). In practice, if you can catalogue the broad costs incurred and list these, and also estimate the net number of knife crimes prevented, you will be doing well. The importance lies as much in informing others who might want to emulate what you have done about the types of costs they should expect to have to incur, as it is to determine whether the initiative was worthwhile in economic terms.
d. To keep tabs on broad costs, you could maintain a simple ledger. For more complex and complete economic assessment, the Manning tool can be employed. This comprises a computer package that allows you to enter relevant figures and then crunch the answers relating to economic costs and values. For large-scale problem-solving initiatives relating to knife crime, it is worth using the Manning tool. An alternative tool for assessing cost-benefit is available on the Knowledge Hub.
Analysis for assessment
9. As data is being collected, analysis can begin, drawing multiple sources of information together. Different analyses test the theory of change that informed the intervention in a range of ways.
a. Analysis of data on what was delivered (drawing on interviews of those involved in the initiative) finds out whether what was planned was actually done. Where there are mismatches, they need to be described and explained. It is here that you will identify issues relating to implementation that can be reported when the project is written up.
b. Analysis of data collected on intermediate steps in your logic model will check whether the causal path was working as expected, and how it diverged (if at all).
c. Analysis of the interviews of those delivering or targeted by the intervention indicates whether they delivered or experienced the initiative as expected.
d. Analysis of the before-and-after data (including those related to comparison groups for estimation of the counterfactual) estimates the effect of the intervention as expected according to the theory of change (and also includes estimates of anticipated possible side effects).
e. Monetising observed patterns of change using Home Office costs of crime will allow you to compare observed net effects to the overall costs of the inputs to your problem-solving efforts to estimate the cost-benefit ratio.
Disseminating assessment findings
10. Problem-solving efforts need to be documented, shared and celebrated. It is important to be honest in your final assessment, to avoid misleading others about what was achieved. Failures can be particularly instructive. SARA provides a neat format for writing up problem-solving work, with general conclusions at the end and a methods appendix that describes the data you’ve used. In a final report, it is generally good practice to produce one-page and three-page summaries before a punchy report. This should rarely need to be more than 25 pages long, but may be succeeded by supplementary material if needed. When you have produced your draft final report, you should always ask for critical scrutiny from a competent independent outsider. Expect also to be asked to present interim and final results verbally as well as in writing – think about the simplest way to get an accurate message across. What are the two or three key messages that you want your audience to take home?
Conclusion
11. The perfect problem-solving assessment has yet to be conducted. What you are able to produce will always fall short of the ideal. We do the best we can do in the circumstances of the project and the resources we have available. If the project you are concerned with is large-scale and you think it may inform follow-up work that you and others may also do, then it makes sense to argue for the resources needed to do a thorough assessment, covering all bases, and to involve external evaluation experts (such as research partners or the College of Policing) to advise on or collaborate in the assessment.
If a major demonstration project is on the cards, a small-scale pilot with qualitative analysis focused on implementation, expected causal chains, and the experience of those delivering and targeted by the intervention may be prudent, to establish plausible parameters of a larger-scale initiative with provision for more elaborate assessment.
Assessment checklist
- Have you decided on the purpose of your assessment? Is it to work out whether the identified problem has been reduced or removed, or is it to determine whether your selected responses were responsible for any observed changes in your identified problem?
- Have you developed a theory of change (logic model) of how your responses are expected to reduce the selected problem?
- Following the EMMIE model, have you devised methods to measure the effects of your response?
- Following the EMMIE model, have you devised methods of capturing information about hurdles to implementing your response and what was done to overcome those hurdles?
- Following the EMMIE model, have you devised methods of capturing information about the costs and cost benefits of your selected responses?
- Have you worked out when and how you will provide feedback to those delivering the response?
- Have you worked out what form your final report will take in terms of sections, tables and figures?
- Using the evidence you have collected, are you able to explain the following?
- The problem.
- Why you selected that problem (from a range of other candidate problems).
- Why the selected responses were chosen and how they were expected to work in your local area against the selected knife crime problem.
- What was implemented in practice.
- The obstacles encountered in delivering your response.
- Whether and how these obstacles were overcome.
- The total cost of the response.
- The outcomes overall and by subgroup.
If you answered ‘yes’ to all of the questions above, then you are ready to write up your findings and share them with others.
Assessment plan for hypothetical knife arch operation
Weapons arches can be used in many ways. They are an integral part of airport security. They may be used at train or tube stations. They are sometimes seen at schools. In this problem-solving initiative, weapons arches are to be used in a violent-crime hotspot area in a town centre over a 12-month period. They will be deployed on busy nights of the week (Thursdays, Fridays and Saturdays) at the three main street entry points to the night-time entertainment area between 6pm and 10pm, when most people arrive for an evening out.
All those appearing to be aged 16-21 are to be invited to pass through the weapons arch, regardless of appearance or behaviour. Those deemed by police plainclothes officers to be deliberately avoiding the arch will be questioned and searched, if their behaviour warrants doing so. All officers will have body-worn cameras and all stops will be recorded whether or not they are accompanied by a search. As shown in the weapons arch logic model, the idea behind this initiative is to increase the perceived risk from weapon carrying by those who might otherwise carry them, who will be unable to select predictable times and places where the arches will not be operative.
Those asked to pass through the arches fall within the typical age range of those who have previously been found locally to carry knives. The unselective requests for individuals to pass through the arch if they look as if they fall within the target age range are intended to minimise the risks that there is either real or perceived discrimination against certain groups – for example, on the basis of ethnicity. The number of weapon arches deployments is set at 51 over the year.
The police force communications team arrange for news coverage for the start of the initiative (radio, print papers, posters, social media), reinforced through the year. This is partly to try to offset any public annoyance at any inconvenience caused by the weapons arches and partly to make sure that those who might carry knives know of the increased risk that they may face from doing so. Past experience suggests that the typical number of knives found in weapons arch deployment in the area is zero, and the maximum in any previous deployment was three. The total number of knife-related crimes in the city centre has been 80, 77 and 89 over the previous three years.
The scenario above describes a typical weapons arch initiative designed to reduce knife-related violence associated with the night-time economy. The table below shows the ideal stages for an EMMIE-informed assessment of this knife arch initiative. It may not always be possible or practicable to complete all stages.
| Stage | Evaluation activity | Include | Why include or exclude |
|---|---|---|---|
| a. | Track number of reported knife-related crimes (robberies, assaults and threats where a knife was used) in the target area (that is, the area intended to be covered by the arches) before, during and after the intervention. If hospital and/or ambulance data are available, use it in the ways described in the following steps as an alternative or addition to police data. Track also all reported crimes and incidents in the target area. | Yes or no | |
| b. |
Effect (intended outcomes): Randomise each of 51 Thursdays, Fridays and Saturdays to days when:
|
Yes or no | |
| c. | Effect (intended outcomes): Compare the numbers of reported knife-related crimes and of all crimes across the three conditions described in b). Compare also the numbers with, for example, the equivalent days in the previous three years. These measurements provide two indicators of the impact from the additional staff resources and from the staff (and ancillary) resources when the knife arches are used. Use this data to estimate the number of targeted (and other) crimes saved by using the weapons arches. | Yes or no | |
| d. | Effect (possible unintended outcomes): Identify the most likely displacement or diffusion of benefit areas without knife arches and track the number of reported knife-related crimes (robberies, assaults and threats where a knife was used) before and during the intervention, as well as the total number of crimes and incidents, to compare with the intervention area. | Yes or no | |
| e. | Effect (possible unintended outcomes): Track changes in footfall in the area in which the arches are used before, during and after the intervention, and compare that with comparable and wider areas without the knife arches. | Yes or no | |
| f. | Mechanisms and moderators: Monitor the number of people subjected to stop and search when refusing to go through or avoiding knife arch. Of these, monitor numbers on whom weapons were found, by type and sub-type of weapon. Observe recorded video footage of a random sample of those stopped and searched (eg, 20), to observe response to the intervention. Interview key personnel delivering the intervention. | Yes or no | |
| g. | Mechanisms: Conduct interviews and surveys with a sample of young people in the target area to gauge their perceptions of the intervention and their reactions to it, as well as a sample of community members, to gauge their knowledge and perceptions of the intervention. | Yes or no | |
| h. | Implementation, mechanisms and moderators: Observe the arches and behaviour surrounding them on 10 randomly selected occasions where arches are in use, to check how they are being operated and how citizens are responding to them and to invitations to pass through them. | Yes or no | |
| i. | Implementation, mechanisms and moderators: Track planned and unplanned publicity accorded to the intervention before, during and after the use of weapons arches, noting both positive and negative comments (including social media, in particular tweet and retweet patterns). | Yes or no | |
| j. | Implementation: Monitor the implementation of knife arches, as well as times and places when arches were used. Note hiccups in implementation and if so, how they were overcome – for example, kit failure, staff absences or business opposition to arches. Check whether randomisation, staffing, and so on accorded with original evaluation plans. | Yes or no | |
| k. | Economy: Track costs in terms of paid personnel (eg, civilians, uniform officers, spotters), transportation, and physical assets used (eg, arches, calibration, storage, maintenance). Use Home Office costs of crimes data to estimate monetised benefits, to compare to costs for economic analysis. | Yes or no | |
| l. | Mechanisms, moderators and effects: Reanalyse data in light of findings from f), g), h) and i), as possible, to check on emerging conjectures about possible mechanism and moderator configurations. | Yes or no | |
| m. | Mechanisms, moderators and effects: Use any ‘natural experiment’ thrown up by the initiative. For example, if a staffing crisis means that planned deployment of the arches is suspended for three months, check whether the number of knife crimes returns towards pre-intervention levels during this period, making any necessary seasonal adjustments. | Yes or no | |
| n. | Remember that with very low numbers of knife crime incidents in the before intervention and intervention periods, detecting effects specifically on kniferelated crimes with any confidence will be challenging. Your results may, at best, be suggestive. | Yes or no | |
| o. |
Write up a report of your assessment under the following headings:
Add an appendix with detailed evidence. Do not be selective here. Include any evidence that counts against knife arches, as well as any evidence that supports their use. |
Yes or no |
Assessment plan for hypothetical knife sweep operation
Knife sweeps are conducted in diverse ways. In this scenario, the problem-solving response includes intelligence-led knife sweeps. Sweeps are undertaken when and where intelligence suggests that knives are most likely to be stashed in public places for use as and when required. Locations may include, for example, sites where street-level drug dealing is common or where gang-related fights are expected.
The assessment plan described here focuses on 12 months, during which intelligence-led targeted sweeps are undertaken within knife crime hotspots. Given that sweeps are planned as responses to emerging intelligence, it is not possible in advance to predict how many deployments will occur, where they will take place and over what geographical area. They are intended to reduce the supply of weapons available for use by offenders who are reluctant to carry them for fear of being stopped and searched.
Sweeps are to be arranged quickly and will involve police staff, as well as volunteers and those from the local authority – in particular, environmental services, who may be in a position to remove or redesign convenient places where knives may be concealed (for example, removing bushes). No publicity is planned for the knife sweeps described here in order to avoid frightening residents by suggesting that they live in dangerous places and to avoid risks to intelligence sources, if they might be identified. Past experience suggests that the typical number of weapons recovered from targeted sweeps is one, with a maximum of three.
The scenario above describes a typical intelligence-led knife sweep initiative. The table below shows the ideal stages for an EMMIE-informed assessment of such an initiative. Again, it may not always be possible to complete all stages.
| Stage | Evaluation activity | Include | Why include or exclude |
|---|---|---|---|
| a. | Track the number of reported knife-related crimes (robberies, assaults and threats where a knife was used) before, during and after the intervention. If hospital and/or ambulance data is available, use it in the ways described in the following steps as an alternative or addition to the police data. Track also all reported crimes and incidents. | Yes or no | |
| b. | Randomly allocate areas with high levels of knife crime to use and non-use of intelligence-led knife sweeps over a 12-month period. | Yes or no | |
| c. | Effect (intended outcome): Compare change in the numbers of recorded knife crimes (knives used in robbery, violence against the person, homicide, rape, threat and sexual assault) in areas where intelligence-led knife sweeps take place and in similar areas with high levels of knife crime where intelligence-led knife sweeps do not take place. | Yes or no | |
| d. | Effect (unintended outcomes): To check on possible displacement or diffusion of benefits, compare changes in the numbers of non-knife crimes using cases of robbery, violence against the person, homicide, rape, threat and sexual assault to those changes found in c). | Yes or no | |
| e. | Effect (unintended outcomes): To estimate displacement or diffusion of benefits, compare the timing and location of subsequent crimes in the area receiving knife sweeps to the timing and spatial patterns of crimes in and around a comparison area not receiving knife sweeps. | Yes or no | |
| f. | Effect (intermediate outcome) and mechanism: Provide a count of weapons collected and by type and subtype of weapon (for example, firearms plus types, knives plus types, other weapons plus types). Add to count of weapons recovered by other means (such as stop and search). Compare year-on-year changes in intervention and non-intervention areas. | Yes or no | |
| g. | Implementation: Track implementation of sweeps. How many sweeps? Who was involved in sweeps? How long did each sweep last? What was recovered? Note any hiccups in carry out the sweep and if so, how they were overcome. | Yes or no | |
| h. | Implementation, mechanisms and context: Track planned and unplanned publicity accorded to the intervention before, during and after the sweeps, noting both positive and negative comments – in particular, social media. | Yes or no | |
| i. | Economy: Note costs in terms of paid personnel, volunteers, transportation and materials used (for costs, include special intelligence gathering, preparation for sweep and any continuing costs after the sweep, as well as the sweep itself). Use Home Office costs of crime data to estimate cost-benefit ratio. | Yes or no | |
| j. | Mechanisms and moderators: Check on other interventions and changes in the areas where the intelligence-led sweeps do and do not take place, to identify potential alternative sources to changes in numbers of knife crimes between them. | ||
| k. | Remember that with low numbers of incidents in the area before, during and after the intervention, detecting effects with any confidence will be challenging. Your results may, at best, be suggestive. | ||
| l. |
Write up a report under the following headings:
Add an appendix with detailed evidence. Do not be selective here. Include any evidence that counts against weapons sweeps, as well as any evidence that supports their use. |