Ref Impact Case Study Guidance

By Prof Mark Reed @profmarkreed

8 points for writing an effective impact case study

1. Create a coherent narrative that explains clearly the relationship between the underpinning research and the impact. It may be useful to briefly explain what was original or distinctive about the research that contributed to the impacts. Consider starting from the perspective of the beneficiaries – how did they benefit and why is that benefit so important to them? Be as specific as possible if you want the link between the research and the impact to be credible, including specific details about the names of researchers, their position and dates and locations of the research activity. Choose a strong headline for your case study and ensure your summary starts with a powerful opening sentence that summarises the case study and draws the reader in

2. Be as clear as possible about exactly what the impact was, adding some sort of precise quantification with numbers wherever possible. Quantitative data and indicators need to be meaningful and contextualised to clearly support the case being made, not used as a substitute for a clear narrative. Avoid generalised or exaggerated statements about your impact.

3. Clearly identify specifically who has benefited from the work or which groups/organisations have changed something as a result of the research. Bear in mind that this may include ‘intermediary’ organisations as well as your intended ‘end users’ or audiences.

4. Be concise. A concise case study that pulls out the key points easily for readers has far greater impact than one that is dense and rambling. The UK’s Research Excellence Framework gives people strict page limits, but if you don’t have this constraint, then you could consider imposing some sort of constraint on your case studies.

5. Keep your language simple and direct. If possible, get advice from a science writer or communications specialist, but try and avoid introducing inaccuracies. Remember your audience, and that your audience is probably not other academics. Readers should not have to have in-depth expert or prior knowledge to be able to understand your case study. It is essential that impact case studies avoid academic jargon, so that they are accessible to all your readers. 

6. If you are writing multiple case studies, identify key features of best practice and be consistent about all your case studies covering these aspects. You might for example want to consider a particular list of sub-headings for each case study to follow. However, avoid making all your case studies sound too similar, as though they are all written by the same person, if you want them to also feel authentic.

7. Related to this, make sure you provide detailed, specific and independent evidence to support every claim you make. All material required to make a judgement about the impact needs to be contained within the case study. Avoid anecdotal evidence or evidence that might perceived as such. If possible, link to published evidence that demonstrates the impact of your research. It is possible to commission work to demonstrate impacts yourself, but you will need to consider carefully how this will be published, to ensure that it is perceived to be sufficiently independent and credible. Rather than just listing sources of evidence, explain how each source of evidence supports a specific aspect of the impact that has been claimed.

8. Bring your case studies to life with quotes that illustrate the impact with greater resonance than could otherwise be done with formal language. If these quotes are from people with high profile and relevant job titles, then this adds significant credibility to your case study, as well as some lived experience. Finding quotes from people years after an impact has occurred can be tricky however, so it is recommended to collect these from people as the impact unfolds, and if nothing else, keep track of contact details so that people can be easily contacted later. Bear in mind that key people who could attest to the impact of your work may retire or pass away before you need to write your case study. Usually it is best to aim for a good balance of quantitative data and quotes to support impacts. However, in case studies where there is no quantitative data to corroborate an impact (e.g. a policy which has been developed but not yet implemented), quotes may be the primary source of corroboration to evidence your impact. 

What do we know about impact in REF2020?

We don’t yet know the rules and weighting that will be associated with the assessment of research impact in HEFCE’s Research Excellence Framework in 2020 (REF2020), but we do know four important things:

  • Impact was worth 20% of an institution’s score in 2014, and most commentators expect this to either remain the same or increase slightly to 25% for REF2020
  • Some impact case studies submitted to REF2014 that have continued to generate further impacts during the current REF period are likely to be eligible for re-submission to REF2020. HEFCE are currently consulting on this, to try and get the balance right between incentivising Higher Education Institutes to continue developing and building on existing impacts, whilst having sufficient incentives to develop new impacts
  • It is likely that the eligibility criteria and guidelines for submitting impact case studies in 2020 will be broadly similar to the 2014 assessment, although these are currently under consultation too
  • Assessment criteria may be expanded (e.g. including impacts on public discourse and attitudes) or become more fine-grained (e.g. considering different elements of reach such as number versus the types or geographical spread of people adopting a new technology or behavior) in response to feedback from REF2014 panels, but they are still likely to address aspects of significance and reach.

HEFCE commissioned an assessment of the potential for metrics to inform REF2020, which reported in early 2015. It concluded that metrics could not replace case study narratives for the assessment of impact in 2020. Research commissioned by HEFCE about the impact case studies submitted to REF2014 somewhat uncritically accepted the breadth and depth of impacts captured in the assessment, implying that a similar approach in 2020 could yield similarly authoritative “evidence” of impact. This approach to impact may be criticized as biased towards a narrow set of self-reinforcing perceptions of what constitutes valid and significant impacts, created primarily between different sectors of the research community, rather than in the eyes of beneficiaries. However, although research has been proposed to address these issues, if funded, it will not report until after procedures have been set in place for REF2020. It is therefore reasonable to assume that REF2020 impact case studies will be broadly similar to REF2014.

This is what Universities were required to submit in 2014, which is likely to be similar in 2020:

  • Each Unit of Assessment (UOA) had to submit an impact statement (REF3a, Figure 1)
  • Each UOA also had to submit between 2 and 7 impact case studies, depending on the number of academics submitted in the Research Outputs section of the assessment (REF3b, Figure 1)
  • Underpinning research had to be “excellent” (2* quality or above) and must have been conducted by University members of staff to be eligible.

Underpinning research could include published reports (e.g. final reports to funders) where it was clear that they were of 2* or above quality, but it is highly risky to submit a case study with underpinning research that has not undergone peer-review in an international journal, in case panel members do not deem it to be of sufficient quality for the impact to be eligible. If you are submitting grey literature as the underpinning research, provide copies or links to any assessments that may have been made by the funders as a form of peer-review, and if this isn’t possible then make sure you get it reviewed by a number of experts in the field to ensure the panel is likely to judge it as 2* or above in quality. Depending on how long there is before REF2020, you may be able to turn a report from the grey literature into a peer-reviewed journal article and submit both as underpinning evidence (the original report to show that the research preceded the impact and the peer-reviewed version to demonstrate 2* quality). The underpinning research has to be conducted (not just published) at the submitting institution, so members of staff appointed between now and the 2020 assessment will need to be able to conduct significant new research to further underpin and extend existing impacts if these are to be claimed by their new institution.

Research has to have made a distinct and material contribution to any impacts that are claimed, such that the impact would not have occurred or would have been significantly reduced without the research. If this link cannot be adequately established, then the case study may be graded as “unclassified”. Research from a number of different individuals, UOAs and institutions may have contributed towards a particular impact, and as a result, there are a number of REF2014 case studies that cover the same impact in different ways, based on the specific impacts related to the research conducted by each of the different submitting institutions. If you are in this position however, it is worth considering whether the impacts that you are able to claim linked to your work will be as far-reaching and significant as those claimed by a competing institution who is submitting to the same UOA. Panels assessing the same impact submitted by different institutions will inevitably compare case studies, and you want to make sure that if this happens, your grade doesn’t suffer as a result.

The Research Excellence Framework (REF) 2014

About REF2014

The rules for impact case studies

Impact case studies from your department (redacted versions):

Read your department’s REF impact statement (REF3a)

Read examples of impact case studies from other universities

About REF 2014

Each Unit of Assessment was required to submit an impact statement (REF3a) – a three or four page A4 document which outlined the unit’s approach to enabling impact.  Additionally, each Unit of Assessment was required to submit 2-7 impact case studies, depending on the number of academics submitted in the Research Outputs section.

In total, the University submitted 25 impact statements and 90 impact case studies. Our statements and case studies are available for internal viewing (see college links above, please do not share these outside the University).

Each impact case study tells the story of how university research made an impact on society and describes what that impact was. They included a summary of the impact, a narrative description of the underpinning research, a list of underpinning publications, a narrative description of the details of the impact  and a list containing evidence of the impact.

Impact case studies were assessed on the reach and significance of the impact and given a quality assessment U (little or no impact), 1* (recognised but modest impact), 2* (considerable impact), 3* (very considerable impact) or 4* (outstanding impact). Underpinning research had to be “excellent” (2* quality or above) and must have been conducted by University members of staff to be eligible. We will be informed of the results of the assessment exercise in December 2014.

The rules and guidelines for impact case studies for REF2014

We don't yet know how impact will be assessed in the next REF. It is very likely that impact will continue to be a measure – and that impact case studies will continue to be used to assess it. We will not decide which case studies to use in the next REF until a few months before submission, once the new guidance is issued. In the meantime, it may be useful to bear in mind the rules and guidelines which were used for REF2014.

For REF2014, an impact case study consisted of a maximum of four A4 pages of Arial 10pt split into five sections.

  1. Summary of the impact
  2. Summary of the underpinning research
  3. Research publications
  4. Details of the impact
  5. Sources to corroborate the impact

Eligibility rules

The impact claimed:

  • must be underpinned by research – the link between the research and the impact must be explained
  • must have occurred during the assessment period (for REF2014, these dates were January 1, 2008 to July 31 2013)
  • must have affected society and the economy

The research the impact is based on must:

  • be of 2* quality or above
  • have been conducted at the University of Leicester between specified dates (for REF2014 these dates were January 1 1993 to December 31 2013) – ownership of impact stays with the institution where the research was conducted and does not travel with individuals between institutions in the way that outputs do

Additionally,

  • Impacts on Higher Education research and academia did not count – the impact case study assessed impact on society and the economy
  • Impacts on students teaching or other activities within the submitting University were excluded.

1. A summary of the impact

  • Should be a summary of the impact – not the research.
  • An explanation of the real-world “problem” that the research has helped to solve, or the issue it has tackled, is a useful way to introduce the impact. What was the environment *before* the impact, what changed as a result of the research? Who benefited?
  • In this section, it is not necessary to describe the research methods, team, findings or funding.
  • Should introduce the subject area to an informed but non-specialist reader.

2. A summary of the underpinning research

This section should outline the key research insights or findings that underpinned the impact, and provide details of what research was undertaken, when, and by whom. References to specific research outputs that embody the research described in this section, and evidence of its quality, should be provided in the next section. Details of the following should be provided in this section:

  • The nature of the research insights or findings which relate to the impact claimed in the case study.
  • An outline of what the underpinning research produced by the submitted unit was (this may relate to one or more research outputs, projects or programmes).
  • Dates of when it was carried out.
  • Names of the key researchers and what positions they held at the institution at the time of the research (where researchers joined or left the HEI during this time, these dates must also be stated).
  • Any relevant key contextual information about this area of research.

3.  References to the research (indicative maximum of six references)

This section should provide references to key outputs from the research described in the previous section, and evidence about the quality of the research.

Include the following details for each cited output:

  • Author(s).
  • Title.
  • Year of publication.
  • Type of output and other relevant details required to identify the output (for example journal title and issue).
  • Details to enable the panel to gain access to the output, if required (for example, a DOI or URL), or stating that the output is listed in REF2 or can be supplied by the HEI on request.

All outputs cited in this section must be capable of being made available to panels. If they are not available in the public domain or listed in REF2, the HEI must be able to provide them if requested by the REF team.

Evidence of the quality of the research must also be provided in this section. Guidance on this will be provided in the panel criteria documents. Where panels request details of key research grants or end of grant reports, the following should be provided:

  • Who the grant was awarded to.
  • The grant title.
  • Sponsor.
  • Period of the grant (with dates).
  • Value of the grant.

4. Details of the impact (indicative maximum 750 words)

This section should provide a narrative, with supporting evidence, to explain:

  • How the research underpinned (made a distinct and material contribution to) the impact
  • The nature and extent of the impact.

The following should be provided:

  • A clear explanation of the process or means through which the research led to, underpinned or made a contribution to the impact (for example, how it was disseminated, how it came to influence users or beneficiaries, or how it came to be exploited, taken up or applied).
  • Where the submitted unit’s research was part of a wider body of research that contributed to the impact (for example, where there has been research collaboration with other institutions), the case study should specify the particular contribution of the submitted unit’s research and acknowledge other key research contributions.
  • Details of the beneficiaries – who or what community, constituency or organisation has benefitted, been affected or impacted on.
  • Details of the nature of the impact – how they have benefitted, been affected or impacted on.
  • Evidence or indicators of the extent of the impact described, as appropriate to the case being made.
  • Dates of when these impacts occurred.

5. Sources to corroborate the impact (indicative maximum of 10 references)

This section should list sources external to the University that could, if audited, provide corroboration of specific claims made in the case study. Sources provided in this section should not be a substitute for providing clear evidence of impact in section 4; the information in this section will be used for audit purposes only.

This section should list sufficient sources that could, if audited, corroborate key claims made about the impact of the unit’s research. These could include, as appropriate to the case study, the following external sources of corroboration (stating which claim each source provides corroboration for):

  • Reports, reviews, web links or other documented sources of information in the public domain.
  • Confidential reports or documents (if listed, these must be made available by the HEI if audited).
  • Individual users/beneficiaries who could be contacted by the REF team to corroborate claims*.
  • Factual statements already provided to the HEI by key users/beneficiaries, that corroborate specific claims made in the case study and that could be made available to the REF team by the HEI if audited*.

* Where the sources are individuals who could be contacted or have provided factual statements to the HEI, the submitted case study should state only the organisation (and, if appropriate, the position) of the individuals concerned, and which claim(s) they can corroborate. Their personal details (name, position, contact details) must be entered separately on the REF submission system and not on REF3b. Details of a maximum of five individuals may be entered for each case study; these data will not be published as part of the submission.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *