Understanding the relative valuations of research impact: Applying best-worst scaling experiments to survey the public & biomedical/health researchers

The aim of the research is to develop a comprehensive ranking of scaled research-impact levels across different groups or type of research-impact for two stakeholder groups: the general population and biomedical and health researchers.

A conceptual framework with different categories of biomedical and health related impact and different intensities of each impact category will be developed initially through desk research. The framework will be tested and refined through focus groups and key informant interviews. This will result in a survey instrument that will elicit rankings for the different impact statements that will be obtained using Best-Worst Scaling Choice Experiments, a robust technique based on Random Utility Theory.

We will develop sets of scaled research-impact levels for each group of research-impacts in which respondents will be asked to select the levels they feel reflect the most and least important impact in each group. These profiles will be generated using experimental design such as fractional factorial orthogonal designs.

Following a pilot of 100 participants from the general population and 100 biomedical and health researchers, the main survey will seek responses from 1000 members of the general population and 500 researchers.

Participants in the general population sample will be recruited via an Internet Panel and will be representative of the UK's demographic profile in terms of a range of dimensions including age, gender, region, level of education and household income. Biomedical and health researchers will be recruited from a pool of MRC grantholders, and we would work with the MRC to ensure that this approach complies with data protection requirements.

We will use discrete choice analysis to analyse the data and obtain full ranking of research impact levels for the two groups of respondents, allowing a comparison of their relative valuations of different types of impact.

Detail

Research Councils "encourage researchers to consider the potential contribution that their research can make to the economy and society from the outset, and the resources required to carry out appropriate and project specific knowledge exchange impact activities". As part of their funding applications, researchers must submit a 'Pathway to Impact' section which is peer reviewed by referees and panel members. Similarly the Funding Council will be assessing impact using a case study approach as part of the Research Excellence Framework.

Case studies will be reviewed by academic peers and non-academic experts providing a private, public and third sector perspective. However, in assessing the adequacy/quality of these impact claims reviewers cannot currently draw on comprehensive evidence of the views of beneficiaries (i.e. the general population) or the producers of research (i.e. biomedical and health researchers) to qualify or justify their recommendations.

It remains unclear how beneficiaries value research impact, how such values vary across different groups and more fundamentally if the UK population understands research impact.

In the absence of knowledge about - and methods for assessing - values of beneficiaries and funders of research, policy that directs researchers to estimate impact and research funders to value such impacts rests on weak empiric foundations.

This study aims to address this gap by refining and adapting a survey-based approach known as Best-Worst Scaling (BWS) to analyse the relative valuations of research impact as perceived by the general population and researchers.

The study will focus on biomedical and health research and will contribute to the MRC's objective to fund studies that "Deliver user/beneficiary views on the contribution of academic research and MRC input to this".

The study comprises five phases:

Desk and qualitative research in the identification and classification of impacts. A thorough desk based review will catalogue various types of impacts and models for classifying them. We will then refine and develop this classification in four focus groups with the general public and interviews with biomedical and health researchers. The output of this phase will be a conceptual framework with different types of biomedical and health related impact and different intensities of each impact category for use in the BWS survey.
Development of quantitative survey. Based on the first phase we will design and test a survey instrument to elicit relative ranks of different types of impact. The questionnaire will be tested through cognitive interviews and the full fieldwork procedures through a pilot with the public and researchers. The output of this phase will be the final survey instrument.
Fieldwork with researchers and the general public. We will use an online panel to survey 1000 members of the public and the list of existing MRC grantholders to survey 500 health and biomedical scientists. The output of this phase will be a dataset for analysis.
Analysis of researcher and public preferences. The analysis will involve two steps. First we will conduct a review and exploratory statistical analysis to identify associations and trends that require consideration in the modelling work. This would include sample composition across key socioeconomic and demographic variables and checking respondents' understanding of the survey (whether they were able to respond to the questions). Second we will develop a BWS model to (i) derive a full ranking of the impact statements by the general public and health and biomedical scientists, and (ii) explore the extent to which preferences are consistent across the two stakeholder groups. The output of this phase will be the study's main findings.
Reporting and dissemination. Findings will be widely disseminated to policy and research communities using a number of channels, including peer reviewed publications, policy briefs and videos.

Impact

The project aims to understand the relative value that two groups of stakeholders place on different types of research impact (the public and producers of research i.e., biomedical and health researchers).

The study aims to test the appropriateness of Best-Worst Scaling to elicit such a valuation. It will help inform the Research Councils and Funding Councils in the UK on their future policies on research impact, and also will be of interest to other research funders in the UK.

The assessment of research impact is being widely discussed in science policy circles internationally, including, for example, Australia, the US, Canada and the EU. The findings from this study are likely to contribute to those discussions, and may ultimately inform the way in which peer review panels consider different types of anticipated impact described in funding applications. Thus the key non-academic audience for this work is likely to be policymakers within research funding organisations in the UK and globally.

We intend to access these audiences primarily through our existing contacts. RAND Europe has strong existing relationships with key stakeholders in the UK and more widely, and counts many as current and former clients.

In the UK, this includes the Medical Research Council, National Institute for Health Research, HEFCE, the Wellcome Trust, and many other charitable funders including Breakthrough Breast Cancer, the Stroke Association and Arthritis Research UK, plus a number of universities.

We have key individual contacts at these organisations and intend to distribute the findings of the research to over 200 people across more than 150 organisations.

We anticipate that the potential impacts of this research will be substantial if the approach is proven feasible. The research will contribute to a better understanding of impact and how it is valued by different stakeholder groups (and in time possibly by different research communities).

Ultimately the research could assist research funders in supporting research that is better aligned with the priorities of different stakeholder groups, as well as in better assessing (retrospectively) the value of impacts produced. This latter application would be particularly useful in, for example, reviewing the impact statements that will form a significant part of the Research Excellence Framework (REF).