Responsible research assessment
We are committed to assessing the quality of our research fairly and transparently.
The San Francisco Declaration on Research Assessment (DORA)
As a signatory of the San Francisco Declaration on Research Assessment (DORA), we support and promote the responsible use of metrics and quantitative indicators of research.
The Declaration recognises the need to improve the ways in which research is assessed. Since its development in 2012, it has become a worldwide initiative covering all scholarly disciplines and all key stakeholders including funders, publishers, professional societies, academic institutions, and researchers.
DORA commits us to assessing research in all its forms through a process of qualitative review rather than using proxy measures of quality, such as publication in journals with a high impact factor.
Since Cardiff University signed DORA in November 2019, we have made significant progress towards implementing its recommendations.
Cardiff University will not consider journal-based metrics, such as journal impact factors, in assessing an individual researcher’s contributions. As a result, we have reviewed and revised existing processes and documentation that relied (explicitly or implicitly) on journal impact factors, including:
- updated probation guidance for staff on one-year and three-year probation, specifically citing DORA
- a set of recommended responsible research assessment interview questions, distributed to HR business partners and HR leads in the schools in January 2020
- revised generic job descriptions for teaching and research staff in March 2020 to delete references to 'leading international academic journals' and 'high-impact conferences', which could be conflated with high-impact journals, and which place the emphasis on the outlet rather than the content of the research
- the incorporation of responsible research assessment into recruitment and promotions panel training. Colleagues on the University academic promotions committee have discussed their engagement with DORA, noting its positive impact on promotions assessment practices: ‘The University academic promotions committee respects and understands our commitment to DORA and is careful to ensure it does not draw upon journal impact factors to make any judgements about the quality of an applicant's outputs.’
Training, development and consultation
- We have organised drop-in sessions, focus groups and training induction sessions on responsible research assessment.
- These have provided valuable opportunities to hear from research and professional services staff about how to change and improve research assessment practices at the University.
- As a result of focus group participant feedback, we are developing processes for reporting and monitoring responsible research assessment in each of our Academic Schools.
- We are also developing an institutional pledge on responsible research assessment. Our aim is to make our commitments concrete in a formal statement.
Working with others
- We have been working with other institutions across the UK to promote best practice around responsible research assessment.
- We are looking to work with research models that focus on narrative CVs (sometimes known as a bio-sketch), rather than a list of publications. This is being piloted at other UK universities and in the wider sector, such as the Royal Society’s Resumé for Researchers.
- We have also collaborated with colleagues at the universities of Kent and Southampton on a panel discussion and blogs related to implementing responsible research assessment practices and principles.
As a University
- We will not use journal-based metrics (e.g. journal impact factors) as surrogate measures of the quality of individual research articles when assessing an individual researcher’s contributions, including in recruitment, promotion or funding decisions.
- We will be explicit about the criteria we use to reach recruitment, tenure, and promotion decisions, highlighting that the scientific content of a paper is more important than publication metrics or the reputation of the journal.
- We will consider the value and impact of all research outputs (including datasets and software) and be attentive to a broad range of impact measures, including qualitative indicators of research impact, such as influence on policy and practice.
As individual researchers and decision-makers
- We will make assessments based on scholarly content rather than publication metrics in decision-making (e.g. internal funding, recruitment, tenure, or promotion).
- We will, wherever appropriate, cite primary literature in which observations and research findings are first reported rather than reviews, thereby giving credit where credit is due.
- We will use a diverse range of article metrics and indicators on personal and supporting statements as evidence of the impact of individual published articles and other research outputs.
- We will challenge research assessment practices that rely on journal impact factors and promote best practice that focuses on the value and influence of a diversity of research outputs.
DORA Working Group
In April 2019, the University established a DORA Working Group (agreed by the University Executive Board and Senate in June 2018) to explore the integration of DORA principles and practices into our research environment and culture.
The Working Group is chaired by the Dean of Research Environment and Culture, Professor Claire Gorrara, who leads the implementation of a DORA action plan, acting on behalf of the Pro Vice-Chancellor Research, Innovation and Enterprise, Professor Kim Graham, who signed the declaration on behalf of the University.
The action plan, approved by University Senate in June 2019, is designed to enhance the University’s alignment with DORA. We have appointed a Responsible Research Assessment Officer, Karen Desborough, to support the delivery of the DORA action plan and facilitate the use of responsible research metrics at the University (funded via Wellcome Trust ISSF funding).
For queries or further information about DORA, please contact:
Dr Karen Desborough
Responsible Research Assessment Officer