The importance of using metrics responsibly is now widely recognised. Responsible metrics should be based on a number of principles that recognise the diverse qualities and impacts of research. Thing 5 will help you understand responsible use of metrics as a general set of principles and explore how to apply these principles to specific situations.
Getting Started
“The metric tide”, a 2015 independent review into the role of metrics in research assessment, proposed five principles for the responsible use of metrics:
Robustness: basing metrics on the best possible data in terms of accuracy and scope;
Humility: recognising that quantitative evaluation should support – but not supplant – qualitative, expert assessment;
Transparency: keeping data collection and analytical processes open and transparent, so that those being evaluated can test and verify the results;
Diversity: accounting for variation by field, and using a range of indicators to reflect and support a plurality of research and researcher career paths across the system;
Reflexivity: recognising and anticipating the systemic and potential effects of indicators, and updating them in response.
To learn about the background and development of these principles, read this introduction to “The metric tide“. Scroll to the bottom of the page to download the executive summary and recommendations. Alternatively, watch the “Metric tide video” featuring James Wilsdon.
Taking the responsible use of metrics one step further, DORA (San Francisco Declaration on Research Assessment) and the Leiden Manifesto are two significant declarations on the responsible use of metrics that individuals, institutions, and organisations are invited to sign or endorse.
Watch the video “The Leiden Manifesto for research metrics“, which explains the Leiden Manifesto’s ten principles to guide research evaluation.
Read the Declaration on Research Assessment (DORA). For examples of how research institutions have applied the DORA principles to the way they assess research, go to DORA good practices – research institutes.
Consider: Does your institution support either both or one of these declarations? What do you think are their relative strengths and limitations?
Learn More
Responsible use of metrics principles have been applied by some publishers. The open access, non-profit publisher PLOS (Public Library of Science), for example, is a signatory of DORA and pledges to follow the five DORA guidelines for publishers. While Elsevier officially announced its endorsement of the Leiden Manifesto in July 2020 (Elsevier’s press release).
Read through the 2017 PowerPoint presentation, “Doing metrics responsibly“, by Chris James, Elsevier’s Product Manager for Research Metrics, for a supplier’s perspective on the responsible use of metrics.
Exercise: Summarise the “Two golden rules” for using research metrics and the “Basket of metrics” developed by Elsevier (outlined in Chris James’ PowerPoint).
Consider: Do you think the “Two golden rules” and the “Basket of metrics” go far enough in addressing the DORA and/or Leiden principles? What else are Elsevier doing to support the responsible use of metrics?
Finally, for some further reading, go to “The blind and the elephant: bringing clarity to our conversations about responsible metrics“. In this post, Lizzie Gadd encourages us to clarify what, why, and how we are measuring before launching into any conversation about responsible metrics.
Challenge Me
How are responsible metrics applied in institutions?
In the 2017 post, “Responsible metrics: easier said than done?” Lizzie Gadd discusses how the responsible use of metrics is working out in practice.
Learn about the process of applying a statement of principles at an institution by reading “What difference does a responsible metrics statement make?“, which outlines the University of Bath’s Principles of Research Assessment and Management.
Exercise: Find examples of similar statements of principles from institutions in your country.
Read through the following websites that explore universities’ approaches to responsible use of metrics, then try to find out about research evaluation at your institution.
- The University of St Andrews’ “Responsible metrics” page.
- The University of Sheffield Library’s overview on “Using metrics responsibly“.
- The University of Melbourne’s “Newsroom” article on their signing of the San Francisco Declaration on Research Assessment (DORA).
Consider: What information is available on your institution’s approach to research assessment? Does it appear in policy or other strategic documents?
