Supporting emerging practices through cultural and technological innovation

This study was underpinned by the work of the KE Task and Finish Group on publishing reproducible research output, led by Juliane Kant and Anna Mette Morthorst as KE Group leads and Birgit Schmidt and Jeroen Sondervan as expert leads.

Executive summary

Introduction

Background and objectives
Knowledge Exchange (KE) (knowledge-exchange.info) has commissioned the present report to explore current practices and barriers in the area of research reproducibility, with a focus on the publication and dissemination stage. Our findings seek to compare and inspire strategies, policies and operational practice and share lessons learned from a wide range of stakeholder groups.

Definitions
In this work, we define research reproducibility as cases where data and procedures shared by the authors of a study are used to obtain the same results as in their original work. We acknowledge that other concepts such as replication, robustness and the generalisation of research findings are relevant to the discussion, but we considered these to be beyond the formal scope of our research.

Methodology
In this report, we captured the views of research funding organisations, research performing organisations, learned societies, researchers, academic publishers and infrastructure and service providers from around the world. We applied the Knowledge Exchange Open Scholarship Framework (KE OS Framework) (knowledge-exchange.info/event/os-framework) – a model to address specific aspects within open science – to investigate reproducible publication practices: this informed the design and delivery of all components of our research, including a comprehensive literature review and a series of interviews and focus groups with a total of 51 contributors. Interview and focus group findings were transcribed and qualitatively coded for thematic analysis.

Framing the research reproducibility discourse

Reproducible practices can take advantage of today’s rapidly growing infrastructures
The growth of digital technologies has led to significant transformation across the research landscape, including new tools and services, novel research approaches and the proliferation of interconnected technical infrastructures. In this context, a variety of options to document, share and analyse data and findings have become more widely available than ever before, fully opening the doors to reproducible workflows and publication practices. The key benefits of reproducible research include increased confidence in findings and results and an ability to continue one’s (or someone else’s) work in the future. At the system level, reproducible research practices can lead to higher transparency, openness and trust in science.

Some barriers can hinder reproducible practices
Some barriers may hinder the publication of reproducible research outputs, including current incentive structures in academia, differences in the technical capabilities of researchers, limited connectivity between technical solutions, and inconsistent reporting standards. Research methods, which tend to vary based on academic disciplines, also affect the effort to make one’s work reproducible. For example, research methods typically associated with quantitative disciplines are relatively straightforward to set up in a reproducible way. On the other hand, reproducible workflows become more complex to implement when a significant qualitative element is present.

Stakeholders, roles and responsibilities

Micro level: Researchers and research groups have direct control over everyday practices
Individual researchers and research groups have an important role to play, because they are responsible for designing, delivering and disseminating research and are the only ones with easy access to all the research objects involved. They can support reproducible publication practices by fostering and applying reproducible workflows and by considering any reproducibility requirements when they act as peer reviewers.

Meso level: Disciplines should communicate their requirements, and publishers should implement them
Research communities and disciplines can influence future policy development by defining and communicating their approaches to reproducibility to other stakeholders such as publishers, who do not wish to impose top-down requirements that may not mirror disciplinary practices. Once requirements are clear, publishers can help by mandating the use of appropriate checklists or guidelines for editors, peer reviewers and authors to encourage reproducible practices.

Meso level: Research performing organisations do not tend to mandate reproducible publication practices
Many research performing organisations do not have dedicated policies focusing on research reproducibility. However, they tend to make significant investments in cases where funder or policy mandates are introduced: since reproducible research practices are currently not a firm requirement, it is likely that research performing organisations will address research reproducibility via ad-hoc approaches according to their individual strategies and researcher bases.

Macro level: Research funding organisations see reproducibility as part of a broader discussion
Few research funding organisations are prominent in the research reproducibility landscape. Reproducibility is often discussed under broader requirements such as those around research data, open science or research integrity. Similarly to publishers, research funding organisations feel that it is difficult to set reproducibility requirements for grantees across a range of disciplines, and there is a limited sense of urgency to develop new policies.

Incentivising and enabling reproducible publication practices

Current incentives and support for reproducible publication practices are limited
Reproducibility efforts are not currently incentivised within the research process, and reproducible publication practices are commonly perceived as additional, unrewarded activities. Systematic efforts to reconsider current academic incentive structures are needed to more consistently reward behaviours that are conducive to reproducible publication practices. The support of research performing organisations can be instrumental in relieving some of the time pressures on individual researchers and complement their skills where lacking. This type of support can take the shape of new institutional roles such as data stewards, data curators or subject librarians.

New training and support pathways are developing across the world
A range of support and training pathways, both within and beyond research performing organisations, are developing worldwide. However, more structured support for reproducible publication practices would be welcome, as these initiatives are the exception rather than the rule. The role of champions was noted as an important awareness-raising mechanism, and interviewees highlighted that there is scope to improve the provision of reproducibility training in student curricula.

Technological innovation

Many digital infrastructures for reproducible publication practices are already available
A wide range of digital tools and infrastructures are available in today’s research landscape, and researchers are generally aware of how these can be leveraged to implement reproducible publication practices. However, not all researchers are equally trained to use these tools (often in line with disciplinary customs and typical research approaches), and the lack of interoperability between infrastructures is seen as a practical obstacle.

FAIR data principles can support reproducible publication practices
The issues described around service connectivity are partly enshrined in the ‘I’ of the FAIR data principles – Findability, Accessibility, Interoperability and Reusability. More broadly, FAIR data principles enable reproducible publication, as many of the practices underpinning data curation, sharing and reuse also support reproducibility.

Covering the costs of reproducible publication practices

The cost of reproducibility checks varies based on timing and responsibilities
Funding the time and other related costs of reproducibility efforts is a key consideration for the future, and many stakeholders have a role to play in ensuring this is possible. Our research highlighted that different types of costs may need to be covered: the time and efforts of researchers in the context of a research project; the time and effort of research support staff based at research performing organisations; reproducibility checks in the context of the publication process; and post-hoc reproducibility checks. The first of these is seen as a necessary condition to move research reproducibility up the agenda, while the others currently are the subject of further discussion and experimentation.

Three main pathways are available to fund the cost of reproducibility checks
In the context of the publication process, we have identified three pathways to implement reproducibility checks. We found that there is scope for publishers to establish in-house roles, such as data or reproducibility editors, and that third party providers could play a role in testing articles for reproducibility. In addition, peer reviewers may take on additional responsibilities by testing articles, data, and code for reproducibility when these are being considered for publication; however, we note that the research community may be reluctant to consider this approach, as it requires time and expertise that not all researchers might have.

Funding for digital infrastructures can be beneficial to pilot new solutions
Our research found that two areas of digital infrastructures may benefit from increased or new funding to better support reproducible publication practices. First, since an increase in open research practices and sharing will require improved features and capabilities, funding may help in extending the role of existing infrastructures. Second, public funding may be considered as a means to develop and pilot early-stage digital infrastructures providing reproducibility-related functionality across the research process, with a view to develop sustainable business models in the medium-to-long term.

Monitoring compliance is complex in practice
Several complexities emerge when it comes to monitoring compliance with reproducibility requirements. Particularly, it is difficult to reach an agreement around where the responsibility for conducting reproducibility checks should lie. This is partly because such an activity includes the review of connected research objects, which, in turn, might require an understanding of (sub)disciplinary standards, methodologies, or subject matter that not all stakeholders are well equipped to monitor and/or enforce.

Conclusions

Reproducibility is part of the vision for open science
Reproducible publication practices are evolving as part of a broad process of cultural change in the research landscape. As a result, progress is slow and sustained over a considerable period of time. On the other hand, technological innovation is moving fast: the imbalance between technical and cultural innovation paints a complex but optimistic picture for the future of reproducible research. The vast majority of researchers hold themselves to high standards: we expect that they will readily adopt reproducible publication practices, as long as a balance is found between increasing expectations and practical rewards.

Diversity will be key in driving positive change
It is essential that the focus on reproducibility does not lead to a “shame and blame culture”, but instead is welcomed as an opportunity to improve research practices. There is a risk for policies and their enforcement to leave little room for nuance. For some epistemic cultures, reproducibility will be harder to understand and implement, or perhaps is not even the goal; in others, reproducibility may not be seen as the key quality hallmark, but just as an option among many. It will therefore be necessary to prioritise diversity as we rethink research practices to preserve and boost trust in science.

© Knowledge Exchange 2021