The implementation of open science requires all research processes and activities to be taken into account in evaluation. To this end, the document lists the types of documents and productions developed within the entire research cycle and their modes of open dissemination as well as the types of activities eligible for evaluation.

Types of documents, productions and activities promoted by open science and eligible for evaluation

Committee for Open Science, Version 1, November 2019

Many stakeholders agree that the implementation of open science requires all research processes and activities to be taken into account in evaluation. This concerns both publications and research projects, individuals, collectives or institutions, and their interactions with society. The National Open Science Plan emphasizes in particular the importance of taking into account a qualitative rather than a quantitative approach and cites the Leiden manifesto (Hicks & al., 2015) and the San Francisco Declaration on Research Evaluation (DORA, 2012). Even if similar approaches existed before the open science movement, it aims to make them visible, legitimate and sustainable.

These recommendations of the Open Science Committee are based on those of the Open Science Platform Policy (OSPP), which unites the efforts and strategies of the European Union around 8 priority pillars, two of which are specific to evaluation issues: encouraging open practices and their recognition; and the deployment of a new generation of research indicators. In these recommendations, intended for all institutions piloting evaluation systems, we have identified two essential dimensions that are at the heart of open science.

1/ The Open Science Committee encourages the promotion of good practices in open science, in particular those that push for the availability of all types of documents and more broadly of research products, including but not limited to data and publications. Indeed, this availability will facilitate, on the one hand, the full recognition of the diversity of individual and collective contributions to research and, on the other hand, will encourage people to discover the content of these productions and to evaluate their quality.

2/ The Open Science Committee supports the recognition of the full range of research activities, in the plurality of disciplinary fields, as opposed to mechanisms that would focus only on scientific publications, or even on a limited part of them or on resulting indicators (McKiernan & al., 2019). Similarly, the recognition of new configurations of multidisciplinary forms of knowledge, and in particular of citizen science, must be strengthened.

To do this, the following are distinguished:

  • 17 types of products and documents; Table 1: “Types of documents and productions developed within the entire research cycle and their modes of open dissemination”,
  • 12 types of activities; Table 2: “Types of activities valued by open science and eligible for evaluation”. The activities in Table 2 may result in documentary production, and in this case it falls under the recommendations in Table 1.

In compliance with the legal and regulatory principles governing the opening and dissemination of research results and data, we recommend the acknowledged principle of “as open as possible, as closed as necessary” be followed.

The documents and productions referred to in Table 1 may be considered fully open only if they are made public under the conditions of a so-called free or open licence. Indeed, an object distributed without a licence can certainly be consulted by third parties, which constitutes a minimum form of opening, but its methods of reuse are not known. The final choice of licence is made by those who make their production public, in accordance with Decree No. 2017-638 (Prime Minister, 2017).

Beyond a licence, the systems for publishing, hosting and archiving documents and productions are intended to comply with the exemplary criteria defined by the Open Science Committee (2019).

The Open Science Committee recommends that all bodies and institutions involved in evaluation activities discuss, disseminate and adapt these good practices to their context. This includes, in particular, research funding agencies, publication and evaluation bodies. Communities and institutions remain sovereign in their adaptation of these recommendations, particularly in the aggregation, qualification and prioritization of types of productions and documents. The Open Science Committee expects transparency from these operations, including the justification of the reasons why certain productions or activities would not be taken into account in their evaluations.

Contact Information

This document is intended to be enriched and updated. To comment on it, please write to coso@recherche.gouv.fr

 

References cited and documents used

Open Science Committe (2019). Examplarity criteria for funding from the National Open Science Fund through platforms, infrastructures and editorial content. Available at: https://www.ouvrirlascience.fr/examplarity-criteria-for-funding-from-the-national-open-science-fund/

European Commission (2017). Evaluation of Research Careers fully acknowledging Open Science Practices Rewards, incentives and/or recognition for researchers practicing Open Science, 2017. Available at: https://ec.europa.eu/research/openscience/pdf/ospp_rewards_wg03112017.pdf

DORA (2012). San Francisco Declaration on Research Assessment. Available at: https://sfdora.org/read/

HCERES, Guides des produits de la recherche, 2019. Available at: https://www.hceres.fr/fr/guides-des-produits-de-la-recherche-et-activites-de-recherche-0

Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015). The Leiden Manifesto for research metrics. Nature, 520, pp. 429-431. Available at: http://www.leidenmanifesto.org/

McKiernan EC, Schimanski LA, Muñoz Nieves C, Matthias L, Niles MT, Alperin JP. (2019). Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. PeerJ Preprints DOI : 10.7287/peerj.preprints.27638v2

Prime Minster (2017). Décret n° 2017-638 du 27 avril 2017 relatif aux licences de réutilisation à titre gratuit des informations publiques et aux modalités de leur homologation, JORF n°0100 du 28 avril 2017. Available at: https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000034502557&categorieLien=id

Indicative bibliography to go further

European Commission. (2016). Evaluation of Research Careers fully acknowledging Open Science practices. Available at: https://ec.europa.eu/research/openscience/pdf/os_rewards_wgreport_final.pdf

Krzton, A. (2019). Support Scholars Who Share: Combating the Mismatch between Openness Policies and Professional Rewards. In ACRL 2019 (pp. 578–586). ACRL. Available at: http://aurora.auburn.edu/handle/11200/49374

Nichols, D. M., Twidale, M. B., (2017). Metrics for Openness. JASIST 68 (4), 1048–1060.
Available at: https://researchcommons.waikato.ac.nz/bitstream/handle/10289/10842/nichols-metrics-openness-jasist.pdf

Turckheim, E. de, Legouy, É. L., & Leclerc, L.-A. (2016). EREFIN Groupe de travail inter-établissements sur l’évaluation de la recherche finalisée. Rapport. Available at: https://prodinra.inra.fr/?locale=fr#!ConsultNotice:352302

Wynne, R. (2019). Using ORCID to Re-imagine Research Attribution. Available at: https://orcid.org/blog/2019/07/25/using-orcid-re-imagine-research-attribution

 

Table 1: Types of documents and productions developed within the entire research cycle and their modes of open dissemination

Type de documents et productions

Description

Examples

Solutions for a dissemination in Open Science

Examples of dissemination

Certified publications

Any publication subsequent to a peer review process

Scientific articles, review articles, book chapters, full text communications, books, data papers, reviews…

Open access outlet and upload to a compliant open archive, preferably HAL

https://hal.archives-ouvertes.fr/

Uncertified publications

Any publication that has not gone through a peer review process

Preprints, working papers, letters, editorial material, experience reports, excavation reports, research reports, collective expert reports, films, exhibition catalogues…

Upload to a compliant open archive, preferably HAL

https://hal.archives-ouvertes.fr/

Communications

Any type of communication in an academic meeting

Posters, video presentations, slides…. except the full text…

Upload to a compliant open archive

https://zenodo.org/

https://www.snphpu.org/posters/base-de- posters_33.html

Degree dissertation

Any text produced as part of a degree, from the Master’s level

Master’s dissertation, Magisterial dissertation, Habilitation thesis (Habilitation à Diriger des Recherches), PhD dissertation, State thesis (thèses d’État)

Distribution via libraries for legal deposit https:// star.theses.fr/; pushed into open compliant archives, TEL, DUMAS and HAL

http://www.theses.fr/

https://tel.archives-ouvertes.fr/

https://dumas.ccsd.cnrs.fr/

Data

Any type of data produced in the course of research, defined as factual records that are used as primary sources for scientific research and are generally recognized by the scientific community as necessary to validate research results.

Tabular data, images, sounds, videos, 3D data, sequencing, observation data, instrument data, relational databases, texts, annotations, etc.

“As open as possible, as closed as necessary”
principle, FAIR principles (Findable, Accessible, Interoperable, Reusable)

https://zenodo.org/

https://www.huma-num.fr/consortiums

Data Management Plans

Plans developed as part of research projects, including formats, data description and sharing rules.

Information and filing in a specialized platform.

https://dmp.opidor.fr/public_plans

Softwares

Any type of code et interface

Distinguish between

(1) services accessible online (e.g. web application),

(2) downloadable-reusable software (but whose code is not open),

(3) free and open-source software.

Qualify its reusability: documentation, user community tutorial, development community.

Upload to an appropriate open archive managing versions and harvested by Software Heritage

www.softwareheritage.org

Patents

All types of technical and/or industrial inventions

Manufacturing processes, methodology, innovative process

From the official date of publication (generally 18 months after filing), patents are visible in the databases of patent offices; they can then be harvested by a compliant open archive. It could also be deposited in the archives of the institutions that created it.

https://bases-brevets.inpi.fr/fr/accueil.html

https://www.openinventionnetwork.com/

Research projects

Any type of research project Summary, synthesis or full proposal of funded projects. The project results are processed in the other types.

Summary, synthesis or full proposal of funded projects. The project results are processed in other types

In the absence of currently defined institutional policies, the aim should be to deposit in a compliant open archive.

Research Protocols

Description of methods, research approach

Clinical trial design

Upload to a compliant open archive

https://www.protocols.io/

https://cos.io/rr/

https://clinicaltrials.gov/

Research hypotheses

Pre-registration and sharing of hypotheses before producing experiments and data

Research blog, registered reports

https://cos.io/prereg/

Hypotheses.org

Calls

Full text of a call

Papers, Conference panels, research projects…

Upload to a compliant open archive

http://sciencesconf.org

http://calenda.org

Reading and evaluation reports

Full text of all types of evaluations

Articles, books, communications, research projects

Deposited on the site of the document concerned or on an appropriate archive with a pointer to the document

https://peercommunityin.org/

Educational documents

All types of support for university level training (Master’s and Doctorate)

Synopsis of courses, thematic bibliographies, reference material, handouts, exercises, tutorials, manuals, presentations (slides, video,…), MOOCs, digital prototypes

Upload to a compliant open archive Integration into the institution’s open portal of educational resources

https://cel.archives-ouvertes.fr/

http://univ-numerique.fr/

https://www.canal-u.tv/

https://www.fun-mooc.fr/

Blogs and websites

All types of presentations and scientific communication for various audiences

Blog posts

Upload to an open archive in conformity or on a dedicated infrastructure

Hypotheses.org

https://hcommons.org/

https://www.hastac.org/

Platforms

A set of data, tools and structures made available to a public, through a web interface or API.

Platforms for visualization, mapping or photography, based on open, searchable and usable data in real time.

Some platforms are flagship participatory research mechanisms

Sustainable archive of the platform

https://www.tela-botanica.org/

https://www.tela-botanica.org/

https://www.citique.fr/

https://www.open-sciences-participatives.org/home/

 

Table 2: Types of activities valued by open science and eligible for evaluation

Activity type

Description

Examples

Academic animation: organization of events

Seminars, workshops, workshops, round tables…

https://www.sciencesconf.org

Academic animation: discussion

Discussing panels, workshops, moderating debates, participating in round tables…

https://www.sciencesconf.org

Expertise

Consulting (associations, companies, administrations…), Juries (prizes, medals…), support for public policies

https://www.anses.fr/fr/content/avis-et-rapports-de-lanses-sur-saisine

Scientific culture: communication

Interviews, conferences for the general public, popular articles, demonstration videos…

https://www.jsb.be/

https://les-savanturiers.cri-paris.org/

https://sms.hypotheses.org/

Social impact

Any type of valorization, whether social, economic, political… It can include the diffusion and accompaniment of any type of document and production of the
table

http://www.enseignementsup-recherche.gouv.fr/cid67054/www.enseignementsup-recherche.gouv.fr/cid67054/www.enseignementsup-recherche.gouv.fr/cid67054/les-satt-societes-d-acceleration-du-transfert-de-technologies.html

Training

Initial teaching, professional training, supervision of students and young researchers, organization of research schools, training in scientific
integrity

Partnerships

Search for, identification of partners, setting up partnerships

Building of a groupement d’intérêt public (GIP), groupement d’intérêt scientifique (GIS), groupement de recherche (GDR)…

Direction and management

Collective management and working group, project, training, scientific network, infrastructure, platform, database management, Scientific integrity officer

Head of a network (GIP, GIS, GDR…),

Infrastructure manager

Participation in journals and platforms

Editor-in-Chief, Editorial Board Member, Reviewer, Content Moderator, Community Animator

Collaborative innovation

Any approach involving several actors or partners in order to develop an innovation object or project. This includes crowdsourcing, crowdfunding, participatory
sciences…

Data creation, collection and/or curation

Creation of data sets for hypothesis testing, enhancement of the expertise of a research group, elaboration of thesis subjects, training of Master students.

http://www.dossiers-flaubert.fr/folios.php?corpus=g226&volume=1&num_folio=1&access=volume&viewf=par_cote

 

Read the report