Improving reproducibility: approaching the individual researcher

Science is in the midst of a reproducibility crisis – that’s old news anno 2015. However, less is known amgonst the research community about what the individual scientist can contribute towards making scientific findings more robust.

Yet, it is at the level of the individual researcher where changes of research routines can transform a flawed current system into a better one, with both improved reproducibility and integrity of research. In other words, through the adaptation of his day-to-day research pratices, each researcher has the chance to make important contributions.

So what exactly can the individual researcher do?

Experimental design

  • Publish your experimental design and data analysis strategy online before conducting the study. Discuss and improve it openly with colleagues and collaborators. Depending on the experimental characteristics, the study plan can be placed on a repository (see below) or into an academic journal.
  • Register your study before commencing it. This counts especially for all studies involving humans. Go to clinicaltrials.gov
  • Consider a multi-center approach also for experimental studies. Even for animal studies.
  • Plan an advanced statistical analysis. Don’t focus on a p-value only. Describe confidence intervals and effect sizes. More statistial tips from Cumming (free access) and on my statistics resources collection. Consult a statistician early!
  • Include a sustainable science statement in project proposals and grant requests. Signal to others what you do to improve the integrity of your research. Set your own standards. Guidance from the COS badges project.

Doing the experiment

  • Conduct the experiment triple-blinded. This also counts for laboratory research.
  • Use an electronic laboratory notebook. Fully searchable, never loose anything, improves collaboration. Recommendations: as simple version Labguru, the advanced people go with Open Science Framework / GitHub. Consider having your labnotebook entirely public or even collaborative (open notebook science).

Analysis, interpretation and write-up/presentation

  • Publish your dataset.  Increasing the transparency of your work, yet often in conflict with privacy issues. Use a repository online (read below), or choose a journal, e.g. Scientific Data.
  • Publish your analysis. This can be done for all statistical programs. R is unquestionable the best solution for this so far, especially when you’re using Rmarkdown and knitr. shiny allows the user to interact with the data.
  • Publish negative findings. If you’d like it as a full manuscript, use e.g. PLOS or the Journal of Negative Results. Don’t want to write it up to manuscript level? Put it on Open Science Framework or figshare!
  • Validate your findings. Verify! For example, use a secondary technique that may strengthen confidence in the present results.  Have a collaborator replicate the study in his lab. Find an independent reviewer of your statistical analysis.
  • Follow a reporting guideline when writing up for publication. They are available for RCT’s, observational studies, systematic reviews/meta analyses, case reports, animal studies, qualitative research, economic evaluations at the Equator Network.
  • Add a reproducibility declaration to your manuscript. In the methodology section or at the end of the paper. State where the data, code, and analysis can be found. If you want, choose a Zenodo DOI with an embargo before publication.
  • Publish your article pre-print online. Discuss it with others in the field (open pre-publication peer review). Have them repeat analyses, let them interact with your data. The last moment you want to do it is upon submission to a journal.
  • Submit your article to a journal with open peer review. This usually means that the whole reviews are being published online together with the final article to promote transparency.
  • Make presentations available online. Posters and seminars are readily available and publishing them is a low-effort way to get more input from other researchers. For example at figshare, but choose whatever you like.
  • Publish your article post-print for open access. Let the world access your research for free. Many publishers allow you to put your manuscript on your own website as long as you don’t use their print PDF. You can see an example here. Whether your publisher allows this can be checked on RoMEO.

Change your microsystem

  • Promote departmental research integrity policies. Expect every graduate thesis to have minimal standards in terms of reproducibility and integrity. Require senior researchers to peer-review in light of the same standards.
  • Improve scientific training at your institutions. Include sessions on open science, data sharing, etc. Expand the statistical literacy of your trainees.

Online scientific data and project repositories

Where to store datasets, files and documents?

  • Open Science Framework. Offers collaborative projects, wikis. Projects can be public or private.
  • Figshare. Put on everything on the web, have it timestamped & DOI citable.
  • Zenodo. Gives you a DOI for almost everything on the web, stored at their servers or elsewhere. Also offers an ’embargo’ version, only making contents available after their publication.
  • GitHub. Steep learning curve, yet best tool out there. Collaborate, make a wiki, share files, excellent version control.
  • arXiv and bioRXiv. These are pre-print archives of research.
  • PeerJ. Both pre-print publications (free) and life-long open access publishing.

Best practices / examples

  • Departmental change: introduce a department open science committee. Example of Felix Schoenbrodt at Psychology / LMU Munich. Develop standards on student evaluation, career & tenure decisions.
  • Commit yourself towards teaching your grad students good research practices. The Reproducibility PI Manifesto of Lorena Barba with reference to the science code manifesto.
  • A great example on full online publication of data, analysis, and paper: How much of the world is wood? by FitzJohn and colleagues.
  • “Is my brilliant idea any good? I am not sure, so I’ve pre-printed it on PeerJ”. Read on Keil’s website.

Some people change the world. Follow them

  • rOpenSci. “We are changing how science works through open data”.
  • Center for Open Science. “We foster the openness, integrity, and reproducibility of scientific research”. Few know that they offer free statistical consulting and online workshops.

Summing up

All the solutions stated above were chosen to be displayed because (1) they are concrete actions, (2) can be readily introduced into the daily work of a scientist and (3) should become the new methodological standard.

I have not included many other tools that are coming up (e.g. scientific markdown, open version control) because they are to techy at this moment and not yet accessible and usable for a broad researcher community.

Where publication of a project is often regarded as the project’s end, it should rather be seen as its beginning. Science needs to be discussed and critically evaluated. Nobody likes to produce papers that are never being read in the end. It has been shown that articles which adopted a more robust methodological strategy are cited more often.

A note on paradigm-based research

Traditional research theory teaches that science evolves over a long term. Cumulative evidence eventually leads to paradigm shifts that change our models of thought (Kuhn). At the same time, experimental results can never prove a theory, but provide evidence for or against a theory (falsification, Popper). While replication of a study may increase our confidence in its results, this does not necessarily make the results a closer approximation to truth. Yet, we provide new datapoints that challenge previous interpretations (models) and lead us to better understand the uncertanties involved.

Bayesian theory embraces this function of certainty and uncertainty. It can be argued that null-hypothesis testing leads to a dichotomous scientific worldview (true / false). On the other hand, Bayesian statistics allow a view based on probabilities for or against a clearly defined model, relative to the current state of evidence.  This also requires the researcher to specifically define an alternative model (paradigm), i.e. to ask the right questions.

Commit yourself: sign a declaration

A model from several open science symposia has been to sign a declaration of intention. Such a declaration would entail that in the future, the signing parties will commit to open science practices in their work. Such statements are embracing the responsibility of individual researchers to work under high qualitative standards.

A team of researchers at LMU Munich around Felix Schoenbrodt have published an open statement Voluntary Commitment to Research Transparency and Open Science (blog | OSF) that can be used to signal to yourself and other researchers/co-authors: I’m determined. See for yourself, and make your commitment!

 

Changelog

August 31, 2015: Added note on paradigm-based research after discussion of the post with @markomanka.

September 28, 2015: Added ‘sign a declaration’ at suggestion of @nicebread303.