Science should be transparent

• Title: Science should be transparent
• Date created: 23 August 2023
• Registration type: OSF Preregistration
• Contributors: Angelos Krypotos
• Category: Interview
• Methodology: Qualitative analysis
• Description: This interview aims to explain how preregistration and Registered Reports could contribute to honest and transparent research by asking questions to an expert in the field.
• Peer-reviewed: Yes

Does preregistration look something like the above? What are the advantages and downsides of preregistering your work? And how does it differ from Registered Reports? We met with Angelos Krypotos, assistant professor in the department of Clinical Psychology to discuss these questions.

“Preregistration is, among others, a way to conduct transparent research. Preregistration consists of a collection of time-stamped documents that describe a study’s research questions, hypotheses, methodology, and statistical analyses. The documents are then typically uploaded on a website such as AsPredicted or the Open Science Framework. Preregistration tells you how exactly a study is going to be performed. However, that does not mean that the planned study is correct in the first place. For example, the statistical analyses might not match the hypotheses the researchers are trying to test.  

Don’t your peers think preregistration is more fuss because it takes up extra time?

“It is nothing more than a change in your workflow. You will have to write down your methods, statistical analyses after you finish your study. Only now you do it in the beginning, which is actually great because that is the most creative part of a study.

My peers, especially in my department, are embracing preregistration. And I have the impression that many people, especially younger researchers, really like the idea, because it makes sense. Also, people are getting more familiar with Open Science, the open sharing of your data, etc.”

My peers, especially in my department, are embracing preregistration.

Is preregistration more relevant in your field of Clinical Psychology than in other studies?

“It is relevant in all scientific fields where data are collected.”

But isn’t it easier to cheat in Psychology? Look at the scandals we have had in the past.

“No. Psychology got a bad reputation after some big scandals came out, especially in the Netherlands. But the biggest efforts into change and more transparency have been put by researchers within psychology, such as Eric-Jan Wagenmakers from the University of Amsterdam. It is a lot harder to cheat nowadays. But if you want to cheat, you just need a person with bad intentions.”

Should preregistration be mandatory?

“No. I would prefer it if it was, but no. I think being transparent in your research is mandatory and you can use preregistration as a tool to achieve that goal.

We should note that there are cases of researchers who do not follow their own preregistration. And of course, checking whether a preregistration has been followed is difficult with the current review system. To date, your preregistration is checked by reviewers who already have a heavy workload, so they sometimes just trust that the preregistration is followed, which of course may not always be the case. There are publishing companies like Elsevier that make millions, they should have people who check whether a preregistration has been followed to the letter. Not the reviewers, academics who are trying to make ends meet. The publishing companies should hire methodologists to check these kinds of things and leave the conceptual part of the paper to the reviewers.”

If you want to cheat, you just need a person with bad intentions.

“Now Registered Reports are a step forward from preregistration. It is in fact a new publishing format made popular due to the efforts of many individuals but most prominently Chris Chambers from Cardiff University. Let’s see how a Registered Report differs from a preregistration. Let’s say: in February 2022 a researcher announces that he is going to do research into gender and ice cream flavour preference. The researcher decides to preregister the study. But in the questionnaire people will only be only asked to report on three flavors: vanilla, chocolate, strawberry. So, the researcher said what he was planning to do, but maybe due to lack of experience or being in a hurry the study was methodologically flawed. Because of course there are more than just three flavours of ice cream. The feedback you will get from the reviewers during the typical peer review is that your study is methodologically flawed and as such cannot be published. So, although you have preregistered the study and you were transparent enough, that does not mean that your study design was sound enough to warrant publication.

Registered Reports protects researchers from such mistakes. In Registered Reports authors submit their introduction, method, and analysis plan for publication before collecting the data. Then, reviewers and editors evaluate the submission and, if theoretically and methodologically sound, the paper is in principle accepted for publication. Only after this acceptance the authors start the data collection and/or the statistical analyses. Importantly, if they follow the accepted plan the paper will be published - independent of the outcomes. But mind you; preregistration and Registered Reports are not prisons. You can always update your preregistration. Do whatever you want as long as you are honest and let the reviewers, editors, and your peers judge your study accordingly.”

You are hinting at the replication crisis?

“There are many reasons for the replication crisis. One of them, although not clear if that was the dominant one, was that researchers would manipulate the data until they would find significant results. This follows the idea that journal editors are looking for interesting findings to publish. If your results do not support your research hypothesis, they were traditionally regarded as less interesting. That was the whole thing with the replication crisis. Researchers operate in a system that tells them: publish, publish, publish. So, researchers were inclined to play with the data to find something significant and results would not replicate of course.

But now with Registered Reports, the situation is changing. A study that does not report ‘significant findings’ is no longer considered ‘a problem’ because the design of the study and the statistical analysis had been checked in advance as sound enough.

I am not an idealist, I am not saying that everything is going right, but in general I think things have improved compared to how it was around 2010.”