Positive publication bias is actively harming science

In early 2024, UU alumni Stefan Gaillard (History and Philosophy of Science, 2018) and Martijn van der Meer (History and Philosophy of Science, 2018) made it to the Forbes 30 under 30 list, for their involvement in the founding of the Journal of Trial and Error (JOTE). This milestone seemed a perfect reason for an interview about the journal. Whether it was indeed the best motivation is debatable, as turned out during the conversation with Stefan.

‘I loved lists like this as a kid,' Stefan says enthusiastically over a fresh juice, 'so I think it's kind of cool to be on one myself now. You have to be nominated and then you are sent a number of questionnaires that you have to fill in. You really have to give your opinion on all sorts of things, including, for example, the European economy. I don't know much about that, so I just shared my ideas as a layman. They kept sending new lists, which I took as a good sign. Funny thing: we were very proud of the amount we had raised for the Journal, until we saw that the others on the list had raised several million.... They didn't even mention our - comparatively very modest - amount on the website.
By the way, it is important to mention that although Martijn and I got on that list, we were not the only founders of JOTE. The group was bigger: Max Bautista Perpinyà, Alex Visser and Nayra Hammann were part of the team too. Martijn, Max and I continued with the three of us after the start. Alex and Nayra remained involved on the sidelines'

You are on a list of successful people because you started a magazine about failure, that seems contradictory somehow

'The three of us have different views on what failure is and to what extent inclusion on such a list of influential people like Forbes 30 under 30 clashes with the magazine's purpose.
I see failure as a ‘stepping stone’ to later success. It is necessary and important to try things, and therefore do them wrong, to move forward. That is ultimately the goal.
Martijn is very pragmatic, he is not in favour of such a ranking, but sees the advantages of our entry. We can make the system work for us. We got several requests for interviews and articles after the list was published. Max sees it differently. For him, the very idea of a list of successful people goes against the principles we started with. Max sees a critical attitude towards the emphasis on success in academia, and certainly individual success, as part of JOTE's identity. 
Of course, research is never done by one person. There is always a team, that also applies to JOTE. We started with five people, some stepped aside, others joined. We now have a team of 15 people we can’t do without. Yet I do not feel the focus on success is wrong, including individual successes. We just shouldn’t ignore failure.'

At first absolutely nobody dared to submit their research.

How did JOTE come into being?

The Journal started in 2018 as a thought experiment, after we had all met at a lecture on Open Access and its potential to give the public insight into how research is done, how the scientific process works. Scientific articles often give the impression that everything works out in one go, when we all know that is not the case. What would it be like to launch a journal about failed research, we wondered. After a presentation we gave, half as a joke, at the Descartes Centre, it turned out that there was indeed a need for such a platform. That's not to say it was easy to get the first Journal filled up, because absolutely nobody dared to submit their research (laughing)’.

Oops. How did you deal with that?

We did have to work hard on that for a few years, actively approaching people who spoke out in favour of Open Science. And stressing time and again that academia needs to be reformed. We now receive enough for one issue a year. Most of it is submitted by scientists themselves. The issue is more and more alive. But of course, we still only get to see a fraction of all the failed research.

What is ‘failed research’ according to JOTE?

'Research with zero results or negative results. That in itself is not a failure you would think, but those studies are very often not published. While they do contain relevant information, think for example of drug trials. It is very relevant to know when a drug study finds no effect.
Another category is research that contains interesting methodological errors. An error in approach can be very instructive for researchers doing similar research. But mistakes or sloppiness in measurement, for example, are not relevant to us.’

How do you convince scientists who are reluctant to publish their failed research?

We especially address young researchers, who are often idealistic and feel the pressure to publish. They do not yet have much to lose in terms of name or reputation, but have much to gain from changes in science. Or people who already have permanent jobs and retired professionals, who have a secure position. You do see movement everywhere, with Recognise and Appreciate and Open Science. The cultural shift is taking place, very slowly but it is coming.

When did you fail successfully?

'Ah, I have to think about that for a moment. I usually get asked what the Journal has failed at. Well, I started studying chemistry and I’m not doing anything with that now. That felt like failure for a long time. I felt I was falling behind more and more with my studies, so I did all kinds of extra-curricular activities, such as a board position at the debate club. Chemistry turned out not to be for me after all, and I’m glad I found out then. Otherwise, I'd probably be doing chemistry research unhappily right now, and not very well either. The experience I gained during my studies now comes in handy when dealing with scientists, because I know what it's like. Although, to be honest, I have only been able to see things in such a positive light for a year or two.'

And the magazine, did that fail?

‘We are in full swing failing (laughing). We are working on a new project, we wish to publish ‘Serendipity anecdotes’ of scientists in addition to research. Stories about the moments when a failure suddenly gave a new insight that led to success. A historical example is the discovery of penicillin. There are, of course, many more stories like that. I think it's a very cool subject, but it's not running so smoothly yet. With much effort, we have now collected four projects. But I will keep trying for a while.'