Science is going through a change period right now and one thing that is being discussed a lot is ‘preregistration’ (especially in Brent Robert’s great post on the new rules of research). While the word its self just sounds like extra work and bureaucracy, there has not been a lot of discussion or a firm standard for what that means and so I’m going to suggest some things to do and avoid and hopefully we discuss.
Most generally, in my opinion, preregistration is ‘formally’ logging, writing down, one’s hypotheses and expectations about some experiment. This normally also implies some words about the methods of the experiment, and maybe even some indication of the analysis plan on how to get the desired result.
And I am hoping that this is how the idea of registration stays, as simple as possible. There is little reason in my opinion that a wordfile or pdf with a timestamp from before the experiment was conducted with some notes could not work.
Of course someone could just ‘preregister’ every outcome and say they got it right, but this would still mean that at least they had thought of it first (and especially hard if they wrote multiple versions!).
Or if this is not formalized enough, I could see much value in utilizing the Internal Review Board application as a preregistration. It already contains some information about the study hypotheses and how the researchers plan to test them. At a minimum, it might do those who are interested in seeing this implemented stress that it could be, or at least it could be (easily) elaborated into one.
is that preregistration will become as time consuming, frustrating, and seemingly worthless as getting IRB approval in the first place. Just another thing slowing science and hurting everyone because of the transgressions of the few (not that I think it is bad, but).
Last week I saw a presentation about ‘preregistration’ that really sort of scared me. A journal was setting up a section where authors could submit an experiment idea, method, plan of analyses, and expected results to peer review before running them.
The reviewers would suggest changes to the proposed study, and based on this the journal would ‘conditionally accept’ the paper or not. After the study had been run and analyzed, the results would again go through peer review, in order to ensure that rigor was upheld.
Who would submit themselves to such torture?! 😀 two rounds of lengthy review just to run a study you potentially didn’t even want to do!
General remarks about preregistration (or any other change initiative)
It should be easy. It should be valuable. Ideally it even makes my job easier. At the current moment, it is difficult to see how going through two rounds of peer review is going to be attractive to researchers (even though this might allow them to get a publication with just one study). I think these things are some of the reasons why it is doing so poorly among psychologists as to what they think should be done (seen in Table 2, even though it is Table 1, more results can be seen here: osf.io/xwnrm).
Most generally, lets Use science to improve science. and talk more about what preregistration (should) mean! 😀