What does it mean to preregister a study?

Science is going through a change period right now and one thing that is being discussed a lot is ‘preregistration’ (especially in Brent Robert’s great post on the new rules of research). While the word its self just sounds like extra work and bureaucracy, there has not been a lot of discussion or a firm standard for what that means and so I’m going to suggest some things to do and avoid and hopefully we discuss.

Most generally, in my opinion, preregistration is ‘formally’ logging, writing down, one’s hypotheses and expectations about some experiment. This normally also implies some words about the methods of the experiment, and maybe even some indication of the analysis plan on how to get the desired result.

And I am hoping that this is how the idea of registration stays, as simple as possible. There is little reason in my opinion that a wordfile or pdf with a timestamp from before the experiment was conducted with some notes could not work.

Of course someone could just ‘preregister’ every outcome and say they got it right, but this would still mean that at least they had thought of it first (and especially hard if they wrote multiple versions!).

Or if this is not formalized enough, I could see much value in utilizing the Internal Review Board application as a preregistration. It already contains some information about the study hypotheses and how the researchers plan to test them. At a minimum, it might do those who are interested in seeing this implemented stress that it could be, or at least it could be (easily) elaborated into one.

My nightmare

is that preregistration will become as time consuming, frustrating, and seemingly worthless as getting IRB approval in the first place. Just another thing slowing science and hurting everyone because of the transgressions of the few (not that I think it is bad, but).

Last week I saw a presentation about ‘preregistration’ that really sort of scared me. A journal was setting up a section where authors could submit an experiment idea, method, plan of analyses, and expected results to peer review before running them.

The reviewers would suggest changes to the proposed study, and based on this the journal would ‘conditionally accept’ the paper or not. After the study had been run and analyzed, the results would again go through peer review, in order to ensure that rigor was upheld.

Who would submit themselves to such torture?! 😀 two rounds of lengthy review just to run a study you potentially didn’t even want to do!

General remarks about preregistration (or any other change initiative)

It should be easy. It should be valuable. Ideally it even makes my job easier. At the current moment, it is difficult to see how going through two rounds of peer review is going to be attractive to researchers (even though this might allow them to get a publication with just one study). I think these things are some of the reasons why it is doing so poorly among psychologists as to what they think should be done (seen in Table 2, even though it is Table 1, more results can be seen here: osf.io/xwnrm).

More results can be seen here: osf.io/xwnrm/

Most generally, lets Use science to improve science. and talk more about what preregistration (should) mean! 😀

Advertisements

4 comments

  1. “At the current moment, it is difficult to see how going through two rounds of peer review is going to be attractive to researchers (even though this might allow them to get a publication with just one study). I think these things are some of the reasons why it is doing so poorly among psychologists as to what they think should be done (seen in Table 2, even though it is Table 1, more results can be seen here: osf.io/xwnrm).”

    I wonder what the value of psychologists’ opinion is in these matters. Let’s say pre-registration, and two round of review are what is best for science in multiple ways. Who cares then if psychologists like it or not. It’s their job to do the best work they can i reason (that’s why society pays them). If they don’t like it, they can find another job.

    Shouldn’t the real issue/question be what is best for science and not what the opinion of psychologists is?

    1. Dear Anon,

      Good question! But I think you forget that it is actually humans that do the science. If scientists would just do the best job they could, whether they liked it or not (because that is what they get paid for), then we wouldn’t be in this situation in the first place! 😀 At least that is what I perceive.

      This being the case, I would say we have to take into mind what is best (and easiest!) for the individual, and make sure that it is also best for the group. The less they like it, the more they will look for ways to cheat (in my opinion), which also answers why it is important to ask what psychologists actually want and think. 🙂

      Also, I would suggest that any changes will be more easily implemented if the people want them. Look throughout history at those times when the 1% forced things on the populace, it didn’t end up working out very well, in general, i would say. 😀

      so ya, basically, when machines can do science, then we don’t need to think about what they like, but so long as they are humans, it is important 🙂

      Thank you for your comment! 😀
      Brett

  2. Thanks for your reply!

    But if you leave things to most psychologists nothing will ever happen, or it takes centuries. For instance, Cohen mentioned problems of low power 50 years ago, and nothing changed at all !? Nothing. Perhaps psychologists didn’t like his recommendations either. Who knows.

    Even at this moment in time, in your survey psychologists only agreed marginally (60.68) to consider sufficient power as a publication criterion. How this is not 100 is somewhat interesting to me.

    I am truly disappointed by the results of your survey. Given several recent, well talked about failures of replicating studies, even now psychologists apparently only marginally agreed (61.78) that there are many non-replicable results in the literature. How this is not 100 is somewhat interesting to me again.

    Now i can think of a few reasons why psychologists agree with/ don’t agree with things in the survey. I wonder if a large reason is some form of self-protection, be it protecting the field or protecting themselves from worrying that most of their work are just false-positives.

    I also wonder how bad do things have to get? Would showing that using questionable research practices (which, if i understood correctly, can partly be tackled by pre-registration) can make anything, even nonsense, become “statistically significant” make psychologists aware that they are not doing their job the best way they can? O wait, that has already been done: http://neuron4.psych.ubc.ca/~schaller/528Readings/SimmonsNelsonSimonsohn2011.pdf

    I am disappointed by the results of the survey, although not surprised. I also wonder if there is a difference between cognitive, and social psychologists in your survey results. I wouldn’t be surprised if there is.

    I still have a tiny bit of hope though for the field: pre-registration and power demands will hopefully make it very clear which research to trust and build up on and which research is severely flawed. If funders, and journals, will start demanding these things, i think things will become much better very soon.

    I wish you all the best in performing research the best way you know how to!

    1. Hey Anon,

      One thing I would say about Sufficient Power is that most psychology research I know of is not that good, to be honest. So then I actually don’t want them to take more resources to do that crap. 😀 I mean, what percentage of the Psych lit is just a regurgitation of things done 50 years ago with a new name? I would say like 70 or 80% ! really.

      Also I think you start to see what I mean about Cohen’s. And I am not necessarily saying preregistration is bad, but I do think the way we are implementing it is suboptimal, if I’m allowed to be honest.

      Yes, can agree that some of the results of the survey are surprising, and I think you are seeing what I mean about needing to take the psychology into account. you are perfectly right about them wanting to protect themselves, and while we (sometimes) apply psych to the problems, we hardly apply it to the change proposals that we are designing. Really, preregistration takes a lot of extra work and I think it will be unpopular for this (and maybe not be implemented long term, like Cohen’s suggestions).

      I also have hope for the field, we are making progress, in my opinion (and most people agree as well). whenever I get down about the state of affairs, I think to Kuhn’s assertion about when the best research happens (basically when we argue about fundamentals and disagree alot 😀 )

      And I wish you the best as well! 😀 Though I am not sure the best solution is for funders to *demand* anything. Check out this paper I wrote about how I think we should change the system. 🙂 It tries to use a twitter or FB like system to encourage scientists to do things that are also good for the group (look at how much the field has benefitted already from such platforms! 😀 )

      Best,
      Brett

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s