When do replications become wasteful?

This writing came about when Dalmeet Singh Chawla emailed me concerning a tweet about the potential slowing of science by engaging in too many replications and not discovering enough novelty.It eventually led to this article but as most of it didn’t get it (only the main quote), I will link the rest of it here.

The main thing to take away from all this is that we actually do know some things about people,  and replicating those things, or re-branding them as new, occurs way too often and substantially hinders (in my own opinion, at least) moving forward as a field. From the classic ‘confirmation bias’ to modern day ‘self esteem’ effects, biases, and processes have a dozen names, each with their own set of studies supporting essentially the same conclusion or underlying construct.

Yes, doing novel and revolutionary work is really difficult, but it is far too often that I go to a conference to see a prediction that most psychologists, if not most of the public would make (e.g., being excluded makes people feel bad, people take the path of least resistance). The focus is now on making sure what we believe is actually true, with many ‘high powered’ replication attempts and large scale efforts, but what are the opportunity costs?

 

It is clear that replications become wasteful as some point, the question is when, and this is what I would like to talk about and what I wanted to talk about when I posted the target tweet.

 

Without further ado, his questions and my slightly edited/ added to answers (I only had a very brief time in which to answer him!).

  1. Could you please explain to me what prompted you to post this tweet?

Put simply, there is a reason the departmental colloquia are more or less mandatory and that people are generally paying only mediocre attention at conferences..  and it is not because the work is extremely interesting!

Yes, interests differ, but I don’t think I would believe you if you said that you have never gone to a (potentially cross disciplinary) talk that demonstrated something that is already well established within your own field. Or demonstrated something that you would consider to be common knowledge and even slightly boring. It might have been done in a different population, with a different manipulation, even with a different name, but it was essentially something that was shown 50 or 100 years ago. I would say we’ve all been there, many times!

But this may not be because of a lack of work on the researcher’s part, o
ften times different fields (e.g., sociology, psychology, communications, advertising, group theorists, basically everyone) contain literature supporting common underlying ideas from different angles. A good example here is social capital and equity theory, which essentially indicate the amount of value they can extract from their network. Each area have hundreds of studies that support the notions, but each area is a silo of its own, rarely interacting with each other.

This is a major problem in my opinion, because we are essentially spending money demonstrating the same phenomenon over and over. and I would suggest ‘rediscovering’ these things over and over or arguing the minute details becomes a waste of taxpayer money at some point.

  1. What exactly is the ‘stasis’ problem that Dorothy Bishop describes in her tweet?

Essentially stasis when a field is not making so much progress or actually new knowledge, I would say. Philosophers of science talk about the best science as being conflicting or ‘creating waves’ (e.g., Galileo, Darwin) and this is basically a lack of that (more on this here and here).

There are, actually, more and less efficient ways of doing science; it is obvious that not publishing when something doesn’t work is bad, but so is doing the same thing over and over again without learning something new. It is important to learn new (real!) things.

  1. How does psychology face this problem and how do you think it can be resolved?

The concern now in psychology is that nothing is replicable and that we have basically constructed a house of cards, but this is simply not true. We Do know treplicationhings about people (e.g., they seek value, they gather and utilize meaning to do so). Those two statements alone explain… a LOT of psychology. And honestly, most of the nonreplicable studies actively sought to demonstrate how the tiniest thing could make such a massive impact.

Priming is actually a great example. Essentially the idea is that small changes in the environment can lead to large (behavioral) differences. It is in some sense all of advertising.

But the thing is that it is that these studies are actually designed to demonstrate how such a fragile little change can have such behavior changes. The studies are designed to be fragile, as fragile as possible! 😀 Things that shouldn’t matter, actually matter, this is what they show.

Now, what is at issue is whether reminding people of being old makes them walk slower. Is that important? I would most respectfully say not really. Would we want to challenge priming in general? I think this would be a poor decision, but maybe. We can easily spend half a million dollars, or 100 million, of taxpayer money to determine whether or not old primes make people walk slow, but that seems like a waste to me! 😀

The way to solve it, I would suggest, would be to first to utilize more social media for scientists. Just knowing what is going on, connecting with those in your field and sharing knowledge amongst the tweeps has a huge benefit in my own experience. More than that, I would suggest (as I did in this paper and a few other places) that we create a ‘special’ site for this. It would be a (functional and valuable!) social network for scientists that is designed in such a way, and has the tools, to aid the discovery and exchange of scientific knowledge. There are many great efforts, but this actually hurts the overall initiative as it splits the majority and thus value. Creating, remembering the passwords for, learning the how to, and building a network up on 5 different sites is Much harder than doing it for one.

  1. I thought the trouble in psychology is lack of replication/reproducibility, no?

This is a problem, and this is what the focus is on, but it is also a problem  to just keep redoing the same thing over and over again without learning anything new.

  1. What do you think is the answer to you the question i.e. is there a standard number of times a study needs to be replicated before it is accepted by a field?

No standard unfortunately, it is up to the individual scientists to choose good beliefs, and this is part of what makes a good scientists, I guess. – added after: we could do some sort of polling to determine the number of scientists who believe in a certain theoretical position or not, but this doesn’t mean the crowd is right. If we asked scientists if Galileo was right, we would conclude not! 😀

  1. Does it vary between different disciplines?

Definitely varies by discipline, again it is up to the individual scientists 🙂  – added after: one Super study in Physics should change the entire world’s mind, but in psychology it is not so easy as this, simply because what we study is more complex. Some things are the same (these are the things I study!) but many things are not, and we don’t Really know yet what changes and what doesn’t.

 

 

 

 

That’s all I will say for now! The resulting article is here but it doesn’t contain most of these ideas, which is why I put them here. 😀

Let me know what you think below or elsewhere!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s