On Expertise

What do you think about Expertise (within the field of Social Psychology)? From recent discussions it would seem that Any (social) psychologist should be able to ‘reproduce’ Any study withing (social) psychology, and I am not sure that this is really the case. It IS a simple fact of the matter that people differ in how good they are at designing experiments/ experimental protocols/ and the things that the participants see

Look at it this way: Some psychologists can’t even get participants to take their experiments; let alone really pay attention or care. that is a real thing. How can you expect to reproduce something if you can’t get participants to even take your experiments?

This point is especially driven home from my own research (and also many other people’s research). 1 Small effects are changed by small things. The wording of a single sentence, even a single word (!!!!), can seriously influence the outcomes of a study. How researchers treat their participants, from the first instance, in the consent form or the contact or the first email contact, conveys to them how much you care about their time, what they think the study is about, and ultimately affects how much they care about the study (and these things are likely to affect the results you get.).



THE THING IS, (from a conversation I had tonight with Levordashka) would you expect a chemist to be able to pick up any chemistry study in their field and be able to replicate it (on their first try.)? The difference is, I guess, that their studies take mixing a little baking soda and etc and ours require 100 people spending 20 minutes each, just to collect the data. 😀 3

No, We would not expect this, at least I would not. Even a specialist in the field, I would guess, needs more than a single try, even more than two or three tries, to get it right. Again, think of it this way: When you were in highschool, did you get the chemistry experiments right each time? MAybe you were better than me at chemistry, but I did good enough to get here, and I Definitely didn’t do it right each time. And those were, basically, the simplest experiments possible, explained in the simplest ways possible, with a bunch of other people around you trying to do the exact same thing at the same time. I mean,.. that is a serious thing! 😀

Do I think that psychologists can (should?) do a better job of reporting their studies and being more honest about what is involved, and even sharing the exact materials they used? Yes, absolutely, it is even better for me as an individual researcher (if the effects replicate better and I am actually interested in doing real science). But to say that just because someone couldn’t replicate an effect, that that effect doesn’t exist, or even that it isn’t robust, is too far in my opinion. I’ve gotten a solid effect 5 times and if you tried to tell me that you couldn’t get it, I would absolutely tell you it is probably your fault. Really. 😀  When I do it, it works. But my work is not important enough to replicate (and I don’t have enough of it, yet :,).


it IS a fact that some people are better at designing experiments (or coming up with ideas for experiments, or writing, or etc) than other people and that is ok for me. These things are why we collaborate, exactly so we don’t have to be great at every aspect. I am not sure how to solve the problem of expertise (or how it affects replication more generally), but I definitely think that to dismiss expertise, especially the expertise of some of the older generation in the field, who have been studying psychology for 50 or 60 years, would be foolish at best. Maybe the expertise is not in designing  or doing high powered experiments, but I could definitely see how that experience might influence participants to care more, or definitely in designing the experiments (in the case of explicit conceptual replications, or trying to directly replicate studies without original materials, consent form, approaching them in the same way, etc).  These things matter, I think we know this (e.g., advertising).

In any case, I have to go, I am tired, and I want to go before I run out of things to say. 😀 Thank you for coming by and I do wonder what you have to say about the things I’ve said! BEst, Brett.



1: We have a quite replicable difference between two conditions, and we have even shown that we can ‘switch’ the person’s behavior by putting one condition after the other. In our last study we made another tiny change, namely pushing it even further by putting the two conditions at the same time. Both effects work, so why not? But it didn’t work. I am pretty sure it is because the participants figured it out (it reveals quite plainly a kind of stupid thing people do). The effect is significant in the first trial but none of the others or all together. This is a ‘conceptual replication’ that failed, even with ‘expertise’ (at creating the original effect).

2: Footnotes are actually pretty useful for sidenotes, so ya, thanks Simine 😀

3: Psychology is harder than Chemistry? 😀 just teasing! but I think so. (thats why I do it?) 😀


Let me know what you think! 😀 On Twitter (@brettbuttliere) or here, or where ever. Thanks again! 🙂



  1. “It IS a simple fact of the matter that people differ in how good they are at designing experiments/ experimental protocols/ and the things that the participants see”

    1) How is one to determine what is a “good” design in the first place. I reason that deciding if a study is well-designed is probably highly related to the study having found “significant” results. However, the high prevalence of low-powered studies, selective reporting, and data-analysis flexibility makes it possible to find significant results for just about anything. This in turn makes it very hard, if not impossible, to even determine which studies are examples of studies that are well designed.

    2) Leaving that major possible problem aside, when i was studying behavioural science i never saw any professor execute their own research. They usually had research assistants (read: undergraduates) to perform it for them. So, if “expertise” is important, apparently it’s importance lies mostly in designing the research, not in executing it.

    3) Now, reasoning from there on, a direct replication should be able to copy the design of the original research quite closely, and thus copy the “expertise” that made it possible to design the experiment in the first place. So, no extra “expertise” is necessary because it is already included in the design, and thus replication, of the study.

    Because of 1, 2, and 3 i see no reason in talking about “expertise”. It clouds any possible discussion about what a “good” design is in the first place, what a “good” replication is, and why some researchers seem to get significant result after significant result while others, e.g. in replicating, do not.

    1. 1) I would suggest it is a similar process to measuring intelligence or most other latent traits. Lets decide what matters and then try to measure it 😀 Also we should expect ‘good’ results when it is done well so this is somehow confounded. Catching those who try to fake expertise is obviously what those doing finding QRPs are trying to reveal.

      2) Yes, potentially in designing, or in how you tell the RAs to interact (e.g., friendly, serious, etc). These do matter, and definitely the boss ya know, looks at the survey before it is taken, it would be interesting to examine different emotions or stress or person variables to the same study designed by different people. That would be Really neat actually.

      3) Yes, close, I think it would be a good way to test this if we had 10 or 15 people design the study and then measure differences in it. Would also answer how close they can be.

      Would you say then that expertise doesn’t exist? what a good design is in the first place is exactly expertise, I would suggest, as the expert uses them. The chemistry example I think is a good one, or what would you say?


  2. “Would you say then that expertise doesn’t exist? what a good design is in the first place is exactly expertise, I would suggest, as the expert uses them.”

    I am finding it difficult to think about what expertise really is. It’s a vague term to me, which makes it useless i gather.

    If you think that “expertise” makes for a “good” design, then I think my main point is that it might be more fruitful to not talk about “expertise” but instead talk about what makes a “good” design. This seems much more fruitful to me.

    Thanks for the post and reply!

    1. Hey,

      No problems of course! 😀 Yes, I think the field needs to spend more time thinking about what exactly expertise is, because it is fuzzy, but I do also think it should not be harder to measure than intelligence, for instance.

      On facebook we also talked about focusing more on the study than the person, but I do also think that good persons will use good studies, which produce those good results


  3. Hey Brett,

    This reminds me of a paper Paul Meehl wrote on a “theory of the instrument.” He talked about how physicists pay very close attention to how a particular measurement instrument works when trying to use it making inferences. This allowed them to tell what was due to the quirks in a particular way they were measuring things, and what was due to real differences in the phenomenon being measured. To me this is a really big thing that needs to be addressed in psychology. We need to make sure our “instruments” are calibrated correctly. And I think you’re right to suggest that there are probably a lot of implicit things that people know about how data collection works–participants earlier in the semester are more conscientious, participants sometimes use only the top half of a rating scale, one of my RA’s is socially awkward and puts people less at ease than the others–that are relevant to making inferences from data. My thought is that these are the places where you’re likely to see effects, not necessarily in the design of a study or being able to get people to take the study (although maybe there are tricks to advertising the study in a given sample).

    But I also think that, ideally, this information would become explicit so it can be accounted for by everyone–regardless of expertise. I am less comfortable with the idea that a scientist just needs to have a “special something” to make a study work when we can’t then unpack and study exactly what that “special something” is. So I agree that there may be pro techniques that rookie psychologists don’t know to do when trying to complete an experiment and analysis, but I would hope that we can start to treat them scientifically and quantify exactly what they are, as opposed to just stating broadly that they are there (and thereby giving people who use QRP’s a plausible out).


    1. Hey Alex,

      Yes, good suggestion, actually I have been reading his things in the last week Theoretical asterisks and watching his class when he talks about the rat models and how they were getting different results. Agreed about trying to identify and and study that special something. but I do think there probably is a special something, many studies feel horrid to take, I would suggest. these probably get different results, ya know? Even just the focus when designing the study can have big effects I guess.

      How do you think we can best address it? It does seem like that Many labs approach would be a cool way, just have a bunch of people implement the study their way in Qualtrics and then examine how they are different. Lets do that also with the survey. Will look at it today as well some more.


      1. Or could also just ask people who studied e.g., ego depletion to send us a link and then quantify the things from there?

        like: anyone who has tried to replicate and can send us a Qualtrics link, please do type thing. but it seems like more work and harder than the other way

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s