What do you think about Expertise (within the field of Social Psychology)? From recent discussions it would seem that Any (social) psychologist should be able to ‘reproduce’ Any study withing (social) psychology, and I am not sure that this is really the case. It IS a simple fact of the matter that people differ in how good they are at designing experiments/ experimental protocols/ and the things that the participants see
Look at it this way: Some psychologists can’t even get participants to take their experiments; let alone really pay attention or care. that is a real thing. How can you expect to reproduce something if you can’t get participants to even take your experiments?
This point is especially driven home from my own research (and also many other people’s research). 1 Small effects are changed by small things. The wording of a single sentence, even a single word (!!!!), can seriously influence the outcomes of a study. How researchers treat their participants, from the first instance, in the consent form or the contact or the first email contact, conveys to them how much you care about their time, what they think the study is about, and ultimately affects how much they care about the study (and these things are likely to affect the results you get.).
THE THING IS, (from a conversation I had tonight with Levordashka) would you expect a chemist to be able to pick up any chemistry study in their field and be able to replicate it (on their first try.)? The difference is, I guess, that their studies take mixing a little baking soda and etc and ours require 100 people spending 20 minutes each, just to collect the data. 😀 3
No, We would not expect this, at least I would not. Even a specialist in the field, I would guess, needs more than a single try, even more than two or three tries, to get it right. Again, think of it this way: When you were in highschool, did you get the chemistry experiments right each time? MAybe you were better than me at chemistry, but I did good enough to get here, and I Definitely didn’t do it right each time. And those were, basically, the simplest experiments possible, explained in the simplest ways possible, with a bunch of other people around you trying to do the exact same thing at the same time. I mean,.. that is a serious thing! 😀
Do I think that psychologists can (should?) do a better job of reporting their studies and being more honest about what is involved, and even sharing the exact materials they used? Yes, absolutely, it is even better for me as an individual researcher (if the effects replicate better and I am actually interested in doing real science). But to say that just because someone couldn’t replicate an effect, that that effect doesn’t exist, or even that it isn’t robust, is too far in my opinion. I’ve gotten a solid effect 5 times and if you tried to tell me that you couldn’t get it, I would absolutely tell you it is probably your fault. Really. 😀 When I do it, it works. But my work is not important enough to replicate (and I don’t have enough of it, yet :,).
it IS a fact that some people are better at designing experiments (or coming up with ideas for experiments, or writing, or etc) than other people and that is ok for me. These things are why we collaborate, exactly so we don’t have to be great at every aspect. I am not sure how to solve the problem of expertise (or how it affects replication more generally), but I definitely think that to dismiss expertise, especially the expertise of some of the older generation in the field, who have been studying psychology for 50 or 60 years, would be foolish at best. Maybe the expertise is not in designing or doing high powered experiments, but I could definitely see how that experience might influence participants to care more, or definitely in designing the experiments (in the case of explicit conceptual replications, or trying to directly replicate studies without original materials, consent form, approaching them in the same way, etc). These things matter, I think we know this (e.g., advertising).
In any case, I have to go, I am tired, and I want to go before I run out of things to say. 😀 Thank you for coming by and I do wonder what you have to say about the things I’ve said! BEst, Brett.
1: We have a quite replicable difference between two conditions, and we have even shown that we can ‘switch’ the person’s behavior by putting one condition after the other. In our last study we made another tiny change, namely pushing it even further by putting the two conditions at the same time. Both effects work, so why not? But it didn’t work. I am pretty sure it is because the participants figured it out (it reveals quite plainly a kind of stupid thing people do). The effect is significant in the first trial but none of the others or all together. This is a ‘conceptual replication’ that failed, even with ‘expertise’ (at creating the original effect).
2: Footnotes are actually pretty useful for sidenotes, so ya, thanks Simine 😀
3: Psychology is harder than Chemistry? 😀 just teasing! but I think so. (thats why I do it?) 😀
Let me know what you think! 😀 On Twitter (@brettbuttliere) or here, or where ever. Thanks again! 🙂