Statistics

Let’s pretend we all agree priming old doesn’t make people walk slower, now what?

The last years there has been an inordinate amount of attention and resources dedicated to examining whether subtly priming people with old concepts can make them walk slower (nocitationneeded). It has gone on for years and taken pages and pages of our limited journal blog and feed space.

Yesterday, I was talking with Brent Donnellan and Uli Schimmack in the Psychological Methods Discussion Group and they suggested that it is important to determine whether old primes make people walk slower. (also look at the way that Donnellan entered his comment with only 1 sentence and then came back later and finished the comment after Uli and I were done talking; this is something I will have to watch out for in the future, especially as the time nuances get lost later)

Anyways, rather than again having some empty argument about replicability, regurgitating all the same old party lines, I simply agreed with them that making people think about old either doesn’t, or only in some contexts, make them walk slower. I’m ok with saying that and I think you should be too (there are at least a few failed replications, so it doesn’t ALWAYS work, even if it does).

Then the question is.. now what?

So we said that, but what did we gain? very little in my opinion.

There are almost no contexts in which this old-slow link matters, I mean, when does it matter whether this link exists or not? Never! 😀 The point of the experiment (as also said by Bargh, Chen, and Burrows) is to say something about how our thoughts and behaviors depend upon factors such as what is salient in our mind at the time.

And this I would say is fairly well established, from the exogenous emotions literature, to stereotype threat, to the IAT, all depend some sort of stimuli priming a certain behavior (which, in the case of the IAT then makes it harder for them to do the opposite).

What would it mean for (Social) Priming Theory to throw out this experiment/ paradigm?

..almost nothing as far as I can tell.

Even the original Doyen et al (2012) failure to replicate says nothing about the theory for which Bargh et al utilize the study to provide evidence for. The whole discussion is about how poor the methods are and how we need to do better. Saying the methods can be better is something I am totally ok with, and I would even suggest that this may be why it is presented in a chapter, rather than in its own paper.

The idea they are suggesting stands with or without this paper. Or does it? This is what I would like to ask and this is the point of the blog post. I see little value in endlessly debating the (un)reality of the ability for old primes to make people walk slower, unless it says something for the theory, but it doesn’t (as far as I can tell).

What part of the theory is at risk here?

What makes this study matter? So far as I can see, it says nothing novel and has relatively little value, theoretically speaking (and even those decrying the study have said little about its theoretical implications). Hold it up as an example of bad methods, that is fine with me, but we are not forwarding ourselves by saying that this effect does not replicate or exist (as far as I can tell). It has no implications (or please point them out!).

Let us define which aspect of the theory (of which this is just an example) is vulnerable and then examine the literature to see if the notion is supported elsewhere. My guess is that it will be.

  • Would we say it calls into question that unconscious stimuli can affect our later behavior? that seems..

    Would it actually matter at all? it doesn’t seem like it.

    far. Certainly I would not feel comfortable saying that there are no examples where stimuli we don’t ‘consciously’ experience change our behaviors. What about nudges, or the IAT, these are essentially demonstrations of a stimuli making a certain response more likely (which also makes it harder to do the opposite, in the case of the IAT).

  • Is it that the stimuli is social? It does not seem absurd to say that our behavior can be unconsciously changed by the people that we (expect to) interact with. After all, I suspect that (at least sometimes) you change the clothes that you will wear based upon who you expect to interact with throughout that day. Even the IAT is social and about prejudices.
  • Is it that the people are not actually there? Neither is santa or ghosts, but they change some people’s behavior! 😀

Once we know which part of the theory behind the potentially false study is being questioned, we will be on our way to making real progress in determining whether there is truth there or not.

Until then, let’s just agree to do better in the future, and we can even pretend that the prime is false, because it will have no substantial effects upon the broader theory (of which this study is only an example).

In sum

I am not so much interested as Brent or Uli in determining whether old primes make people walk slower. To me, this is not an important question, unless it says something about the theory, but nobody seems to be arguing that (social) priming doesn’t exist. So I wonder how much it actually matters. 😀 Maybe instead we should move on to something more interesting, like how we can use science to improve science or understanding why many of the female pronouns are longer than the male pronouns (except in the family setting). 

me priming you to like this post, or does it not work?! we’ll never know if we keep up like this.

Advertisements

As simple as possible ANOVA

Ok, here we will explain, as simply as possible, what an ANOVA is and how to do it. We explain it using a simplified 2 group example, which means it is basically the same as the t-test.

This is ANOVA You're going to use it for the rest of your life - This is ANOVA You're going to use it for the rest of your life Scumbag Teacher

The whole point of the ANOVA is to test, generally, whether two groups are different from each other or not. The way this is done is called an ANalyses Of VAriance.

Basically, we are testing the amount of variation that is between the groups vs. the amount of variation that is within each group. If there is more variation between groups than within the groups, then we say there is a ‘statistically significant’ difference.

This relies on two concepts. The first is variance, which is the spread of the scores, in general (i.e., all of the scores you have). The second is the distinction between what is called ‘within’ group variance and ‘between’ group variance. In Figure 1 the within group variance is how much the number of books varies within each group, those who read the blog and those who don’t. The between group variance is how much the two group averages differ. We essentially examine if the averages of the two groups are sufficiently different, given how much variation there is within each group. 🙂

For instance, let’s pretend we have some data on the number of books read by two different types of people; those people who follow this blog vs those who do not. We have asked 100 people in each group and have the results in table 1. The means are the blue diamonds and the standard deviations [how we quantify the variation] for each group is the green or orange colored bars around the diamonds.

Now, here is where the ANOVA comes in. The critical question is whether the two groups are different, on average (note that we can rarely say anything about the individual level).

If there are no group differences between the two groups, then the mean number of books read should be close to equal. If they are very different, then there might be reason to say that those who follow are somehow different from those who do not. We use the ANalyses Of VAriance to tell us.

Specifically, we look at the averages of the groups and how much the scores vary around those averages.

In figure 1, you can see three potential outcomes of our book study which easily demonstrates the ANOVA.

In the first two potential outcomes, the averages are the same, 36 and 58. In the third potential outcome, the group means are about 25 and 75.  The question in all three potential situations is whether these averages are different; and whether they are depends upon how much variation there is within each group (the orange and green ‘error’ bars). In the first potential outcome, there is much variation within each group and it is hard to tell is a person in the middle follows the blog or not. In the second potential outcome, there is less variation between the groups, and we can more confidently predict whether a person is in the blog or not based upon how many books they read.

ANOVA

Notice that there does not necessarily need to be ‘a lot’ of distance between the groups, if the variation within the groups is small (Potential outcome 2). Also recognize that even if there is a Lot of variation between the groups, if the group differences are large enough, it still may be considered ‘statistically significantly’ different (Potential outcome 3).

 

 

The keys in ANOVA are how much the two groups differ, and how much the individuals within each group differ. If there are large differences within each group, it makes it harder to say that the groups are ‘statistically’ different because there is more overlap between groups (given the same mean difference; see Figure 1).

 

 

For an excellent mathematical treatment of the ANOVA, please see here.

And that is how we can explain ANOVA as simply as possible.