There is much nice discussion recently, most recently a Data Colada post about how difficult it is to estimate an effect size.
i am not sure if I wrote it up before or not, but the argument is succinctly captured by trying to estimate the average size of fish in any particular pond (except that is way way easier). Each particular effect size a (group of) researcher(s) gets is like catching a particular fish in that pond (though of course sampling error is essentially taken care of in the pond example).
If one catches 10 or 15 fish from the pond, one can begin estimating the ‘average size of fish in the pond’. But of course, this is only ‘the average size of fish that we caught, in the pond’.
What you catch depends on how you fish…
The key is that, of course, the size of fish one catches in the pond is related to how one fishes, where one looks, what type of bait one uses, the strategy of reeling, the time of day, the depth that one looks at, and even things like how one approaches the spot one will try to fish.
and of course one only takes a picture/ documents with the largest fish, and the fish become bigger over time (so long as no picture is present), and of course, when it is the person who owns the land doing the documenting and they want people to come fish their land, there is some.. potential upward drift of the average size of the fish.
This says nothing of the fact that in our real world example, we don’t actually know how big the pond is, or even if there are fish in it, and we can mistake an old shoe for a fish and a fish for an old shoe.
The key is, of course, that one must be careful in making proclamations about the size or presence of fish in the pond.
On suggestions there are no fish in the pond..
This is especially true after any single fishing trip, or any group of fishers that all use the same lure, or look in the same area or the same way… as maybe they were just doing it wrong, or at the wrong time, or with the wrong bait, or etc etc.
In any case, I think you get the point. Would be happy to discuss below.
Love ya, Keep on,