postcards from nowhere
...or triple entendre, or even more [depending on your source].
Stolen from Neuroanthropology's Wednesday Round Up #62:
Others have already written about the new voodoo-esque paper by Kriegeskorte et al. (2009). Having blogged extensively about the original controversy, I don't have anything to add... other than the links above... and some fun and wholesome pictures.Research Digest, It’s Those Voodoo Correlations Again … Brain Imagers Accused of “Double Dipping”
More methods problems for imaging researchers – using the same data twice, first to find the area and then to show that area is really the one responsible for whatever hypothesis is at stake. For more commentary, see Neuroskeptic, Mind Hacks, and Newsweek.
Reference
Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci. 12:535-40.
A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.
Of course, this problem is not present only in neuroimaging research. For instance, single unit recordings often use a similar logic. First one isolates units that satisfy certain criteria. Then, these units are tested with a number of related tasks.
ReplyDeleteAll these recent methodological "look at how smart I am and how stupid everyone else is" papers are nearly worthless, in my opinion (and they reflect on the pathetic quality of current editorial boards and reviewing system at high-profile journals). All these issues can directly be solved by replicating one's findings. Period.
Of course, the Kriegeskorte et al. paper doesn't confine itself to fMRI:
ReplyDelete"Although the example here concerns the selection of voxels in a neuroimaging experiment, the same caution should be applied in analyzing other types of data. In single-cell recording, for example, it is common to select neurons according to some criterion (for example, visual responsiveness or selectivity) before applying further analyses to the selected subset. If the selection is based on the same dataset as is used for selective analysis, biases will arise for any statistic not inherently independent of the selection criterion. For neurons and voxels, selection should be based on criteria that are independent of any selective analysis."
Unfortunately, most high-profile journals are not at all interested in failures to replicate.
42% of published papers contain at least one circular analysis. Wow!
ReplyDeleteI agree with anonymous -- there is something massively wrong with the editorial system. Who is letting these things through? Are they doing anything to fix it?
Nice post, Neurocritic. We have heard the ROIsts and the anti-ROIsts over the last couple of years screaming at each other over who is the "purest". You know who you are, leaders in the field. Both camps act like the students who just took their first statistics class and believe what's in their textbook comes from God. Has anyone read "Statistics as Principled Argument?" Research (I mean REAL research and discovery, not the neat post-hoc logic that goes into an NIH grant proposal) does not follow a clean recipe. Statistics are flexible and context-dependent tools. Until we solve the file drawer problem and we find a way to encourage and disseminate replication, I'm afraid this field will just keep going around in circles, with opposing cults calling each other names in high profile journals.
ReplyDeleteI highly recommend the Kriegeskorte paper, particularly the supplementary materials (in the form of a FAQ), to those who would like to better understand this methodological issue.
ReplyDeleteIt turns out that "circularity" can arise from non-obvious properties of BOLD fMRI data. For example, the use of orthogonal contrasts to define a region of interest (a standard practice with a sound statistical basis) can lead to bias from two mechanisms. The first is imbalance in the experimental conditions, a problem known to statistical sophisticates. But the second is from temporal autocorrelation in the time-series data, an idiosyncratic feature of BOLD fMRI, and is a problem that is often missed even by experts in the field.
Further, recent multi-voxel decoding techniques are exquisitely sensitive to hidden circularity.
Contrary to the comments of others here, simply replicating your results do not guard against the bias, if the subtle bias is present in both studies.