Motivating a Cumulative Cognitive Neuroscience

Why are large-scale structured databases and meta-analyses important to advance the field of human brain mapping? One reason is that individual functional magnetic resonance imaging (fMRI) studies can be notoriously unreliable and underpowered (Bennett & Miller, 2010; Fliessbach et al., 2010; Kriegeskorte et al., 2009; Vul et al., 2009). At the recent CNS 2010 Annual Meeting, symposium organizer Dr. Tal Yarkoni gave the first talk in a session on the value of a cumulative cognitive neuroscience.

---------------
Symposium Session 1
Sunday, April 18, 10:00 am - 12:00 pm, Westmount et al Ballroom

Towards a cumulative science of human brain function

. . .

Talk 1: Motivating a cumulative cognitive neuroscience

Tal Yarkoni; Columbia University and University of Colorado at Boulder

Thousands of functional neuroimaging studies are published every year. Only a small fraction of these studies explicitly attempt a formal synthesis of previous findings. In this talk, I argue for an increased emphasis on cumulative approaches to the study of brain function that aim to synthesize and distill the results of previous studies. Three different motives for such an approach are discussed, including (a) the need to distinguish real findings from false alarms; (b) the desire to organize both cognitive tasks and brain activations into coherent ontologies; and (c) the high likelihood that many fMRI studies are underpowered and consequently produce distorted results. I focus primarily on the last of these points, using simulations and empirical analyses to demonstrate that the results of many individual fMRI studies are likely to appear considerably stronger and more selective than they actually are. I conclude by arguing that these limitations are difficult or impossible to overcome in individual studies, necessitating a stronger focus on consensus building at the disciplinary level.
---------------

What are the motivations for consensus building? Here are four major reasons:

The value of a cumulative science

• Make the literature manageable
• Distinguish true positives from false positives
• Develop overarching frameworks
• Minimize the effects of low power

Yarkoni's talk focused on the last point. The problem with most individual fMRI studies is a lack of statistical power. Yarkoni (2009) argued that:

the primary cause of grossly inflated correlations in whole-brain fMRI analyses is not nonindependence, but the pernicious combination of small sample sizes and stringent alpha-correction levels. Far from defusing Vul et al.'s conclusions [from their notorious 2009 paper], the simulations presented suggest that the level of inflation may be even worse than Vul et al.'s empirical analysis would suggest.

Fig. 2 (Yarkoni, 2009). Inflation of significant r values as a function of sample size (x axis) and population effect size (lines). Each point represents the result of 10,000 simulated correlation tests, each conducted at a threshold of p < .001, reflecting the most commonly used whole-brain threshold. Dashed lines represent the true correlation size; solid lines represent the mean observed correlation in the sample for only those tests that produce significant results.

Simply put, small n's result in massively inflated brain-behavior correlations. What can be done about this problem? Include more participants in your studies! And make use of the tools that were described by the subsequent speakers (Van Essen, Wager, Poldrack) for synthesis of mega-databases.

For more information, the slides from Tal's talk are available online (PDF).

References

Bennett CM, Miller MB. (2010). How reliable are the results from functional magnetic resonance imaging? Ann NY Acad Sci. 1191:133-55.

Fliessbach K, Rohe T, Linder NS, Trautner P, Elger CE, Weber B. (2010). Retest reliability of reward-related BOLD signals. Neuroimage 50:1168-76.

Kriegeskorte N, Simmons WK, Bellgowan PS, Baker CI. (2009). Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci. 12:535-40.

Vul E, Harris C, Winkielman P, Pashler H (2009). Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition. Perspectives on Psychological Science 4:274-290.

Yarkoni, T. (2009). Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power-Commentary on Vul et al. (2009). Perspectives on Psychological Science, 4 (3), 294-298. DOI: 10.1111/j.1745-6924.2009.01127.x