Dr. David Amodio has graciously taken the time to respond to many of the criticisms of his paper, Neurocognitive correlates of liberalism and conservatism, made by The Neurocritic and others. [EDIT: His comments first appeared here, and I have taken the liberty to repost them as a new entry.]
Hi Neurocritic,And here is my reply.
I like your blog – it’s important to a have a critical voice out there. However, given how this study has been misconstrued and sensationalized in the media, as well as among science bloggers, it’s important to address the criticisms directly. Though I’ve generally not worried about the “lay” coverage (how can you argue science with pundits?), it might be worthwhile to respond to a blog that is read by neuroscientists (including myself from time to time):
1) There were no gender differences on any variable. Moreover, 63% women is actually fairly balanced for a psychology study, so I’m surprised this has even come up. We didn’t report gender effects for the sake of brevity, though in hindsight, I wish we had slipped it in.
2) People have complained that there were more liberals the conservatives in the sample. True, in an absolute sense. But this is typical in political psychology: Americans are more conservative on average, and so more extreme conservatives usually rate themselves as moderate conservatives, whereas moderate liberals tend to rate themselves more extremely (see Linda Skitka’s work and comments on the paper). It’s a scaling issue that psychologists deal with all the time.
Nevertheless, we’re talking about a correlation. The clear linear effect suggests the stronger liberalism is associated with greater conflict-related ACC activity. Not sure how anyone can argue with that.
3) The sample was actually rather large for a neuro study. Also, please note our use of *inferential statistics* – I’ve been surprised by the criticism of the size given the strength of the effect!
4) Outliers? There weren’t any. (Not sure what Broussard was referring to in the highly-critical comment you posted. Maybe someone should lend him a stats book…)
5) On reporting group differences in RTs and error rates – to be clear, we did not conduct group analyses (though one graph displays the median split of ERN waves). We looked at correlations along a continuum. Group analyses would have been psychometrically problematic, and furthermore, we didn’t want to suggest that political orientation is categorical. Though of course this didn’t stop the media and bloggers to speak in terms of categories…
In the end, the study reports a correlation. You can’t “disprove” it – you can only interpret it. Our interpretation was face valid – this measure of political orientation was strongly correlated with the ERN and No-Go N2 from the Go/No-Go task. Simple as that. Might there be 3rd variables at play? Probably. But that doesn’t contradict our interpretation or cast doubt on the quality of the study.
Good science is an art. But so is good science critique. Without a plausible alternative interpretation, you don’t have a critique. I suppose people are just cranky because this bullet-point of a study has been so over-sensationalized. Or maybe some folks just aren’t familiar with how you do this kind of research. So I hope this post clarifies some things.
DA
Hi DA,Thank you for taking the time to respond and for being so gracious despite some...well...highly critical remarks. Some of which are not my own, so perhaps I should not have been such a "sounding board." I just have a couple of replies to your comments.
1) I (personally) didn't have an issue with possible gender differences, but it's good to have that clarified.
2) I'm rather ignorant of standard rating systems in political psychology, but is self-rating always used? Aren't there more "objective" questionnaires used to classify American participants along the liberal-conservative continuum?
I'm not familiar with Linda Skitka's work. I wouldn't know where to start [even if I had time to read her papers], perhaps with Skitka & Tetlock (1993)? 1 [Hmm, Mullen et al. (2003) looks quite interesting, if not entirely relevant here.]
3) Your correlations are strong, yes. It seems people had more of a problem with the restricted range of the conservative sample.
4) You're right, he's wrong, not sure which points would be considered outliers. I went back to the original post and put the offending passages in strikeout font. However, it does seem the 7 conservatives' ERNs were quite variable (values from -2 to -23).
5) I was basing my comments about group analyses on the fact that you had to do them to show the EEG data in Figure 1b. And the fact that mean accurate rates were reported in (ahem) newspaper articles. My criticism about the lack of RT data in the paper still stands:
There was absolutely no information about RTs at all, so we don't know whether there was a speed-accuracy trade-off in the conservatives (a reckless and disinhibited response style) or whether they were "conscientious" (RT comparable to [or slower than] liberals), but just couldn't stop themselves from pressing the key on No-Go trials.That could comprise part of an alternate explanation, along with "3rd variables at play" as I've sarcastically suggested with my Liberals Are Neurotic and Conservatives Are Antisocial quip.
At any rate, I would beg to differ that good science critique must provide an alternate interpretation. Some science critique can be based on methods, analysis, reporting of results, etc. Standard peer-review stuff.
My closing remarks are based on the assumption that the error-related negativity (ERN) brain wave is a direct measure of conflict monitoring in the anterior cingulate cortex (ACC). [I've focused on the ERN rather than the N2 because the former is illustrated in the paper and the latter is not.] A quick review of the literature indicates that's not necessarily the case. First, not everyone agrees that the ERN measures response conflict rather than error processing more specifically (Carbonnell & Falkenstein, 2006), or that ACC hemodynamic activity during error commission is a reflection of response conflict (Critchley et al., 2005; Garavan et al., 2003). Second, when people make mistakes, it seems that more of the brain is active than just the ACC (Klein et al., 2007; Ullsperger & von Cramon, 2006).
At least, these are my impressions...
TN
Footnote
1 MeasuresReferences
Liberalism–conservatism
To measure the cognitive conservatism and liberal-humanism factors identified in earlier research (high scores on dogmatism, authoritarianism, and identification with the political right; Skitka & Tetlock, 1992) we adopted Altemeyer's (1981, 1988) 30-item balanced Right-Wing Authoritarianism (RWA) scale. The RWA scale is an updated measure with sound psychometric properties (see review by Winter, 1990) that we judged to be superior to the scales we used in earlier studies of ideo-affective resonances. The scale ranges from liberal-democrat to authoritarian (see Altemeyer, 1988, p. 263). Altemeyer (1981) reported Cronbach's alpha of .88 for a sample of 965 University of Manitoba students. We observed an alpha of .86 with the present sample, indicating adequate internal consistency in measuring the construct.
Carbonnell L, Falkenstein M. (2006). Does the error negativity reflect the degree of response conflict? Brain Res. 1095:124-30.
Critchley HD, Tang J, Glaser D, Butterworth B, Dolan RJ. (2005). Anterior cingulate activity during error and autonomic response. Neuroimage 27:885-95.
Garavan H, Ross TJ, Kaufman J, Stein EA. (2003). A midline dissociation between error-processing and response-conflict monitoring. Neuroimage 20:1132-9.
Klein TA, Endrass T, Kathmann N, Neumann J, von Cramon DY, Ullsperger M. (2007). Neural correlates of error awareness. Neuroimage 34:1774-81.
Mullen, E., Bauman, C. W., & Skitka, L. J. (2003). Avoiding the pitfalls of politicized psychology. Analyses of Social Issues and Public Policy, 3, 171-176.
This article provides two arguments for using caution when interpreting the results of a Global Change Game simulation indicating that people high in right-wing authoritarianism are particularly likely to bring the world to ruin. First, we review research that demonstrates that extremists on both the political left and right share characteristics likely to be associated with poor performance in the Global Change Game (e.g., lower levels of integrative complexity) and that there are reasons to be cautious about letting political extremists on either the left or right inherit the earth. Second, we caution that political psychologists need to be aware of how their own values shape the types of research they conduct and the inferences they draw from that research and that the same results can be construed very differently depending on the lens through which they are viewed.
Skitka, L. J. & Tetlock, P. E. (1993). Providing public assistance: Cognitive and motivational processes underlying liberal and conservative policy preferences. Journal of Personality and Social Psychology, 65, 1205 - 1223.
Previous research in a wide variety of policy domains (e.g., azidothymidine for AIDS patients, low-income housing) has indicated that under no scarcity, liberals tend to help all claimants for assistance, whereas conservatives withhold assistance from people who are personally responsible for their predicament (Skitka & Tetlock, 1992). Three studies explore 6 explanations for this robust finding: deterrence, self-interest, punitiveness, mindlessness, value orientation, and avoidance of trade-off reasoning. The findings shed light on both the cognitive strategies and motivational priorities of liberals and conservatives. It was discovered that liberals are not mindlessly egalitarian, but try to avoid socially awkward value trade-offs that require placing monetary values on lives. By contrast, conservatives are motivated to punish violators of social norms (e.g., deviations from traditional norms of sexuality or responsible behavior) and to deter free riders.
Ullsperger M, von Cramon DY. (2006). The role of intact frontostriatal circuits in error processing. J Cog Neurosci. 18:651-64.
Neurocritic,
ReplyDeleteMy own modest contribution to the discussion:
My blood boiled when I read the abstract of the study, but I see no that Dr. Amodio is right to some degree that the results have been overexagerrated by the media. (The LA Times article on the study was simply journalistic malpractice; it wasn't for another several days that I read a more balanced article about it, which calmed me down a good bit).
The problem with using self-identification is that people tend not to do it honestly, as Dr. Amodio noted. It does not seem to me that they deviate to the right, as he said, but to the center -- everyone likes to describe themselves as moderates or independents. I suppose it's all a matter of where you stand, though. Nonetheless, there were actual objective measures that could have been used. I understand Kerlinger's scales of liberalism and conservatism are fairly reliable and valid.
My only other beef with Dr. Amodio at this point is his claim that Americans tend to be more conservative, which he used to justify a smaller sample of self-identified conservatives in the study. Politics is entirely a relative thing; in comparison to whom are Americans more conservative? Europeans/Canadians? One might just as easily respond that Americans are more liberal in comparison to, say, Middle Easterners and east Asians. In the long run, I sincerely doubt the results could be generalized to populations outside the United States, which seems to be what Amodio et al. were aiming for., so adjusting for "more conservative" Americans was probably unnecessary.
SW
As they note in the paper, the self-report measure (one question) does correlate with self-reported voting behavior as well. I suppose that's not a bad quick measure, though it would be nice to have a more nuanced measure. It sees odd to think of "conservative" and "liberal" as real psychological dimensions, rather than as clusters of values on several dimensions, with a significant amount of within-group variance on those dimensions.
ReplyDeleteAlso, I think your point about the low number of conservatives is a good one. Because there aren't many on the conservative side of the spectrum (do conservatives sometimes rate themselves as liberal, 'cause we live in a conservative country?), I'd want to see how much of the variance is accounted for by liberals alone. If the conservatives aren't contributing much to the effect, then you have to wonder what the results say.
But the biggest problem with the study has to be the theoretical connection between monitoring conflicts in information processing streams (not the definition they give in the paper, but it is the actual definition of conflict monitoring) and anything associated with conservatism or liberalism. It's not that I question their data, just that I don't think there's any reason to believe that this is a political effect alone, and since it's correlation data, it's gonna take a lot of future research to change my mind
Another new study on political psychology in Psychological Science. Apparently RWAs also have issues with interpersonal disgust, which may mediate some of the previous prejudice findings associated with this trait.
ReplyDeleteHodson & Costello (2007). Interpersonal Disgust, Ideological Orientations, and Dehumanization as Predictors of Intergroup Attitudes. Psychol Sci, 18, 691+
http://www.blackwell-synergy.com/doi/abs/10.1111/j.1467-9280.2007.01962.x
Sure would like to see an fMRI study on this. I would assume we might get some decactivation of the mPFC (c.f. Harris & Fsike, 2006) and possibly some insula activity.
Thanks for the reference to the Hodson & Costello paper. One would indeed predict that right-wing authoritarians might show an exaggeration of the extreme out-group "dehumanizing" pattern of brain activity reported by Harris & Fisk (2006):
ReplyDeleteDehumanizing the lowest of the low: neuroimaging responses to extreme out-groups. Psychol Sci. 17:847-53.
Kudos to Amodio for responding directly to your comments. Unfortunately, his defense is characterized by the same intellectual sloppiness that I found so objectionable in the presentation of his study. One would have to have considerably more detail about the work in question to essay the sort of scientific critique he claims to prefer; notwithstanding his disclaimers, however, the basis for media reaction is to be found in the casual assumptions which proliferate at every turn.
ReplyDeleteFor example, he defends the glaring disparity in numbers between "liberal" and "conservative" participants with a questionable generality which has virtually nothing to do with why such a disparity is or is not seriously problematic: "Americans are more conservative on average, and so more extreme conservatives usually rate themselves as moderate conservatives, whereas moderate liberals tend to rate themselves more extremely (see Linda Skitka’s work and comments on the paper)."
More conservative than whom? Even Skitka doesn't explicitly fill in that blank, per Judy Peres at the Chicago Trib: "'We're not a very liberal country,' she said. 'We're more likely to find extreme conservatives in the U.S. than extreme liberals.'" This falls into the "everybody knows" class of assertions made by political groups of almost every persuasion. Like many of Amodio's citations, it is neither as persuasive nor as definitive on the actual point at issue as one might have expected.
Essentially, both Amodio and Skitka operate from a vaguely postulated underlying presumption of what is, in reality, a patently Eurocentric international "norm." It's certainly helpful to know this in terms of bias, but it doesn't begin to make up for the stark imbalance in his study. As an aside, if he's arguing that the study therefore included 11 conservatives, not 7, one can hardly be blamed for wondering if the 85% predictability factor which he cited to bolster his self-selecting criterion is not also in play. Self-designation basically has only one advantage; it saves Amodio the almost impossible task of actually defining his terms. That fundamental lack of rigor strikes me as emblematic.
Amodio's claim that this is just a typical "scaling issue" to the contrary, there's simply no good reason for buidling such an obvious disproportion into this sort of research project. I suspect he simply went with whoever walked in the door, and only 7 conservatives showed up. That, in itself, could have significant ramifications with regard to the psychological profiles of his volunteers. If the limited number of conservatives available for testing indicate they are a distinct minority on a liberal campus, for example, one would also have to factor in an environment decidedly at odds with the very presumption of a conservative norm that Amodio has just asserted. Indeed, it would become one of numerous independent reasons to question the use of college students to profile liberals and conservatives generally. The jury is still decidedly still out on whether the specified "associations between political orientation and cognitive styles have been shown to be heritable, evident in early childhood, and relatively stable across the lifespan" -- a statement which is, itself, considerably broader and, I daresay, more categorical than the supporting studies Amodio cites in that regard can be claimed to suggest.
Amodio states, "The sample was actually rather large for a neuro study." Although most "neuro studies" are also scanning subjects at rest, not recording ERP's, size is only really problematic here because of the study's ambitions with regard to neurological political profiling. Fatal sampling flaws kick in before total size can even begin to matter. For methodological contrast, here's another Error Monitoring based study. In terms of relevance, I would note that ADHD can also be construed as a continuum, vs. a discreet category too.
In defiance of the way he, himself, has framed this study, Amodio says that they did not conduct group analyses because: "Group analyses would have been psychometrically problematic, and furthermore, we didn’t want to suggest that political orientation is categorical." Problematic? No joke. If you look at conservatives as a group here, you would almost think that greater conservatism is also "associated with greater conflict-related ACC activity" than "lesser" conservatism too! Given the limited number of conservatives, I'd certainly stipulate that it's hard to make the case that they actually constitute a group -- but that also weakens the case for inferring that they're representative of conservatives generally. It also makes it impossible to assess whether the two lowest scoring conservative subjects might actually be outliers or not, even though the other 5 have more than a little company on the liberal side of Amodio's vaguely defined divide.
There's plenty more to critique here, but what's really astonishing is that having acknowledged that there are, indeed, probably "3rd variables at play" here, Amodio proceeds to call an interpretation which completely ignores such probabilities altogether not just "plausible" but essentially immune to criticism on that basis. The question, of course, is not whether others can offer up competing interpretations, but whether there's enough "there" there to interpret in the first place.
Maybe conservatives just don't sweat the small stuff. Or if we want to dress it up a bit: perhaps conservative ACCs are less inclined to high intensity conflict montioring when there's no moral dilemma involved or no particular outcome-based reward for error free performance, while in contrast, liberals just personalize everything. Perhaps liberals are best suited for athletics where superior hand eye coordination is a must, while conservatives are better suited for intellectual pursuits. In fact, however, it takes a huge leap to get from Botvinick and the primitive state of the relatively recent science here to the kind of sweeping generalizations from which it appears this study was born, per Amodio's intro, and toward which it is clearly headed. The proverbial blind man with an elephant springs to mind -- in more ways than one.
Coming from someone ostensibly researching political psychology, the idea that "people are just cranky because this bullet-point of a study has been so over-sensationalized" stikes me as disingenuously obtuse -- especially when, by his own lights, he should have been expecting a "fixed response style" from all the more conservative than average among us. By the time I hit his citation of Adorno's "Authoritarian Personality," I was hard pressed not to exit laughing. Conservative authoritarianism is a virtual idée fixe on the left.
Many thanks, btw, for the timely reminder on contacting Congress about public access to NIH-funded research in response to my earlier comment about the $30 threshhold to what was, alas, a very slim document.
Congratulations, you won a Thinking Blogger Award
ReplyDeleteThe participation rules are simple:
1. If, and only if, you get tagged, write a post with links to 5 blogs that make you think,
2. Link to this post so that people can easily find the exact origin of the meme,
Please, remember to tag blogs with real merits, i.e. relative content, and above all - blogs that really get you thinking!
This all started at:
http://www.thethinkingblog.com/2007/02/thinking-blogger-awards_11.html
The post in which I tagged you can be found at:
http://psychcentral.com/blog/archives/2007/09/21/thinking-blogger-award-and-our-top-five-thinking-blogs/
This is some of the worst research I have ever seen! I'm glad my kid doesn't go to NYU.
ReplyDeleteThis issue brings up a number of titilating research directions in whch to take these results.
ReplyDelete1. The most important thing here is the attempt to associate behavior with neurophysiology. Regardless, of the actual conclusions reached by this study, if [eventually] destructive behaviors are linked to brain structures and genetic determinations of them, we can undertake to discuss destructive "people" impersonally, rather than fearfully or confrontationally, and reduce the effectiveness of manipulation by "structured" individuals.
2. Assuming Dr. Amodio's findings demonstrate fact *for the sake of argument* (at least), it would be logical to expect opposition (as opposed to rational contradiction) from "structured" individuals to the extent that such opposition could be taken to prove the hypothesis. Therefore, the psychology of persons expressing "rigid" antithetical comments needs to be determined in order to interpret those comments. "Rigid" comments could not be taken at face value. Science itself, being an inherently "analytical" activity (and capable of exposing "structured behaviors") would necessarily be opposed by "structured" persons as well.
3. It would be practical to expand the study to include persons expressing "psychopathic" traits (Hare Checklist) and to determine whether any of the traits observed in the experiment subjects match the traits on the Hare Checklist, in order to relate the "structured" mentality to the "psychopathic" mentality.
4. IMHO, all historical examples of destructive behaviors (e.g. Hitler has been mentioned, "terrorists" have to be so categorized) have resulted from aberrant brain structures in perhaps 20% of populations, with extremes (Psychopathy per se) in 1-3% of populations. It would be practical to work toward developing neurophysiological explanations of historical human aberrations (e.g. see the work "Political ponerology"). Start by correlating the similarities between historical examples and experimental results.
5. In the same regard, it would be highly desirable to make lists of all potentially "structured" and anti-analytical human behaviors, starting with analysis of political discourse, since manipulative "structured" individuals in positions of power can be highly detrimental to society. Religion might get on that list, of course, but I have some ideas as to how it can be rescued.
6.I have proposed that, thanks to the theory of "neuroplasticity", the brains of "structured" persons and be influenced to become analytical (within the necessary parameters of intelligence). I would do this by requiring all secondary school students to pass a course in "critical thinking" that includes explication of "rigidity", "bullying" and psychopathy. This theory of resolving "rigidity" could turn out to be self-proving, in that opposition to it could be categorized as a form or defense of "rigidity" that is destructive.
7. Needless to say, I also advocate unlimited collaboration between all disciplines (and experts in those disciplines) capable of examining the negative aspects of human behavior and correlating them with causes, solutions, and historical consistency: some are obvious (neurophysiologists, psychologists, sociologists), some are not (discourse analysts, data miners, educators, etc.) First, make a full list of all possibilities and a directory of all persons.
John G. Marr
jmarr@flash.net