Saturday, March 18, 2006

Anonymous Peer Review Means Never Having to Say You're Sorry

The Neurocritic has been feeling reproved this past week, so it's time to post an oldie but goodie about the arbitrary nature of peer review, published in Brain (full-text available free).

Might as well take up gambling.
Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain. 2000 123:1964-9.

We aimed to determine the reproducibility of assessments made by independent reviewers of papers submitted for publication to clinical neuroscience journals and abstracts submitted for presentation at clinical neuroscience conferences. We studied two journals in which manuscripts were routinely assessed by two reviewers, and two conferences in which abstracts were routinely scored by multiple reviewers. Agreement between the reviewers as to whether manuscripts should be accepted, revised or rejected was not significantly greater than that expected by chance [kappa = 0.08, 95% confidence interval (CI) -0.04 to -0.20] for 179 consecutive papers submitted to Journal A, and was poor (kappa = 0.28, 0.12 to 0. 40) for 116 papers submitted to Journal B. However, editors were very much more likely to publish papers when both reviewers recommended acceptance than when they disagreed or recommended rejection (Journal A, odds ratio = 73, 95% CI = 27 to 200; Journal B, 51, 17 to 155). There was little or no agreement between the reviewers as to the priority (low, medium, or high) for publication (Journal A, kappa = -0.12, 95% CI -0.30 to -0.11; Journal B, kappa = 0.27, 0.01 to 0.53). Abstracts submitted for presentation at the conferences were given a score of 1 (poor) to 6 (excellent) by multiple independent reviewers. For each conference, analysis of variance of the scores given to abstracts revealed that differences between individual abstracts accounted for only 10-20% of the total variance of the scores. Thus, although recommendations made by reviewers have considerable influence on the fate of both papers submitted to journals and abstracts submitted to conferences, agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone.
Or one could take up lobbying for open peer review. The new journal, Biology Direct, has already done so.
Koonin EV, Landweber LF, Lipman DJ. A community experiment with fully open and published peer review. Biol Direct. 2006 Jan 31;1(1):1.

. . .

The general view of the current system of peer review of scientific work boils down, more or less, to the tired Churchill quote on democracy: it is the worst system imaginable except for all others. Yet, all publishing scientists are painfully aware of the growing problems of the peer-review system. The crucial feature of the present peer review approach is that it is, predominantly, anonymous, and a reviewer, safe behind the veil of anonymity, has virtually absolute power over the helpless author. And absolute power we know only too well can corrupt absolutely. Hence the biased reviews, the inexplicably delayed perfunctory reviews, the pedantic reviews making a huge deal of minor quibbles, and other kinds of unfair and upsetting reviews that we all dread reading but, unfortunately, may even find ourselves writing. Of course, it is not some sinister villains who produce these obnoxious reviews, it is we peers, we distinguished members of the active scientific community. Indeed it would not be much of an exaggeration to state that all of us, on one occasion and another, have been on the receiving end of peer review abuses, and (almost) any frequent referee has probably been an abuser as well, whether accidentally, intentionally, or at least in the eye of the author.
The Neurocritic is happy to provide a new form of anonymous peer review, free of charge.

Subscribe to Post Comments [Atom]

Thursday, March 16, 2006

I Predict . . . Mindreading in the News

Note the difference below between the BBC headline and the next couple of sentences:
Scientists 'can predict memories'

Scientists say it may be possible to predict how well we will remember something before the event has even taken place.

By analysing scans, they discovered the brain must get into the 'right frame of mind' to store new information.

For top performance, the brain must mobilise its resources, not only at the moment we get new information, but also in the seconds before.
It's not that the scientists can actually read your mind and predict what you'll remember, but that they can predict how well you'll remember something. The paper is available online at Nature Neuroscience.

The authors (Otten, Quayle, Akram, Ditewig & Rugg) presented the participants with a list of words to memorize for a later memory test. Each word was preceded by a cue that told subjects either what task to perform with the word -- an orthographic or a semantic decision (Exp. 1) -- or whether the word would be presented in the auditory or visual modality (Exp. 2). EEG activity was recorded while the participants performed these experiments, and event-related brain potentials ("brain waves") were averaged separately for the cue words and the memorized words. The major new finding is that a late-onset, long-duration negative brain wave (akin to the CNV, or Contingent Negative Variation wave recorded in the interval between cue and target in S1-S2 tasks) was observed to cues preceding words that were later remembered, but not to cues preceding words that were later forgotten. Otten et al. didn't acknowledge that their negative wave resembled the CNV or related brain waves, and they discounted the explanation that greater attentional resources were marshalled to the cues preceding remembered words, instead arguing that the neural activity reflects the "adoption of a 'task set' optimal for semantically oriented processing." Hmm, that's not a very satisfying explanation, but it does account for the fact that they failed to observe the "prediction wave" for cues signaling that subjects should perform the orthographic task. What is not accounted for, however, is that the "prediction wave" was not seen to cues preceding auditory words that were later remembered in Exp. 2. But why?

SUMMARY from The Neurocritic: kinda interesting, but how did it get published in Nature Neuroscience??

ADDENDUM: The supplementary material for the paper indicates that the memory test was a surprise part of the experiment, so the participants weren't deliberately memorizing the words, merely performing the orthographic and semantic tasks blissfully unaware of the impending exam. SO the purpose here was to predict memory performance even when subjecs weren't trying to remember.

Subscribe to Post Comments [Atom]

Monday, March 13, 2006

More on Humor and Hot Flashes

Someone might [did] ask, "Why is it problematic to find that some of the neural correlates of X are the same as the neural correlates of a seemingly unrelated ability/state Y?"

Fair question.

It could be that the neural correlates of humor, hot flashes, and empathy are all the same. This would not be a problem for some cartographers of cognition (and emotion) who don't really care about phenomenology (i.e., laughing at a joke seems to be a different experience than "feeling someone else's pain" or having a hot flash, but some circuit in the brain can't distinguish between these experiences). In my opinion, this sort of result would mean that the methodology is entirely unsatisfactory in explaining mental states, if it turned out to be the case that statistically indistinguishable brain states were associated with watching Dave Chappelle vs. watching Bob Flanagan, Supermasochist. Of course, one might hope that this would be due to the crude level of analysis permitted by fMRI.

OR it could be that the similarly-located BOLD activations observed in fMRI studies of humor, hot flashes, and empathy for pain are driven by some underlying commonality (let's say an increase in autonomic functions such as heart rate, blood pressure, and sweating). There is some evidence that the brain areas mentioned below are, in fact, sensitive to increases in arousal and autonomic activation (see the post "More Lies... Damn Lies..."). If the degree of activation in the anterior cingulate cortex and frontoinsular cortex are correlated with autonomic activation, one would of course expect hot flashes to be the worst [NOTE: the scanner environment would preclude humor that induces guffawing and belly-laughing]. Anyway, one would need to test peri-menopausal women on humor and empathy for that comparison...

Subscribe to Post Comments [Atom]

Sunday, March 05, 2006

Humor, Hot Flashes, and Empathy for Pain

These three phenomena activate the same brain areas (anterior cingulate cortex and frontoinsular cortex), according to recent findings. Any theory about the neural correlates of empathy must take into account the fact that the same brain regions are activated during menopausal hot flashes and the appreciation of humor.

Freedman RR, Benton MD, Genik RJ 2nd, Graydon FX.
Cortical activation during menopausal hot flashes.
Fertil Steril. 2006 Mar;85(3):674-8.

OBJECTIVE: To determine regions of brain activation associated with menopausal hot flashes and sweating. DESIGN: Controlled laboratory study. SETTING: University medical center. PATIENT(S): Symptomatic postmenopausal women and asymptomatic eumenorrheic women. INTERVENTION(S): None. MAIN OUTCOME MEASURE(S): Brain activation measured by functional magnetic resonance imaging. RESULT(S): Significant (P<.001) areas of activation during hot flashes in symptomatic women included the insula and anterior cingulate cortex. Sweating in the eumenorrheic women was associated (P<.001) with activity in the anterior cingulate and superior frontal gyrus. CONCLUSION(S): Activation of the insular cortex is associated with the "rush of heat" described during menopausal hot flashes. Thermoregulation in humans appears to be represented in a distributed cortico-subcortical network rather than in a single localized structure.

Watson KK, Matthews BJ, Allman JM.
Brain Activation during Sight Gags and Language-Dependent Humor.
Cereb Cortex. 2006 Mar 2; [Epub ahead of print]

Humor is a hallmark of human discourse. People use it to relieve stress and to facilitate social bonding, as well as for pure enjoyment in the absence of any apparent adaptive value. Although recent studies have revealed that humor acts as an intrinsic reward, which explains why people actively seek to experience and create humor, few have addressed the cognitive aspects of humor. We used event-related functional magnetic resonance imaging to differentiate brain activity induced by the hedonic similarities and cognitive differences inherent in 2 kinds of humor: visual humor (sight gags) and language-based humor. Our findings indicate that the brain networks recruited during a humorous experience differ according to the type of humor being processed, with high-level visual areas activated during visual humor and classic language areas activated during language-dependent humor. Our results additionally highlight a common network activated by both types of humor that includes the amygdalar and midbrain regions, which presumably reflect the euphoric component of humor. Furthermore, we found that humor activates anterior cingulate cortex and frontoinsular cortex, 2 regions in the brain that are known to have phylogenetically recent neuronal circuitry. These results suggest that humor may have coevolved with another cognitive specialization of the great apes and humans: the ability to navigate through a shifting and complex social space.

Saarela MV, Hlushchuk Y, Williams AC, Schurmann M, Kalso E, Hari R.
The Compassionate Brain: Humans Detect Intensity of Pain from Another's Face.
Cereb Cortex. 2006 Feb 22; [Epub ahead of print]

Understanding another person's experience draws on "mirroring systems," brain circuitries shared by the subject's own actions/feelings and by similar states observed in others. Lately, also the experience of pain has been shown to activate partly the same brain areas in the subjects' own and in the observer's brain. Recent studies show remarkable overlap between brain areas activated when a subject undergoes painful sensory stimulation and when he/she observes others suffering from pain. Using functional magnetic resonance imaging, we show that not only the presence of pain but also the intensity of the observed pain is encoded in the observer's brain--as occurs during the observer's own pain experience. When subjects observed pain from the faces of chronic pain patients, activations in bilateral anterior insula (AI), left anterior cingulate cortex, and left inferior parietal lobe in the observer's brain correlated with their estimates of the intensity of observed pain. Furthermore, the strengths of activation in the left AI and left inferior frontal gyrus during observation of intensified pain correlated with subjects' self-rated empathy. These findings imply that the intersubjective representation of pain in the human brain is more detailed than has been previously thought.

Subscribe to Post Comments [Atom]

Friday, March 03, 2006

The Media? Sensationalistic?

This article appeared on the BBC news site. Finally, a story about the sensationistic coverage of scientific findings in the popular press.

Media 'sensationalising science'

A report by the Social Market Foundation (SMF), an independent research group, has accused the UK media of sensationalising science.

It says irresponsible reporting can undermine public confidence in science and government, and on issues like vaccination may even cost lives.

Here's my favorite part of the article:

Claudia Wood of the SMF said journalists tend to seek black and white stories and look for certainties which cannot be provided by science.

"The media has to be very aware that what it says can have huge impacts on the public's behaviour," she told the BBC.

"I think the media has to be very cautious in how it gives over scientific evidence, and has to make sure that people understand that there are certain risks to some things but a lot of the time evidence isn't conclusive."

The Neurocritic is of the opinion that the desire to get one's research published in high-profile journals and the popular press can drive many (formerly open-minded) scientists to write pathetically simplistic papers that completely ignore any findings in the literature that may disagree with their own results. Competition between editors at the various high-profile journals only feeds this compulsion for the black & white result that'll look good in the NY Times, when the reality is a shade of gray. Gray doesn't sell, however.

Subscribe to Post Comments [Atom]

eXTReMe Tracker