In The Neurocinema Collection™, we established that the term "neurocinema" doesn't really refer to movies with "Neuro" in the title or films about neurology. Instead, neuromarketers have have used the word in the following fashion:
Neurocinema is a new filmmaking process that studies a viewer's sensorimotor, cognitive, and affective response to film stimuli. Researchers use technologies such as functional magnetic resonance imaging (fMRI) to measure changes in activity in parts of the brain,electroencephalography (EEG) to measure activity in specific regional spectra of the brain response, and/or sensors to measure changes in one's physiological state (heart rate, respiratory rate, galvanic skin response) to learn exactly what scenes excite or disinterest the viewer.There have been no peer-review studies on this methodology, only articles in the popular press (e.g., Wired.com and CNN.com). And from what was available in those reports, there was plenty to be neurocritical about.
On the other hand, Uri Hasson and his NYU colleagues published an article entitled Neurocinematics: The Neuroscience of Film in Projections: The Journal for Movies and Mind. Granted, that article is lacking the methodological details one would see in NeuroImage or Human Brain Mapping, but in fact the authors have published their work in prestigious journals (like Science and Neuron) without the "Neurocinematics" title.
In their Projections article, Hasson et al. (2008) state:
We propose that ISC [inter-subject correlation analysis] may be useful to film studies by providing a quantitative neuroscientific assessment of the impact of different styles of filmmaking on viewers’ brains, and a valuable method for the film industry to better assess its products. Finally, we suggest that this method brings together two separate and largely unrelated disciplines, cognitive neuroscience and film studies, and may open the way for a new interdisciplinary field of “neurocinematic” studies.
Hasson and colleagues first described the ISC method in their 2004 paper. In that study five subjects watched 30 min of The Good, The Bad, and The Ugly while laying supine in an MRI scanner. Unlike a typical fMRI experiment involving controlled presentation of individual pictures and performance of a categorization task, free viewing of a dynamic film is a more naturalistic situation. It also presents a more challenging data analysis problem. To tackle this, the researchers eschewed standard hypothesis-testing and adopted a purely data-driven approach to determine commonalities in brain activation across subjects.1
Figure 1B (Hasson et al., 2008). The ISC analysis measures similarity in brain activity across viewers by comparing the response time course in each brain region from one viewer with the response time courses obtained in the same brain region from other viewers during movie watching.
The results (Hasson et al., 2004) demonstrated that...
Despite the free viewing and complex nature of the movie, we found an extensive and highly significant correlation across individuals watching the same movie. Thus, on average over 29% ± 10 SD of the cortical surface showed a highly significant intersubject correlation during the movie..... Close inspection of this across-subject correlation revealed that the synchronization was far more extensive than the boundaries of well-known audiovisual sensory cortex defined with conventional mapping approach. ...the across-subject correlation covered most of the visual system, including early retinotopic areas as well as high-order object areas within the occipitotemporal and intraparietal cortex. Moreover, the correlation extended far beyond the visual and auditory cortices to the entire superior temporal (STS) and lateral sulcus (LS), retrosplenial gyrus, even secondary somatosensory regions in the postcentral sulcus, as well as multimodal areas in the inferior frontal gyrus and parts of the limbic system in the cingulate gyrus.When corrected for non-selective elements, the intersubject correlation remained at 24% ± 8.5. The other 75% of the brain not "in sync" across subjects included parietal lobe areas (supramarginal and angular gyri) and the bulk of prefrontal cortex. Subcortical areas important for emotion and memory (amygdala and hippocampus) were not examined.
A second analysis was also performed, which took a "reverse-correlation" approach to examine activity in two well-know category-specific areas: the fusiform face area (for faces) and the parahippocampal place area (for buildings). This type of functional specialization was demonstrated in an earlier paper as well (Bartels & Zeki, 2004). In that study, the participants watched a James Bond movie (Tomorrow Never Dies). Activity in category-specific regions (for color, faces, language, and human bodies) was correlated with the perceived intensity of these specific attributes (as rated by a separate group of subjects who watched the same movie).
The most fascinating result in my opinion was obtained in subsequent experiments (described in Hasson et al., 2008) that compared the ISC for The Good, The Bad, and The Ugly to those observed for an episode of the TV series Alfred Hitchcock Presents (which is on Hulu!!!), an episode of the semi-improvised comedy Curb Your Enthusiasm, and a real-life unedited video shot in Washington Square Park. Not surprisingly,
The extent of ISC differed for the four movies (Figure 7A). The percentage of cortex exhibiting high ISC provided a measure of the overall effectiveness, or collective engagement power, of each movie to induce similar responses across viewers (Figure 7B). The Hitchcock episode (green) evoked similar responses across all viewers in over 65 percent of the cortex, indicating a high level of control of this particular episode on viewers’ minds. The high ISC was also extensive (45%) for the Good, the Bad and the Ugly (blue), but much less so (18%) for Curb Your Enthusiasm (red). Finally... the unstructured segment of reality (orange) induced high ISC only in a small fraction of the cortex (less than 5%).
Figure 7 (Hasson et al., 2008).
Glossing over the technical details [I didn't read the 5 other cited papers] and discrepancies between the 2004 and 2008 studies [divergence in ISC values of 25% vs. 45% for the same film], the implication is that different directors, and different filmmaking styles, exert varying levels of "control" over audio-visuo-higher-level cortical responses in the brains of the viewers:
The fact that Hitchcock was able to orchestrate the responses of so many different brain regions (Figure 7, green), turning them on and off at the same time across all viewers, may provide neuroscientific evidence for his notoriously famous ability to master and manipulate viewers’ minds. Hitchcock often liked to tell interviewers that for him “creation is based on an exact science of audience reactions” (Douchet 1985).Different filmmakers strive to achieve different levels of control over their viewers’ reactions. Our findings provide empirical evidence to support the long-lasting distinction in film theory between films that remain faithful as much as possible to reality and those that seek to control or distort it.The amount of control employed by filmmakers exists on continuum that ranges from real life --> documentaries --> art films --> Hollywood --> propaganda. The authors conclude with a discussion of collective engagement in film and suggest that it can be measured in an "objective" fashion using ISC. Overall, the 2008 Projections paper was a BOLD proposal for a true merger of art and science in academia.
Neurocinema, Neurocinematics? A semantic distinction perhaps, but the terms first appeared in different contexts. The former was used to describe an unvalidated filmmaking "process" [and underdeveloped commercial application] aimed squarely at attracting Hollywood megabucks, whereas the latter was used in a scholarly publication to describe a new field of inquiry. Will a merger (of sorts) be far behind? Or more to the point, has the NYU group applied for a patent yet?
Footnote
1 The methods for this were pretty brief and largely contained within Footnote 15 of the paper:
15. In order to search for correlation between two corresponding regions across subjects (intersubject correlation), we first aligned all brains into Talairach coordinate system and used a Gaussian filter of 12 mm full width at half maximum value (FWHM) to the data... To remove preprocessing artifacts, we excluded the first and last 10 time points of the experiment from the analysis. We then used the time course of each voxel of the source subject as a GLM predictor for modeling the activity in the corresponding voxel of the target subject. Only voxels whose P value was no more than 0.05 (corrected) were considered significant.References
Bartels A, Zeki S. (2004). Functional brain mapping during free viewing of natural scenes. Hum Brain Mapp. 21:75-85.
Hasson U, Landesman O, Knappmeyer B, Vallines I, Rubin N, Heeger DJ. (2008). Neurocinematics: The Neuroscience of Film. Projections 2:1-26. [PDF]
Hasson, U. et al. (2004). Intersubject Synchronization of Cortical Activity During Natural Vision. Science, 303 (5664), 1634-1640. DOI: 10.1126/science.1089506
Further Reading
Neurocinematics - The Valve - A Literary OrganNeurocinematics - Neurophilosophy blog
Neurocinematics! Where Neuroscience Meets Filmmaking - NYU panel discussion held May 4, 2009
When I read that 2004 Science paper I really did not get why it was published there (other than the fact that the senior author frequently manages to publish in these journals, for whatever reason). How much of the intersubject correlation in brain activity can be explained by common eye movements driven by the movie? I think there is a trivial explanation. Good movies will make you look where the movie maker wants you to look. Bad movies (or clips of random people walking) won't. I'm not sure what brain imaging contributes. Perhaps all one needs is an eye tracker. Have these findings been replicated by another lab with the variance due to common eye movements covaried out somehow?
ReplyDeleteI tried to find a published critique of the 2004 paper and its methodology but I couldn't. I agree that differential eye movements could play a role in the patterns of cortical activity, but they can't account for everything (e.g. auditory cortex). The researchers also recorded from single neurons in auditory cortex of epilepsy patients and correlated that with BOLD activity in healthy subjects during the movie (Mukamel et al. 2005). Lots of gunshots and explosions in The Good, The Bad, and The Ugly.
ReplyDeletePlus you'd expect to see very high ISC in the frontal eye fields. I didn't have the time or the stamina to read the other 5-6 papers for details, but in Projections they say:
"...we measured concurrently eye movements and brain activity to movies (with no sound track) played forward and backward in time (Hasson, Yang et al. 2008). The eye movements were highly correlated across viewers and very similar across repeated presentations of the same movie for both the forward and backward movies (see figure 4 in Hasson, Yang et al. 2008). The brain activity in visual cortex was, likewise, highly correlated for both the forward and backward films. But the correlations in brain activity in some other cortical areas (precuneus, LS, TPJ, and FEF) were much smaller during backward than during forward presentations (see figure 2 in Hasson, Yang et al. 2008). The similarity of the eye movements for both forward and backward films suggests a comparable level of engagement, mitigating potential concerns that the lower correlations in brain activity during the backward movies occurred because viewers paid less attention to them."
Thanks NC. Let's focus on vision for a moment. Let's say we find that "FFA" activation is correlated across individuals (which it was). How would this be surprising if it was also found that all subjects were looking at the same locations with faces throughout the movie? Of course, if you foveate onto a face, you are going to get a spike in FFA. The really interesting phenomenon is not the brain imaging, is the fact that the movie maker was able to manipulate our attentional system and make everyone look consistently at certain spots.
ReplyDeleteRegarding the auditory cortex correlation, that is easy to explain because you cannot move your ears and select sounds. So, why shouldn't there be correlation across subjects, since everybody would hear the same soundtrack?
Again, after all these years and after listening to some of their talks, I have not been able to shake the sense that the brain imaging results in these studies are rather trivial. I may be wrong... :)
Yes, in a sense the ISC results form the basis for group studies in the first place: loud gunshots activate the auditory cortex in a similar fashion across subjects, and compelling images activate common areas in the visual processing stream. That much is trivial. Linking the degree of ISC to the level of directorial control and the style of filmmaking is the innovative aspect. Plus, in comparison to the outlandish clams in the Wired article (and results from some other neuromarketing studies), the techniques here seem ultra-sophisticated.
ReplyDeleteAnonymous's comments about blinking reminded me of the very interesting study of synchronized blinking by Nakano et al. This was recently mentioned by Vaughan over at Mind Hacks (http://bit.ly/141U2o) and I wrote about it on my blog today at Lucid Thoughts (http://bit.ly/GgdBc).
ReplyDeleteSometimes simpler, more elegant methods can get you closer to understanding what you're interested in than defaulting to the giant magnetic donut.
Steve - Thanks for linking to the blinking posts. The Radiolab podcast is interesting too, if you get a chance to listen.
ReplyDeleteThanks for the post. I wonder how much of the authors' conclusions can be attributed to other factors like age of the participant watching the movies, and/or number times that the subject(s) had seen the movie in the past. Some of the areas activated with Hitchcock, compared with CYE, are associative areas and thus more higher cognitive functions, memory recall, etc might account for the differences.
ReplyDelete