tag:blogger.com,1999:blog-21605329.post2690456233569081624..comments2024-03-14T23:52:09.893-07:00Comments on The Neurocritic: The Idiosyncratic Side of Diagnosis by Brain Scan and Machine LearningThe Neurocritichttp://www.blogger.com/profile/08010555869208208621noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-21605329.post-55377418073736125112015-08-04T10:32:28.619-07:002015-08-04T10:32:28.619-07:00I generally agree, 10-Fold-CV should be used inste...I generally agree, 10-Fold-CV should be used instead of LOOCV, but due to the small size of the data set, I doubt that the results would look much different from LOOCV (they should have at least reported the SD). So I don't think this is a reason to dismiss their results, especially not by suggesting that they are not "reality check[ing]" their system - they are, they could just be doing a better job. (I don't expect brain imaging people to do a good job when it comes to statistics.)<br /><br />I don't know about the particular choice of emotions, that wasn't the point I criticized (not my field of study or interest). It sounds plausible that the bio marker itself is debatable, I just didn't like the way you presented it.<br /><br /><br />PS: You should really add disqus to your blog.Anonymousenoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-89519765417912274422015-08-03T13:11:16.530-07:002015-08-03T13:11:16.530-07:00Here's an illustration of the important differ...Here's an illustration of the important differences between guilt and shame:<br /><br /><b>Anonymouse:</b> "<i>I also don't understand why you summarize the procedure for their bio marker in a way that makes it sound complicated and questionable without concretely criticizing anything about it. This is just unprofessional. And I don't think what they describe is very complicated.<br /><br />This blog is generally a fun read, but this post has been disappointing.</i>"<br /><br /><b>Guilt reaction:</b> "<i>Huh, maybe I should have read more articles and explained the problem with the biomarker. Perhaps I can do that in a future post.</i>"<br /><br />If I have wronged the authors, then I can correct this mistake and give them a more thorough reading. Guilt can be adaptive (one can correct the transgression).<br /><br /><b>Shame reaction:</b> "OMG, I'm unprofessional and I've disappointed a reader. I'm a failure!"<br /><br />I have wronged the authors and the readers, therefore I'm a terrible person unworthy of love. Just more proof that one is bad person. This reaction is much more strongly related to depression than guilt (Tangney et al., 1992).<br /><br /><br />The participants made these types of guilt, anger, shame, etc. judgments on the study stimuli (e.g., "I acted stingily towards my best friend") <b>after</b> the scan was over. So we don't really know what the hemodynamic response was during this sort of evaluation. During the scan, subjects rated the intensity of their feeling (“extremely unpleasant” or “mildly unpleasant”), but did not rate <b>the specific emotion</b> they felt. This makes the biomarker further removed from actual guilt feelings, because the participants were busy making a decision about their generic unpleasantness level.<br /><br />If I'm understanding everything correctly, I believe this is a more specific critique...The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-29091489363031737932015-08-03T10:35:09.813-07:002015-08-03T10:35:09.813-07:00Anonymouse - Thanks for your comment. Others who a...Anonymouse - Thanks for your comment. Others who are much more expert in this realm (<a href="https://twitter.com/NeuroStats/status/627915828650864640" rel="nofollow">@NeuroStats</a>) have pointed to problems with the leave-one-out procedure and linked to this paper (<a href="http://frostiebek.free.fr/docs/Machine%20Learning/validation-1.pdf" rel="nofollow">PDF</a>): <br /><br /><i>A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection</i> (1995), by Ron Kohavi.<br /><br />As well: <a href="https://twitter.com/NeuroStats/status/627899777007534080" rel="nofollow">remarked</a>: "Also leave one out cross validation should be abandoned in favor of 5-fold or 10-fold CV."<br /><br /><a href="https://twitter.com/NeuroStats/status/627900270794506240" rel="nofollow">and</a>: "The ML community realized this 2 decades ago. Test set size needs to be representative of overall heterogeniety"<br /><br />I didn't engage in an in-depth critique of the biomarker itself because that was beyond the scope of the post. I didn't have time to read 3-4 other papers to determine how and why they chose a contrast of "guilt" vs. "indignation/anger towards others" from this long list of possible answers: <i>guilt, contempt/disgust towards self, shame, indignation/anger towards self, indignation/anger towards others, contempt/disgust towards others, none, other feeling</i>.<br /><br />Why guilt and not <a href="http://www.ncbi.nlm.nih.gov/pubmed/1500604" rel="nofollow">shame</a>, which is the <a href="http://www.ted.com/talks/brene_brown_listening_to_shame?language=en" rel="nofollow">more toxic emotion</a>, according to many researchers and clinicians? <br /><br />"<i>Shame-proneness was strongly related to psychological maladjustment in general. Guilt-proneness was only moderately related to psychopathology; correlations were ascribable entirely to the shared variance between shame and guilt. Although clearly related to a depressogenic attributional style, shame accounted for substantial variance in depression, above and beyond attributional style.</i>" (Tangney et al., 1992)<br /><br />To be honest, I wanted to write about one of the earlier papers by these authors over 2 yrs ago (and at subsequent times since), but found that an adequate review of the clinical and neuroimaging literatures was overwhelming. Other fMRI researchers have not found the same results for guilt, <a href="http://www.ncbi.nlm.nih.gov/pubmed/23051901" rel="nofollow">for instance</a>.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-11445766114660930672015-08-03T09:35:20.581-07:002015-08-03T09:35:20.581-07:00Training a simple classifier like LDA for nearly 5...Training a simple classifier like LDA for nearly 50 data points isn't nonsense - why would it be? Yes, overfitting. That's what the cross-validation is for. Also they ran the classifier without the guilt vs. indignation marker and the performance dropped dramatically, which doesn't really speak for mere overfitting either.<br /><br />Cross-validation is good and standard practice for small data sets, so I don't understand your point about not having tested an "independent group". What does that mean? Do you mean that the same participants have been imaged in another study before? In that case I agree that this might introduce a bias, but for exploratory research this seems perfectly reasonable, especially as they say they are carrying out a validation study right now (with other participants, I'm sure).<br /><br />I also don't understand why you summarize the procedure for their bio marker in a way that makes it sound complicated and questionable without concretely criticizing anything about it. This is just unprofessional. And I don't think what they describe is very complicated.<br /><br />This blog is generally a fun read, but this post has been disappointing.Anonymousenoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-59479060413078188592015-08-02T02:18:35.995-07:002015-08-02T02:18:35.995-07:00How many features does each data point have? Surel...How many features does each data point have? Surely it's a nonsense to train a classifier on such a small data set if they are using more than a couple of features even before the failure to use a test data set?Richardhttps://www.blogger.com/profile/14539175222290571773noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-87374374831757816602015-08-01T20:20:40.913-07:002015-08-01T20:20:40.913-07:00Thanks for that link.
I'm not sure how some ...Thanks for that link. <br /><br />I'm not sure how some of these articles can call the results "accurate" or say they have "a high degree of accuracy."<br /><br />Next time, I'll raise the point that people working in this field should all try out everyone else's algorithms on their own datasets. But the <i>“choose one feeling that (they) would feel most strongly” from the following list: guilt, contempt/disgust towards self, shame, indignation/anger towards self, indignation/anger towards others, contempt/disgust towards others, none, other feeling</i> task is rather limiting...The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-91983496043648510852015-08-01T19:36:19.615-07:002015-08-01T19:36:19.615-07:00Yikes on that sens/spec. But also, am reminded o...Yikes on that sens/spec. But also, am reminded of my usual shoutout to https://www.psych.umn.edu/faculty/grove/112clinicalversusstatisticalprediction.pdf <br /><br />Over time, I suspect clinical prediction and classification will be supplanted by algorithm, although currently our rates are better.Tannahill Glennoreply@blogger.com