tag:blogger.com,1999:blog-21605329.post2493191737881495373..comments2024-03-22T00:30:09.536-07:00Comments on The Neurocritic: This Neuroimaging Method Has 100% Diagnostic Accuracy (or your money back)The Neurocritichttp://www.blogger.com/profile/08010555869208208621noreply@blogger.comBlogger15125tag:blogger.com,1999:blog-21605329.post-41180921501458103662016-01-28T19:00:50.402-08:002016-01-28T19:00:50.402-08:00I point to the red rectangles in Table 5, in the p...I point to the red rectangles in Table 5, in the post, where every ROI discrimination was 100%. The visual readings (which were a separate qualitative measure, the scans as read by a human being) were less accurate at discriminating patient groups from each other (but still surprisingly high).<br /><br />Not EVERY SINGLE ONE of Bansal et al.'s comparisons were 100%. That's all. Bansal et al. also went into much, much greater detail to describe every step of their analytic methods.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-11976457284515442802016-01-28T18:43:16.056-08:002016-01-28T18:43:16.056-08:00This is getting silly. I said "not every sing...This is getting silly. I said "not every single one is 100%." Here's the text:<br /><br /><i>"They discriminated the brains of children with ADHD from HC with 93.6% sensitivity and 88.5% specificity (Fig. 7, left); children with TS from children with ADHD with 99.83% sensitivity and 99.5% specificity (Fig. 7, right); adults with BD from HA with 100% sensitivity and 96.4% specificity (Fig. 8, 1st column); adults with SZ from adults with TS with 99.99% sensitivity and 100% specificity (Fig. 8, 2nd column); adults with SZ from adults with BD with 99.99% sensitivity and 100% specificity (Fig. 8, 3rd column); adults with SZ from healthy adults with 93.1% sensitivity and 94.5% specificity (Fig. 8, 4th column); adults with TS from HA with 83.2% sensitivity and 90% specificity (Fig. 9, left); children with TS from HC with 94.6% sensitivity and 79% specificity (Fig. 9, right); and participants at HR for depression from those at LR for depression with 81% sensitivity and 71% specificity (Fig. 10)."</i><br /><br />Higher than anything prior to it? Yes. Shockingly high? Of course. My point was that not all discriminations had 100% sensitivity and 100% specificity.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-34708651869738423342016-01-28T18:40:50.100-08:002016-01-28T18:40:50.100-08:00A couple more observations regarding the Bansal vs...A couple more observations regarding the Bansal vs Amen papers:<br /><br />If the best performance of Amen's method were actually the same as the best performance of Bansal's method (ie only 99.99% sensitivity and 100% specificity) then you would still expect Amen to report 100% sensitivity. That's because Amen used ~100 subjects for their Group 1 analysis, and with 99.99% sensitivity you would expect zero errors. Actually, Amen's sensitivity could be ten times worse than Bansal's (ie 99.90% sensitivity) and you would still expect zero errors with their sample size. It takes at least 10000 subjects before you should start expecting errors at sensitivity = 99.99%.<br /><br />Secondly, Amen reported three comparisons with perfect accuracy: Group 1 TBI vs CTL, Group 1 TBI/PTSD vs CTL, and Group 1 PTSD vs CTL. They also reported other comparisons with lower accuracy.<br /><br />Likewise, Bansal reported three comparisons with near-perfect accuracy (ie you would expect no errors in a sample with n ~ 100): pediatric Tourette's vs ADHD, adult schizophrenia vs Tourett's, and adult schizophrenia vs bipolar.<br /><br />So if one is willing to accept Bansal as "potentially huge", the results of Amen et al shouldn't stretch credulity.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-83197616917508076412016-01-28T18:11:36.431-08:002016-01-28T18:11:36.431-08:00Bansal reported sensitivity and specificity of 99....Bansal reported sensitivity and specificity of 99.99% and 100%. As far as I'm concerned, 99.99% is within margin of error of 100%. They are scientifically indistinguishable, at least in neuroimaging.<br /><br />But it sounds like the difference between Amen et al. and Bansal et al. is not 0.01%, it's the list of authors. That's fine, you don't have to trust what Amen says. But if you don't, then what's the point of reading the abstract?<br /><br />Just write "Amen published another paper today, which I didn't need to read because I don't trust his work". You've already made up your mind, so be willing to admit it. No need to pretend you want a scientific analysis of methodology when it really comes down to a judgment of character.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-5331077901406892322016-01-28T15:15:40.352-08:002016-01-28T15:15:40.352-08:00Oops, the first author's name is actually Bans...Oops, the first author's name is actually Bansal, not Bonsal...The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-29171374060258773422016-01-28T15:12:34.802-08:002016-01-28T15:12:34.802-08:00I'm glad you mentioned that Bonsal, Peterson e...I'm glad you mentioned that Bonsal, Peterson et al. paper, I was thinking about that one in particular. Yes, the classification accuracies were extraordinarily high, but still, not every single one was 100%. And at the time, there was a mixture of amazement and skepticism about it. <br /><br />This tweet by <a href="https://twitter.com/fnielsen/status/278060617137520640" rel="nofollow">@fnielsen</a> sums it up: "Twittersphere discussing high precision/recall rates for a neuroimaging classification study"<br /><br />Here's <a href="https://twitter.com/vaughanbell/status/277483910043164672" rel="nofollow">another</a>: "Accurate diagnosis of mental illness via MRI http://is.gd/gsTeeP Potentially huge, replication please"<br /><br />I tweeted about it myself, <a href="https://twitter.com/sarcastic_f/status/277706542529257472" rel="nofollow">asking</a> "Anyone read & evaluated that 21 page paper claiming Anatomical Brain Images Alone Can Accurately Diagnose Chronic Neuropsychiatric Illnesses" (there's a discussion that follows). <br /><br />I felt <a href="https://twitter.com/sarcastic_f/status/277708486052286464" rel="nofollow">underqualified</a>: "I'm afraid I don't know enough about Marching Cubes algorithms and Dirac delta functions to evaluate this paper myself" I don't know whether anyone ever blogged about it.<br /><br />The point is, people took the Bonsal et al. paper seriously. Given the <a href="https://www.washingtonpost.com/lifestyle/magazine/daniel-amen-is-the-most-popular-psychiatrist-in-america-to-most-researchers-and-scientists-thats-a-very-bad-thing/2012/08/07/467ed52c-c540-11e1-8c16-5080b717c13e_story.html" rel="nofollow">shady reputation</a> of the Amen Clinics, few academic researchers (to my knowledge, other than you) are taking the SPECT findings seriously.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-68739445256754909502016-01-28T14:14:28.468-08:002016-01-28T14:14:28.468-08:00A priori, precisely 100% is as improbable as preci...A priori, precisely 100% is as improbable as precisely 65%. We only attach emotional significance to the former.<br /><br />Has it happened before? Sure, for example:<br /><br />http://www.ncbi.nlm.nih.gov/pubmed/23236384<br />http://www.ncbi.nlm.nih.gov/pubmed/21839143<br /><br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-9031605016973314382016-01-28T09:41:50.615-08:002016-01-28T09:41:50.615-08:00Have you ever seen another neuroimaging metric tha...Have you ever seen another neuroimaging metric that is able to correctly classify 100% of patients? No matter how much overfitting and lack of generalizability? I haven't seen a perfect metric before, and I find this result statistically improbable.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-80456446369693241312016-01-28T03:09:37.444-08:002016-01-28T03:09:37.444-08:00This paper didn't make any claims regarding re...This paper didn't make any claims regarding replication, reliability, or even diagnosis (ie classification of unknowns).<br /><br />Let's be clear about what the authors claim: AFTER measuring SPECT values in patients and controls, they were able to construct a metric that completely separated the patients from controls. This is quite different from claiming that the same metric must work perfectly in a NEW group of patients with unknown diagnosis. The latter claim awaits validation from an independent sample. <br /><br />As an analogy, AFTER asking your family all their favorite movies, you might be able to isolate a movie that all of the females and none of the males in your family enjoyed. I bet I could, with my family. But that's quite different from claiming that movie preferences can perfectly determine a random person's gender. And it claims nothing about whether your family will have the same preferences next year.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-75007902797611285672016-01-28T01:03:04.149-08:002016-01-28T01:03:04.149-08:00Quick comment on statistics: how likely is it that...Quick comment on statistics: how likely is it that all four measures from the ROI analysis (sensitivity on-task, sensitivity at rest, selectivity on-task, selectivity at rest) would be 100% for six different group comparisons? Could another neurologist or psychiatrist replicate with 100% accuracy the classification of 397 participants into one of four groups (based Clinician 1's diagnoses)? In other words, how can SPECT exceed the <a href="http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2990547/" rel="nofollow">reliability of psychiatric diagnosis</a>?The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-76922567564265806272016-01-26T21:07:16.108-08:002016-01-26T21:07:16.108-08:00Scientific results should certainly be subject to ...Scientific results should certainly be subject to scrutiny. My concern is that you subject "amazing" studies to different scrutiny than the rest. If instead you compare this study to similar studies with less amazing results, I believe that you would find that the methodology is pretty similar in some respects, and better (sample size) in others. That's why I wondered whether you are roundly critical of neuroimaging, or biased against certain results. If the latter, keep in mind that one hundred percent accuracy is not unexpected once in a while. Statistically speaking, it would be unsettling if out of thousands of studies published yearly, *every single one* had FP>0 or FN>0.<br /><br />Regarding the relative value of imaging modalities:<br /><br />FDG-PET is clearly superior for some applications, such as characterization of dementia sub-types.<br /><br />MRI is clearly superior to FDG-PET for other applications, such as detection of acute stroke and neoplasm.<br /><br />CT is clearly superior to both MRI and FDG-PET for still other applications, such as detection of fracture, aneurysm, thrombosis, and sinusitis.<br /><br />And HMPAO-SPECT happens to be the imaging standard for establishing brain death when preparing for organ donation.<br /><br />In short, no imaging modality is a panacea, and each already has well-defined use cases. But we are talking about TBI and PTSD, where neither FDG-PET nor MRI have proven clinically effective, as supported by your own citations. And there is no reason to presuppose that they would be superior to HMPAO-SPECT, that is clearly an empirical question. <br /><br />In this case, it appears HMPAO-SPECT may the best suited modality for this indication. Time will tell if this is the case. Sadly it might take longer than we would like, and you and I both know why: academics rarely get paid to reproduce someone else's work.<br /><br />Finally, self-censorship out of concern for how your results will be represented by commercial interests IMHO is incompatible with the spirit of scientific inquiry. Record the numbers, report the numbers, and let the chips fall where they may.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-13287953273574247822016-01-26T14:48:04.839-08:002016-01-26T14:48:04.839-08:00The claim of 100% accuracy has huge diagnostic and...The claim of 100% accuracy has huge diagnostic and commercial implications (if true), so I'm not sure why you don't care about that. Or that patients might be mislead by dubious claims in advertising, as noted by less "smug" and more respectable commenters like <a href="http://www.ncbi.nlm.nih.gov/pubmed/19296729" rel="nofollow">Farah, 2009</a>; <a href="http://ajp.psychiatryonline.org/doi/abs/10.1176/appi.ajp.2010.10020157" rel="nofollow">Adinoff & Devous, 2010a</a>, <a href="http://ajp.psychiatryonline.org/doi/full/10.1176/appi.ajp.2010.10050671r" rel="nofollow">2010b</a>; <a href="http://www.tandfonline.com/doi/full/10.1080/21507740.2011.611123" rel="nofollow">Chancellor &, Chatterjee, 2011</a> and the <a href="http://www.nimh.nih.gov/about/director/2010/brain-scans-not-quite-ready-for-prime-time.shtml" rel="nofollow">former NIHM Director</a>.<br /><br />Further, would you not expect a claim of a "clinically relevant range" of sensitivity and specificity in a patient population of over 20,000 to exceed 82% and 60%, respectively? (see <a href="http://deevybee.blogspot.com/2010/07/difference-between-p-05-and-screening.html" rel="nofollow">BishopBlog</a>).<br /><br />This blog is called The Neurocritic. What you may consider "dampening people's enthusiasm for interesting results" I call "subjecting amazing new findings to scrutiny." But I'm not a complete naysayer on the usefulness of neuroscience research (or of brain imaging in particular) to improve the diagnostic accuracy of TBI and psychiatric disorders. It's not like I've never considered or discussed important issues related to <a href="http://neurocritic.blogspot.com/2015/08/will-machine-learning-create-new.html" rel="nofollow">neuroimaging biomarkers</a> and <a href="http://neurocritic.blogspot.com/2015/08/the-idiosyncratic-side-of-diagnosis-by.html" rel="nofollow">machine learning</a>, without regard to Amen. And I will own up to my mistakes here if I'm ultimately proven wrong and HMPAO-SPECT is a superior diagnostic tool to FDG-PET and *MRI. However, the general consensus is that <a href="https://scholar.google.com/scholar?hl=en&q=accuracy+of+HMPAO-SPECT+vs+FDG-PET+neuroimaging" rel="nofollow">FDG-PET is more accurate than HMPAO-SPECT</a>.<br /><br />e.g., "We recommend (18)F-FDG PET be performed instead of perfusion SPECT for the differential diagnosis of degenerative dementia if functional imaging is indicated." (e.g., <a href="http://www.ncbi.nlm.nih.gov/pubmed/25453043" rel="nofollow">O'Brien et al. 2014</a>).<br /><br />"FDG-PET is quantitatively more accurate and thus better suited to multicenter studies than perfusion SPECT." (<a href="http://www.ncbi.nlm.nih.gov/pubmed/21729421" rel="nofollow">Herholz, 2011</a>).<br /><br />A recent review of <a href="http://journal.frontiersin.org/article/10.3389/fnene.2013.00013/full" rel="nofollow"> FDG-PET in mild TBI</a> stated that "...10 published reports evaluating FDG-PET after mTBI, excluding cases of complicated mTBI where damage was observed on the CT or MRI scan after an apparent mild injury ... demonstrate <b>varying degrees of sensitivity to detection</b> at acute, subacute, and chronic phases of injury." <br /><br />Finally, a recent FDG-PET study (<a href="http://www.ncbi.nlm.nih.gov/pubmed/24102309" rel="nofollow">Petrie et al., 2014</a>) found that FDG-PET was not able to distinguish between participants with versus without PTSD. As you mentioned, however, the number of patients in Amen's database vastly exceeds the number in most studies.<br /><br />Ultimately the decision to publish in an open data journal like PLOS ONE could propel the field forward if independent investigators are able to validate Amen's findings. If you think HMPAO-SPECT is the future, perhaps you can consider redirecting your research efforts away from MRI if that's the best way to help patients with TBI and PTSD. Or at the very least request Amen et al.'s dataset to analyze yourself, since you think this study is quite interesting.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-69253030783963460882016-01-26T12:59:58.705-08:002016-01-26T12:59:58.705-08:00"no other neuroimaging method has come close ..."no other neuroimaging method has come close to that"<br /><br />There have been relatively few studies using HMPAO-SPECT to study TBI/PTSD. Most have used fMRI, DTI, and/or structural MRI. There is no reason to suppose that HMPAO-SPECT should have the same results as the other modalities, it relies on completely different principles.<br /><br />"The medical point is, the doctors were never wrong."<br /><br />And how many times must doctors be wrong before you'll believe them?<br /><br />"SPECT is completely superfluous, as it added nothing to the psychiatrist's clinical diagnosis"<br /><br />Well, if Amen were declaring victory and closing his research program, then I agree it was a waste of time. But presumably, he will follow this up with studies using the same methodology to evaluate asymptomatic patients at risk, therapeutic responders vs nonresponders, and other clinically useful predictors. You know, the same formula used as "Future Directions" for pretty much every analogous grant proposal in the MRI world. <br /><br />"One can also wonder why the accuracy of SPECT was perfect in the smaller "selected sample""<br /><br />The "smaller sample" was still much larger than most comparable imaging studies of TBI and PTSD. And it was apparently designed to address some of the very objections you initially raised regarding sources of variability: they all came from just one site, and their medical history is more tightly controlled. More importantly, they were compared to matched healthy controls, whereas the larger sample compared patients PTSD & TBI to random patients who had SPECT scans for other reasons. No one should be surprised that results are cleaner in the smaller sample.<br /><br />Overall, the methodology used in this paper is pretty run-of-the-mill for neuroimaging studies, except that (1) they used HMPAO-SPECT instead of *MRI, and (2) they achieved much better results. Oh, and their sample size was pretty good, even for the subgroup. But if you insist on smugly dampening people's enthusiasm for interesting results, then I suppose you could point out that like almost every other neuroimaging study there was no independent validation group. Without a doubt, if someone bothers to repeat the study they will find that accuracy drops a little. If you read between the lines, even the authors you mock acknowledge this, because that is the nature of this type of analysis. And when 100% drops to 90%, we can finally return to feeling comfortably dismal about the prospects of neuroimaging in helping people with psych disease. Right?<br /><br />Disclosure: I use MRI to study patients with TBI and PTDS, I think this study is quite interesting, and I don't give a hoot whether reported accuracy is 90% or 100%.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-21605329.post-79261140778859765812016-01-26T11:39:58.155-08:002016-01-26T11:39:58.155-08:00I realize that accuracy is a function of TP, FP, T...I realize that accuracy is a function of TP, FP, TN, and FN, but I was questioning HOW sensitivity and specificity could be so high, when no other neuroimaging method has come close to that. I wasn't referring to precision in the statistical sense. The medical point is, the doctors were never wrong. This is a claim that impacts patients' lives, and my concern about whether it's true isn't mere numerological superstition.<br /><br />And if the entire exercise wasn't entirely circular (using scans for clinical diagnosis, then finding scans agree 100% with clinical diagnosis), as Amen et al. told us it was not, then SPECT is completely superfluous, as it added nothing to the psychiatrist's clinical diagnosis! Or as someone on <a href="https://twitter.com/mehta_mitul72/status/690641910189461504" rel="nofollow">Twitter said</a>:<br /><br />"<i>if it's 100% vs the clinical diagnosis it is completely redundant. (Dr Amen, so impressed by this insight closes his clinic)</i>"<br /><br />One can also wonder why the accuracy of SPECT was perfect in the smaller "selected sample" (extreme overfitting? <i>how</i> were they selected? why was the incidence of ADHD so high?), but not so perfect when analyzing the entire database. In <a href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0129659#pone-0129659-t006" rel="nofollow">Table 6</a>, we can see that for TBI vs. PTSD sensitivity on-task = 82% and specificity on-task = 60%.The Neurocritichttps://www.blogger.com/profile/08010555869208208621noreply@blogger.comtag:blogger.com,1999:blog-21605329.post-73515425846526551652016-01-26T07:03:58.859-08:002016-01-26T07:03:58.859-08:00You wrote: "So the claims of Amen et al. are ...You wrote: "So the claims of Amen et al. are remarkable. Stunning if true. But they're not. They can't be. The accuracy of the classifier exceeds the precision of the measurements, so this can't be possible."<br /><br />Accuracy is a function of TP, FP, TN, and FN. Measurement precision is not part of the calculation. And if FP=FN=0, then accuracy = 100% by definition.<br /><br />In other words, if you call a coin flip heads and it turns up heads, then your accuracy for that coin flip is precisely 100%. Naturally, your results for the coin flip are limited in generalizability due to a sample size of one. But despite the caveats, the accuracy is still 100%. What else could it possibly be? <br /><br />And likewise there are caveats for any scientific paper. But why you belabor them for Amen et al? If they had reported an accuracy of 67%, would you likewise have been compelled to point out that their accuracy may turn out to be a different value if/when their method is repeated by others? After all, the same issues of motion artifact, etc are equally applicable regardless of what value of accuracy is reported. And most papers reporting SPECT results expect their readers to understand these methodological issues implicitly, without a giant disclaimer at the end regarding the modality. <br /><br />In short, does a paper reporting 100% accuracy require special scrutiny that is not required of a paper reporting 67% accuracy? If so, then I suggest that getting worked up over a result of 100%, but not 67%, is nothing more than numerological superstition. Anonymousnoreply@blogger.com