The Pre-Registration Challenge
A number of scientists have been vocal proponents of study pre-registration, in which detailed methodological and statistical plans for an experiment are registered in advance of data collection. The admirable goal is to eliminate questionable research practices such as failing to report all of a study's dependent measures, deciding whether to collect more data after looking to see whether the results are significant, and selectively reporting studies that 'worked.' Along these lines, the journal Cortex has recently launched a new type of article called Registered Reports, largely through the efforts of Dr Chris Chambers:
Unlike conventional publishing models, Registered Reports split the review process into two stages. Initially, experimental methods and proposed analyses are pre-registered and reviewed before data are collected. Then, if peer reviews are favourable, we offer authors “in-principle acceptance” of their paper. This guarantees publication of their future results providing that they adhere precisely to their registered protocol. Once their experiment is complete, authors then resubmit their full manuscript for final consideration.
Many of these same researchers are also strong advocates for publication reform, some even calling for journals to be eliminated altogether in favor of post-publication, crowd-sourced review and reputation ranking. But supporters of both pre-registration and the Open Science Framework haven't yet utilized its capability to submit their new work (as opposed to the Reproducibility Project for replications).
Since calls for pre-registration of basic research studies have been ongoing for years, perhaps its proponents have been too conservative with taking matters into their own hands. One might even say there's a distinct lack of risk-taking among the strongest believers. What's to prevent them from pre-registering studies in public databases or on their own blogs (without formal peer review but perhaps soliciting comments)? Or publishing a Study Protocol in a BMC journal (like @SallyScientist did)?
You know, Unilaterally Raising the Scientific Standard... Jona Sassenhagen has already done this by registering his study in the German Clinical Trials Register. Why haven't more followed suit? Or maybe they have, and I just don't know about it. If that's the case, please enlighten me in the comments.
posted by The Neurocritic @ 3:34 AM
24 comments
Subscribe to Post Comments [Atom]
24 Comments:
Speaking of, errrr, me, I've just decided I'd pre-register a student project, a really small-scale, simple replication (albeit of some high-profile, controversial data), and was considering where to register it to. So, okay. Maybe I should take it to OSF.
In defence of researchers, pre-registration takes time. As I've told Neuroskeptic, it's a bit of a scary thing; you want to be sure everything works out properly, so if you're at least as conscietous as me, you'll test and retest your idea again and again on as much preliminary data and comparable data sets as possible. You likely want to pre-register something that doesn't suck, and that's work - precisely specifying the parameters so that they're likely to measure something meaningful. It's likely not actually more work than if you'd do the post-hoc p-value fishing, which also takes time; it's just more work up front. So, it takes time.
Big fan of your work, by the way. Top of my blog roll.
Nobody's doing it because the overwhelming effect of pre-registration will be to force researchers into spending more effort perfecting the designs of a smaller number of studies, therefore reducing the total number of potential publications a researcher can get out of a research program in a given time window. Good for the field, but not so good for individual researchers' CVs if everybody else isn't doing it. The most prolific and most senior people really need to lead here.
"...perhaps its proponents have been too conservative with taking matters into their own hands. One might even say there's a distinct lack of risk-taking among the strongest believers. What's to prevent them from pre-registering studies in public databases or on their own blogs (without formal peer review but perhaps soliciting comments)?"
Zinger! I hereby offer to review the acquisition, and I'll cast a gaze over stimuli & exptl design with the caveat that I get out of my depth pretty quickly once the data leave the scanner.
@JRQ: "The most prolific and most senior people really need to lead here."
Would seem to be the case. Same goes for use of "pre-publication" services like arXiv, and post-pub. peer review services like PeerJ.
Even once a trial has been pre-registered, there's nothing to stop researchers going on to ignore their own protocol.
This publicly funded trial has refused to release outcomes in the manner laid out in their protocol, and is fighting an FOI request for them:
https://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po
As it's related to CFS, the researchers have also been trying to claim that interest in their protocol defined outcomes is a result of anti-psychiatry harassment, or whatever other excuse they can come up with.
This trial's results are also important as it relates to the DWP's biopsychosocial benefit reforms. The DWP part funded the trail (a first for them) and the results we have had seem to hint that they did not get the results they needed to justify their reforms: psychosocial management led to no improvement in employment rates, or declines in benefit claims. The DWP also expected the results to show that being in work helped patients achieve better outcomes - these results have still not been the case, but a poster presentation would seem to indicate that this was not the case.
So long as researchers are able to selectively release results, we will end up with a deeply distorted view of reality.
Well said. You're absolutely right that some of us who advocate pre-registration could be doing more on a personal level to push reform. We need to be taking risks ourselves and leading by example.
I've realised this fact since I pitched the Cortex Registered Reports initiative last October. Over the last 6 months I've been reinventing the culture in my own lab to fit the kind of world I would like to see: pre-registration, open science, full data sharing. It wasn't easy at first; there was justifiable resistance from some in my group who felt it would disadvantage their careers. In time, though, I'm glad I was able to convince them that doing this makes us a force for good. That said, we are probably going to take a hit in terms of quantity of publications. But I'm confident that we will more than make up for this in terms of quality. What you will see from my lab will be honest science - no cherry picking, no p-hacking, no HARKing.
The catalyst for me was being awarded an £800K BBSRC grant on the cognitive neuroscience of impulse control. I have committed to make this entire grant open science in every respect, and as far as I know it will be the first UK research grant to make such a commitment.
What this means is that every empirical paper from our grant will be initially pre-registered before data collection takes place. Unfortunately, I may not be able to do this via a journal because I am currently the only editor for a cognitive neuroscience journal that has invested in pre-registration, and I obviously can't submit to my own journal. So until someone else takes the bull by the horns, we will pre-register with Figshare or the Open Science Framework.
Jona Sassenhagen - Thanks! It was a gutsy move. There are a lot of unanswered questions regarding this new approach to science, like which sorts of studies "qualify" for pre-registration (mandatory or not) and which would not fit. Will those who pre-register become 'saintly' while all others are suspect? Will this foster a climate of distrust, always assuming the worst about people? I've seen some blog commenters go as far as saying that even pilot studies are bad because they "bias" your approach. This seems relatively naive when considering the time, money, and resources required to conduct full-scale experiments.
JRQ - Very good point. The most prolific and senior people might not be inclined to take the lead on this in a way that protects their students and post-docs from the additional work. It's fine for a lab head to say "hey, submit Registered Reports to Cortex" but how many are willing to do it all by themselves?
practiCal fMRI - Very generous offer! You might be very busy once the proposals start rolling in. That brings up a related issue: pre-registered reports should be highly technical and detailed regarding acquisition parameters, analytic methods and statistical approaches (right?). Those who are most expert in those areas are likely to be very busy reviewing submissions.
Wilson - It's true that some clinical trials registered at places like ClinicalTrials.gov never seem to post results. The Registered Report format for journal articles locks you into submitting your paper, for better or for worse.
Chris Chambers - Thanks for commenting! I was composing this string of replies when your message came in. I'm glad to see you are taking the lead on this, starting with a brand new set of studies. Congratulations on receiving the grant. I'll look forward to reading the papers from your lab, perhaps at a longer delay than was previously customary.
Chris, I see the Cortex RR more as an additional support structure/incentive for researchers who want to pre-register. Everybody can (and likely should) easily unilaterally pre-register already. What C-RR gives us is, basically, reviewer feedback on the pre-registration protocol :) … and IPA.
In theory, C-RR - like formats are not necessary for pre-registration. They're just helpful. (Practically, something like this may be necessary.)
@Neurocritic:
"You might be very busy once the proposals start rolling in."
Perhaps, but I'd have to balance that effort against the time I waste reading flawed, published papers.
"That brings up a related issue: pre-registered reports should be highly technical and detailed regarding acquisition parameters, analytic methods and statistical approaches (right?). Those who are most expert in those areas are likely to be very busy reviewing submissions."
I would need all the acquisition details, yes. My goal would be to check that the acquisition has received due consideration. Note, however, I'm not advocating becoming a "parameter Nazi." Rarely is there a single best way to do anything. Often with fMRI, there are a number of acquisition protocols that can be considered suitable; small differences in spatial or temporal resolution, say, may make no difference in the final analysis. On the other hand, there are a number of "red flags" that I would want to see the proposers consider before I acquiesced. (And anyone is always free to ignore my advice and proceed regardless!) For instance, if the study involved kids with ADHD and the authors wanted to use GRAPPA-accelerated EPI on a 32-channel head coil, I'd want them to tell me why they think they need it, e.g. results of a pilot study, or a previous paper showing GRAPPA-EPI on a 32-ch coil being less motion sensitive than unaccelerated EPI on a 12-ch coil.
You've actually hit upon one of the reasons I started my blog. I noticed there were many groups who didn't or couldn't provide consideration of their acquisition protocol. (Well, that's my assumption when I see inappropriate use of methods!) It seems that some labs limit the choices of fMRI protocols offered, others leave neuroscientists to fend for themselves. Both approaches tend to lead to sub-optimal acquisition. I find it unconscionable on many levels that an experiment is done badly for wont of a teeny bit of input up front.
I'll close by noting that I am still ambivalent about pre-reg; it's complicated, there will be costs and benefits, etc. But if it's going to be done then I'd like to see it done as well as possible, just like any other experiment.
practiCal fMRI - Thanks for the detailed input. That's the thing, few people know as much about acquisition parameters as you do. How much time can you (and experts in other technical areas) devote to reviewing these submissions, in an idealized world?
I agree that pre-registration is a good idea in theory, and I think it'll be an interesting experiment to see whether research practices improve and "scientific quality," or replicability, increases. But even the Guardian post stated, "Study pre-registration doesn't fit all forms of science, and it isn't a cure-all for scientific publishing." Which forms of science does it fit, and which doesn't it fit? I can see the danger in pre-registered science being viewed as "saintly" research with the rest of it tainted.
@Neurocritic:
Yup, you are raising valid points. I believe Sophie Scott will be presenting a case for the opposition in due course.
Much as I value the debate, I think we agree that we just have to try it and see how it goes. It's a relatively low risk proposition for science as a whole, I think. And if the pioneers are in stable, relatively senior positions then the career risks should be manageable, too. Which is rather the point of your post... Just do it!
I'll link to Sophie Scott's piece here (in an addendum) when it appears. Always good to present both sides...
"Put your money where you mouth is!"
Excellent call!
Even before my move to Regensburg from Berlin, I've been revamping the entire infrastructure of our lab to immediately publish our our data as soon as we have collected them and I've been telling others about these efforts:
http://www.slideshare.net/brembs/some-technical-hurdles-towards-open-science
http://www.slideshare.net/brembs/fixing-the-infrastructure-for-open-science
Preregistration is something new to me and as soon as there is an experiment which I can preregister (rather than a student playing around with a setup), we will do that.
So, from our perspective, what's keeping us from going all-open are technical problems, mainly due to our abysmal infrastructure. Most of what I do beyond research and teaching is geared towards fixing this problem.
Very good points.
My group is moving towards routinely publishing protocols, registering studies etc., but there are some barriers. One is that many established pre-registration and protocol publication platforms incur charges. These costs, together with eventual open access charges, can rapidly add up. This is a particular problem for (unfunded) student projects, although we do have one example where we pre-registered a student project:
http://www.controlled-trials.com/ISRCTN02532638/
...and it was eventually published:
http://www.ncbi.nlm.nih.gov/pubmed/22539781
Interestingly, although ISRCTN only provides for very basic details, the reviewers were extremely enthusiastic that we'd done this for this kind of study.
For grant-funded projects we'll now be publishing our protocols, with @sallyscientist’s recent Trials paper being one. This perhaps doesn't work so well for secondary data analysis projects where the data have already been collected, although we’re trying this in one case. For unfunded projects we'll look at alternatives - OSF is one, and the idea of posting protocols on a blog is a great idea which we'll look into.
However, one of the reasons I'm enthusiastic about more journals adopting an additional submission format along the lines of Registered Reports is that it links all of these different component parts. Another barrier, and a problem with the current clinical trials registration model, is that it's extremely fragmented, and it therefore takes effort to link registrations, protocols and subsequent publications.
Little, if any, of the work we do would suit Cortex, and it would be great to have a wider range of journals offering a Registered Reports format (or similar). I think this will be important if we want to have a proper sense of whether this approach actually makes things any better in the long term (or if there are unintended consequences).
Marcus Munafo - Thanks for your comments. Great to see that your group is already taking advantage of ISRCTN to register protocols. For the ISRCTN database, is it mandatory to report all of your proposed outcome measures?
@Neurocritic:
Not exactly - you should, and the assumption is that you will, but there's no way that ISRCTN can enforce this. The same applies to publication of protocols.
At the moment, the onus is on the reviewer or reader to link these parts together, and this takes effort (so often isn't done - see Mathieu et al. 2013).
http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0059910
This is why I like Chris's Registered Reports format - it links both the protocol and the eventual publication of the results in one place.
The current system is better than nothing - in principle one can check what was done vs what was originally planned, but it's not trivial to do so.
"Cortex", the same scam journal that faked their impact factor by publishing a ton of self-referential papers not too long ago? Look at the data, it's obvious.
The intent is noble, but we all know that noble goals in social contexts are unstable strategies because of cheaters. It is obvious how this is going to play out in the long run: people will do the studies and THEN "pre-register" them.
The problem is money, as always. If research grants always came with a pot of money dedicated to replication work, we would not have a replicability problem. Funding agencies tend to favor the "transformational" qualities of grants, rather than solid and replicable work. If you read many of the "transformational" grants that get funded, it does not take long to realize they are mostly pies in the sky that somehow dazzled the mediocre researchers who acted as reviewers. Innovation is good, but nonreplicable innovation is useless.
In Internet time, this conversation is long past. But, three quick points:
(1) if you'd like to suggest a meta-data format for registration that is not yet available at OSF, we would be happy to implement it (right now there are very very simple meta-data forms for submitting the registration).
(2) to find recent and popular public registrations at OSF see this link: http://openscienceframework.org/explore/activity/
(3) If you know any journal editors that are open to the practices, but are concerned about the infrastructure development to support them, tell them to contact us - we will do the work and adapt it to their needs.
Late post:
You can find recent and popular public registrations on OSF here:
http://openscienceframework.org/explore/activity/
And, if you'd like to contribute to developing metadata standards for any registration types (neuroscience or otherwise), contact us. We will be pleased to collaborate. (Registration metadata right now is extremely simple.)
And, if you know any editors that are interested in registration but leery of the back-end development work, have them contact us. We will help and adapt OSF functions to what they need for their journal.
Anonymous said:
The intent is noble, but we all know that noble goals in social contexts are unstable strategies because of cheaters. It is obvious how this is going to play out in the long run: people will do the studies and THEN "pre-register" them.
You raise important issues, Anon. Starting with earliest discussions within APS, I have been arguing that any useful implementation of pre-registration must include three elements: the preregistry must (a) store the date, (b) make all preregistrations public and open to search engines either instantly or, upon author request, after say 1 year (but not more), and (c) require the author to make a solemn affirmation that the data corresponding to the registration have not been collected as of the date of the registration (they should affirm something like "I understand that if this statement is incorrect, I will be deemed guilty of serious scientific misconduct.") And all publications that assert data were preregistered must include the URL for readers to inspect the registration info.
True, you cannot force a university to treat a fraudulent assertion of (c) as a firing offense, but given (a) and (b), it would be very imprudent for anybody to assume that abusing the registry would not get them in a heap of trouble. A full-on fraudster would presumably not be deterred, but I think even our corner-cutting colleagues would give it some pause.
But I agree with you that weakly implemented preregistration can easily be worse than no preregistration at all.
"But I agree with you that weakly implemented preregistration can easily be worse than no preregistration at all."
In my limited experience, not really. Likely, weak preregistration (like what I did with the DRKS) will have little effect, but still probably a positive one.
Fraud is always possible, in every system, and pre-registration isn't about preventing fraud at all. If you're the kind of "scientist" who'd "pre"-register a study after data has been collected and lie about it, you may as well just make up the data.
Also, I just registered a small thing at OSF, yay.
"If you're the kind of "scientist" who'd "pre"-register a study after data has been collected and lie about it, you may as well just make up the data"
If you are a cheater, pre-registering is the way to go because it gives the study a further level of legitimacy (and so it decreases the chance of getting caught). If pre-registration gains any traction (which I doubt) I anticipate that cheaters will do just that in the future: pre-register and then fake the data as usual.
Thanks for the additional comments.
If you're going to fake data outright, why draw further attention to yourself by registering in advance? But maybe that's part of the thrill, for the most egregious offenders. I'd hope that co-authors would be able to detect a fraudulent pre-registration (if not before submission, then after publication)... Certainly all the authors would need to sign off on it in advance.
Brian Nosek - It's never too late (I say 3 days late)! Thanks for the information.
Post a Comment
<< Home