In response to the reproducibility crisis, some in the field have called for a “CERN for Psychology.” I believe the time is right for building just such a tool in psychology science by building on current efforts to increase the use of multi-site collaborations.
What would a CERN for Psych look like? It certainly would not be a massive, centralized facility housing multi-billion dollar equipment. It would instead be comprised of a distributed network of hundreds, perhaps thousands, of individual data collection laboratories around the world working collaboratively on shared projects. These projects would not just be replications efforts, but also tests of the most exciting and promising hypotheses in the field. With StudySwap, an online platform for research resource exchange, we have taken small steps to begin building this network.
Ideally, a CERN for Psych would also have a democratic and decentralized process for the selection of projects to devote collective resources to. Researchers could propose exciting study proposals, publicly post them, and the community of laboratories comprising the distributed network would decide autonomously and freely which projects to devote their resources to. This could be seen as a new form of peer review. Only those projects that a collection of one’s peers deem worthy of time and energy will be supported with large-scale data collection. The most exciting and methodologically sound ideas, as determined by the community, would be those receiving the greatest amount of resources. Again, StudySwap already provides a basic starting point for this feature and can eventually fulfill this requirement more fully with changes to the site.
Finally, a CERN for Psych should involve projects that are open and transparent for their full research life cycle. Using the Open Science Framework, projects would be open from idea proposal to methods development to data collection to eventual dissemination. Any interested party could fully review, criticize, praise, build upon, or reanalyze any component of the projects, their data, and their disseminated summaries.
This is not a pipe dream. The basic constituent parts are already in place, but there is much work to be done. What do we need to build a CERN for Psych?
-We need a large distributed laboratory network. If even 10% of psychological scientists devote a small portion of their lab resources to the CERN for Psych, we would be able to harness a massive amount of data collection capacity. This work has already begun, and dozens of labs have signed on for these efforts. Please join the network by filling out this 3-item form.
-We also need researchers who want to use the network to test important hypotheses and who are brave enough to take an innovative approach to their data collection practices. I believe that if we build it, they will come. We are currently recruiting 10 labs to each collect data from 100 participants in 2018 (Total N = 1,000) for just such a proposed study from someone not on the collection team. We will release a call in the near future soliciting study proposals from researchers without access to large samples at their home institutions and who can demonstrate that they would particularly benefit from a geographically dispersed and relatively diverse sample. Email me (cchartie@ashland.edu) if you have ideas you’d already like to propose. This will be a small demonstration of the feasibility of such projects.
-We also know we need a better online platform for StudySwap. The current page, using the OSF for meetings structure was a short term hack that we are already outgrowing after just 6 months. The new site will need much more sophisticated searching, tagging, and categorizing capabilities. We are working with OSF on these improvements.
-We need funding. For now, we can build the beginnings of a CERN for Psych without big money, but eventually, this endeavor will be much more successful with financial resources at our disposal. We are actively seeking funding to support early adopters of this system.
Please join us in building a CERN for Psych. Eventually, this project could involve data collection from millions of participants, conducted by thousands of research assistants, supervised in hundreds of labs, coordinated by a democratically selected and constantly changing set of dozens of leaders in the field.
Dr. Christopher R. Chartier 
This looks promising! I do have a few questions that i hope you are able and willing to answer:
1) Will the studies be pre-registered and will there be a link to this pre-registration in the final paper so people are able to check this information?
2) How will you make sure the studies will not end up in the file-drawer?
LikeLiked by 1 person
Thanks for your interest. Great questions! 1, yes, the studies will be pre-registered, even if the selected studies are largely exploratory in nature (don’t yet know this, of course). To number 2, this project is likely to attract a lot of eyeballs, so I can’t imagine we could “hide the results” even if we desired (which we do not). I am committed to disseminating the results publicly, and am relatively confident that a suitable publication outlet can be found whether the studies find positive or negative results (if they are analyzed via NHST).
LikeLike
“1, yes, the studies will be pre-registered, even if the selected studies are largely exploratory in nature (don’t yet know this, of course)”
The term “pre-registered” seems to have already taken on different meanings for some journals, and is therefore becoming ambiguous (and possibly meaningless). If i am not mistaken, some journals call their research “pre-registered” without providing the reader access to the pre-registration information via a link in the paper.
For instance, i can’t find any pre-registration information in articles published in “Comprehensive Results in Social Psychology”, which is supposed to be “the first pre-registration only journal in social psychology”. I have checked this for multiple articles there, multiple times, as i assume i must be missing something. I reasoned that after all that happened with social psychology in the last decades, a social psychology journal wouldn’t call itself transparent, and use pre-registration without giving the reader access to this crucial information. I fear however, that this is exactly the case but please correct me if i am overlooking this pre-registration information.
I am hoping you will provide the reader access to the pre-registration information via a simple link in the paper, as should (of course) always be the case for research that aims to be transparent and calls itself pre-registered.
LikeLike
We will most likely use the OSF pre-registration form and provide a link in resulting manuscripts (although I am reluctant to speak unilaterally for the full team that is emerging).
LikeLike
“We will most likely use the OSF pre-registration form and provide a link in resulting manuscripts (although I am reluctant to speak unilaterally for the full team that is emerging).”
That would not only be great, but also kind of necessary and in line with common sense concerning terms like “transparency” and “pre-registration”. In this post, and comments, you mentioned these terms and i reason it would be strange if you would not provide the readers with access to the (crucial) pre-registration information. It would be even stranger if people were opposed to doing that.
If this project will provide the reader with pre-registration information by including a simple link to this in the resulting papers (as should always be the case with pre-registration), and if the project will not contribute to publication bias, i wish this project all the best!
LikeLike
Reblogged this on POLITICAL PSYCHOLOGY.
LikeLike