A recent twitter discussion on the merits (or seeming lack thereof) of small N studies led to an interesting idea for a new tweak on the RRR model. Check out the convo, but here’s the tl:dr version: I think that even small N studies can make valuable contributions to the literature because many of them can be combined into an unbiased meta-analysis of results from the individual Registered Reports (which Chris Chambers places at the top of his “evidence pyramid” for controversial research…but the pyramid seems applicable to all research in my opinion). We should create spaces in peer-reviewed journals for these small N studies to be published as individual bricks that can eventually be combined to build a solid house.
How can these small N studies ever build a house? A hypothetical example:
Dr. C is interested in replicating Experiment 2 from Loftus and Palmer (1974), a classic study on memory reconstruction. Dr. C works at a small university (we’ll call it Grandpa’s Cheesebarn University) and can only collect 100 participants per year for any given study. Instead of toiling away for years to reach an N large enough to make a stand alone contribution on the replicability of this seminal experiment, Dr. C would like to publish his own modest brick, in the hopes that others will come along and add their bricks at a later date.
Enter the BBBRRR (the power of the name is in its awfulness). Dr. C submits a Stage 1 Registered Report manuscript for review at the BBBRRR Journal with a plan to collect data from only 100 participants. The BBBRRR Journal sends the paper out for review. The reviewers and editor all agree that Dr. C has provided a strong rationale for replicating this particular experiment, he has planned reasonable and rigorous experimental procedures and data analysis plans, and he has provided a well justified number of similarly sized bricks (let’s say 12 total studies in this case) that would be needed to build a “house” of replication evidence for this experiment. The BBBRRR Journal extends an in-principle acceptance (IPA) to Dr. C. He can proceed with data collection with the assurance that the BBBRRR Journal will publish his paper if he follows his plan in good faith. There is one final catch. Dr. C also must agree to conduct a meta-analysis of all 12 studies, and write a summary report of this analysis at the end of the process. Dr. C agrees, conducts his study, writes his final report, and it is published in the BBBRRR Journal.
This now opens the door for Researchers A, B, D, E, F, G, H, I, J, K, and L to submit their own “mini RRs” specifying their plans to collect data from 100 participants following the same procedures and analysis plans laid out in Dr. C’s now published replication report. Once all 12 total replications have been conducted and published, Dr. C conducts the meta-analysis, including the results from all studies, and writes a summary report to “close out” the BBBRRR. All contributors to the independent replication reports are listed as authors on this final summary.
Key Benefits
- Researchers can publish results from relatively small samples, if these samples are considered meaningful contributions by reviewers and editors. In some areas of research, for example those involving hard-to-reach populations or rare diseases, even a small N can represent a serious investment of research resources. These contributions should be valued when they can contribute to bias-free meta-analyses over the long-term.
- No individual researcher is burdened with the considerable effort of coordinating a simultaneous, large-scale replication effort across many labs.
- All researchers can proceed with data collection having an IPA in hand.
- The final meta-analysis is free of publication bias.
- Replication results can be evaluated as they come in.
- We slowly publish many houses comprised of many solid bricks.
Dr. Christopher R. Chartier 