Registered Reports are a relatively new form of empirical article in which the conceptualization, methods, and proposed analyses are pre-registered and reviewed prior to data collection and/or analysis being conducted. This format is designed to minimize bias in deductive science, while also allowing flexibility to conduct exploratory (unregistered) analyses and report serendipitous findings.
The cornerstone of the Registered Reports format is that the manuscript will be assessed prior to data collection, with the highest quality submissions accepted in advance. Initial submissions will include a description of the key research question and background literature, hypotheses, study design and procedures, analysis pipeline, a statistical power analysis or other sample size rationale, and full description of planned confirmatory data analyses. Pilot data (where applicable) may also be included.
The review process for Registered Reports
Stage 1: Initial manuscript submission and review
Initial Stage 1 submissions are submitted through ScholarOne as Registered Reports and should include the following sections:
- Introduction
- A review of the relevant literature that motivates the research question and a full description of the study aims and hypotheses. Please note that following IPA, the Introduction section cannot be altered except stylistic edits (see below).
- A review of the relevant literature that motivates the research question and a full description of the study aims and hypotheses. Please note that following IPA, the Introduction section cannot be altered except stylistic edits (see below).
- Methods
- Full description of proposed sample characteristics, including criteria for data inclusion and exclusion (e.g. outlier extraction). Procedures for objectively defining exclusion criteria due to technical errors or for any other reasons must be specified, including details of how and under what conditions data would be replaced.
- A description of study procedures, in sufficient detail to allow another researcher to repeat the methodology exactly. These procedures must be adhered to in the subsequent studies for the IPA to be implemented.
- Deviation from Stage 1 submissions: In cases where the pre-registered protocol is altered after IPA due to minor changes, the authors may continue their research, but must make note of the changes, provide an explanation for the deviation upon submission of Stage 2 manuscript, and report the original registered analyses in an online supplement. Additional review may be necessary.
- Proposed analysis pipeline, including all preprocessing steps, statistical power analyses, and a precise description of all planned analyses, including appropriate correction for multiple comparisons. Any covariates or regressors must be stated and indicated that they will be subject to robustness testing (testing models with and without covariates included). Where analysis decisions are contingent on the outcome of prior analyses, these contingencies must be specified and adhered to. Pre-planned analyses must be reported in the main Results section of Stage 2 submissions. However, unplanned exploratory analyses will be admissible in a separate section of the Results (see below).
- NOTE: Estimated effect sizes must be justified and the proposed study must in turn be designed around a compelling rationale for sample size that links their planned sample to a particular statistical inference of interest. Authors can justify their sample in a variety of ways, including classic power analysis, but also based (for example) on a smallest effect size of interest (e.g., 80% power on d=.3); a Bayes Factor of interest (e.g. continue to the first of |BF| > 10 or 200 subjects), etc. Authors may also consider basing their estimated effect sizes on the existing literature. However, since publication bias has the potential to overinflate published estimates of effect size, power analysis must be based on the lowest available or meaningful estimate of the effect size based on an a priori power of 0.8 or higher. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis is permissible but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined.
- NOTE: Registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript.
- Full descriptions must be provided of any outcome-neutral criteria that must be met for successful testing of the stated hypotheses. Such quality checks might include the absence of floor or ceiling effects in data distributions, positive controls, or other quality checks that are orthogonal to the hypotheses.
- Timeline for completion of the study and proposed resubmission date if Stage 1 review is successful. Extensions to this deadline can be negotiated with the action editor.
- Any description of prospective methods or analysis plans should be written in the future tense.
- Pilot Data (Optional)
- Can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Any pilot studies will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the pre-registered studies.
- Can be included to establish proof of concept, effect size estimations, or feasibility of proposed methods. Any pilot studies will be published with the final version of the manuscript and will be clearly distinguished from data obtained for the pre-registered studies.
- Secondary Registrations
- The journal welcomes submissions proposing secondary analyses of existing data sets, provided authors can supply sufficient evidence (e.g. letter from independent gatekeeper) to confirm either (a) that they have had no prior access to the data in question or (b) the focal independent and/or dependent measure was not available prior to the pre-registration (e.g., new or re-coding of existing observational or interview data).
Stage 1 submissions that are judged by the Associate (Action) Editor handling the submission to be of sufficient quality and scope will be sent for in-depth peer review to subject area experts. As developed by the Open Science Framework, reviewers will be asked to assess the following when considering papers at the registration stage:
- The logic, rationale, and plausibility of the proposed hypotheses.
- The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
- Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed study procedures and analysis pipeline.
- Whether the authors have pre-specified sufficient outcome-neutral tests for ensuring that the results obtained are able to test the stated hypotheses, including positive controls and quality checks.
Following Stage 1 peer review, manuscripts will be rejected outright, offered the opportunity to revise, or be accepted. Proposals that meet the evaluation criteria listed above will be issued an in-principle acceptance (IPA), indicating that the article will be published pending completion of the approved experiments and analytic procedures, passing of all pre-specified quality checks, and a defensible interpretation of the results.
Following IPA, Stage 1 protocols are not published in Child Development, though all Stage 1 manuscripts will be hosted indefinitely on the SRCD website. Following approval of the final Stage 2 manuscript, Wiley will publish the final manuscript electronically and in print.
Stage 2: Full manuscript review
Once the study is complete, authors prepare and resubmit their manuscript for full review through ScholarOne using the same manuscript number as the Stage 1 review manuscript, with the following additions:
- The manuscript must contain a link to the approved Stage 1 protocol on OSF. The Stage 2 cover letter should state the page number in the manuscript that lists the URL.
- Submission of raw data and laboratory log
- Child Development encourages the authors to make research products including raw data, study materials, and analytic code freely available in a archival public repository (e.g., OSF or dataverse). To increase research transparency, data files uploaded to a public repository should be appropriately time stamped to show that data were collected (in the case of new data collections) or acquired (in the case of archival studies) after IPA and not before. Other than pre- registered and approved pilot data, no data acquired prior to the date of IPA is admissible in the Stage 2 submission. Where raw data are presented, they must be accompanied by guidance notes to assist other scientists in reproducing the analysis pipeline. Authors are also encouraged to upload any relevant analysis scripts and other study materials that would assist in reproducibility (e.g. stimuli & presentation code). Code and data may be required during Stage 2 review by Action Editors per Child Development’s instructions to authors.
- Any supplementary figures, tables, or other text (such as supplementary methods) can either be included as standard supplementary information that accompanies the paper, or they can be archived together with the data. Please note that any raw data provided should be archived (see above) rather than submitted to the journal as supplementary material.
- The authors must collectively certify in the resubmission Cover Letter that all non-pilot data was collected after the date of IPA.
- Authors are encouraged to deposit their data in any repository that renders it freely and publicly accessible and provides a digital object identifier (DOI) to ensure that the data remain persistent, unique and citable. Potential repositories include (but are not limited to) Zenodo, Figshare, Harvard Dataverse, and Dryad. For a comprehensive list of available data repositories, see http://www.re3data.org/. Please see Wiley’s data sharing policy for more details.
- Background, Rationale and Methods
- The framing, rationale and stated hypotheses in the Stage 2 Introduction must be the same as the approved Stage 1 submission. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods (e.g., correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of IPA should be covered in the Discussion.
- The framing, rationale and stated hypotheses in the Stage 2 Introduction must be the same as the approved Stage 1 submission. At Stage 2, any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense. Any textual changes to the Introduction or Methods (e.g., correction of typographic errors) must be clearly marked in the Stage 2 submission. Any relevant literature that appeared following the date of IPA should be covered in the Discussion.
- Results & Discussion
- The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. The analysis must still be mentioned in the Methods and include a description/warning of why it was wrong so that others do not cite it inappropriately. The analysis can be omitted with justification from the Results.
- It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be clearly justified in the text, appropriately caveated, and reported in a separate section of the Results titled “Exploratory analyses”. Authors should be careful not to base their conclusions entirely on the outcome of statistically significant post hoc analyses.
- Authors reporting null hypothesis significance tests are required to report exact p values and effect sizes for all inferential analyses.
The resubmission will most likely be considered by the same reviewers as in Stage 1, but could also be assessed by new reviewers. In considering submissions at Stage 2, reviewers will be asked to decide:
- Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions (such as quality checks, positive controls)
- Whether the framing, rationale and stated hypotheses are the same as the approved Stage 1 submission. Deviations from the Stage 1 Introduction and Methods section in general must be minor (e.g., addition of newer citations) and in no way inconsistent with the Stage 1 manuscript
- Whether the authors adhered precisely to the registered study and analytic procedures
- Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative
- Whether the authors’ conclusions are justified given the data
Reviewers are informed that editorial decisions will not be based on the perceived importance, novelty, or conclusiveness of the findings. Thus, while reviewers are free to enter such comments into the record, they will not influence editorial decisions. Reviewers at Stage 2 may suggest that authors report additional post hoc tests on their data; however, authors are not obliged to do so unless such tests are necessary to satisfy one or more of the Stage 2 review criteria. Papers accepted after Stage 2 review will be immediately published online and all papers in the Special Section on Registered Reports will subsequently be bundled with an editorial in the regular print version of the journal as a set.
Incremental Registrations
Authors may add studies to approved submissions. In such cases the approved Stage 2 manuscript will be accepted for publication, and authors can propose additional studies for Stage 1 consideration. If these studies extend the approved submission (as opposed to being part of new submissions), the editorial team will seek to fast-track the review process. This option may be particularly appropriate where an initial study reveals a major serendipitous finding that warrants follow-up within the same paper. In cases where an incremented submission is rejected (at either Stage 1 or 2), authors will retain the option of publishing the most recently approved version of the manuscript. For further advice on specific scenarios for incremental registration, authors are invited to contact the editorial office.
Manuscript withdrawal and Withdrawn Registrations
It is possible that authors with IPA may wish to withdraw their manuscript following or during data collection. Possible reasons could include major technical error or an inability to complete the study due to other unforeseen circumstances. In all such cases, manuscripts can of course be withdrawn at the authors’ discretion, though the Stage 1 IPA will remain on the SRCD website indefinitely and identified as withdrawn. Partial withdrawals are not possible; i.e. authors cannot publish part of a registered study by selectively withdrawing one of the planned studies. Such cases must lead to withdrawal of the entire paper. Studies that are not completed by the agreed Stage 2 submission deadline (which can be extended in negotiation with the editorial office) will be considered withdrawn.