Risk of Bias


Introduction

As part of the Metapsy initiative, we developed a version of the widely used Cochrane RoB 2 tool (Sterne et al., 2019). Our version is tailored for psychological intervention trials and provides clear, detailed guidance for answering RoB 2’s signalling questions in this field. To make it easier to use, we also created digital assistants to help with the rating process. All materials and tools are freely available at metapsy.org/rob.

The tool can be applied in two ways: using (1) an online RoB assistant, or (2) Excel version of the tool.


🤖 Online RoB Tool

The most user-friendly option is the online RoB assistant, in which each of the items can be directly answered as an online questionnaire. Brief guidance is provided with each item, and extended guideline can be found in the online handbook or in the current PDF version of the handbook.

With this assistant, reviewers can go through all relevant signalling questions step by step. Based on their answers, the assistant automatically skips any questions that aren’t needed for the rating. Once all questions are answered, domain scores and overall scores are calculated instantly by the built-in algorithm. The results can then be downloaded in various formats.


📄 Excel Version

If you prefer using Excel, you can rate studies with the rob-template. This version is more flexible than the online RoB assistant and can be quicker for experienced users.

One key benefit is that some items (like unbalanced clinical severity) in the Excel can be calculated automatically from a Metapsy database, if certain data has already been collected. This includes the baseline means and standard deviations, the type of rating (clinician or self-report), the number of participants and dropouts in each group, and the randomization ratio. To link this data correctly, the extraction sheet of the database should follow the Metapsy data standard (see the “Metapsy database” example).

You can then use the Database Application or the createRobRatings function from the metapsyTools R package to populate some of the domain ratings in the RoB template, and generate overall RoB ratings.

The Excel tool also works with other useful functions in metapsyTools. The checkRobDiscrepancies, for example, can be used to check for differences between reviewers and provides Cohen’s kappa as a measure of agreement in the end.


Online Handbook

On metapsy.org/rob/handbook, you can find an online handbook. This handbook provides further explanations on how to answer the signaling questions and includes all algorithms used to obtain the ratings. The handbook is always based on the latest version of the guidance document (see “Versioning”).


Versioning

The key document for the Metapsy risk of bias tool is the PDF guidance document, which is assigned a new version when changes are made. Updates to the guidance document will also be reflected in the Risk of Bias Assistant, the Database Application, R functions, and the online handbook.

The latest version will be displayed at metapsy.org/rob in the fingerprint widget, like this:

By clicking on the fingerprint button, you can copy a permalink to the current version to your clipboard (e.g., metapsy.org/rob/#v1.0-0 for version 1.0-0). This permalink ensures rating decisions can be reproduced even when new versions are released.


Downloads

1 The first five rows of this sheet have already been removed for use in the database application.