Copyright © 2020 pubrica. All rights reserved 1

What is the Risk of Bias Assessment and Different Tools used to Assess

Systematic Review

Dr. Nancy Agens, Head,

Technical Operations, Pubrica

sales@pubrica.com

In Brief

The risk of bias is defined as the risk of

systematic error or a deviation from

reporting the truth or an appropriate

evidence finding. It occurs when there is

a systematic flaw or limitations in the

study design or the conduct of the study.

This article describes different

assessment tools and the types of study

designs for which the tool is applicable

along with the flow of how to avoid the

risk of bias.

I. INTRODUCTION

A systematic review guideline will often

determine the study design to answer the

formulated question, and it is not enough

in trusting the evidence of systematic

review over observational studies. There

are several terms like quality assessment,

critical appraisal, or internal validity,

which are used for the evaluation of

studies supported by the guideline.

Whereas in the risk of bias, the bias

determines the factor that can affect the

observations and findings of the study

systematically and report it to be different

from the actual conclusion. In other words,

a study affected by bias can be inaccurate

and thus leads to an inappropriate

guideline recommendation. Therefore,

inadequate study design or the conduct of

the study will give false findings which

result in wasting time and resources, and

missing opportunity for effective

intervention.

Quality assessment or risk of bias

assessment helps in regulating and

establishing transparency of evidence

synthesis (data collection methods, search

strategies, etc.) and findings and it is often

performed for each study in a review; thus

the collected evidence eliminate bias in

the outcomes. Generally, a quality

assessment outside of a systematic review

does not require evidence synthesis

method; however, it completely depends

upon the utilization of evidence synthesis

method. The risk of bias assessment can be

done by professionals who may be

conducted by the guideline development

group or requires a methodological expert

or by experiences systematic review

researchers. Once the evaluation is done,

then it can be used for the synthesis of

study results and findings and combined

into the complete assessment of the

evidence. There are certain common

factors which can introduce bias in many

research areas, that include

Selection bias – Participant or

Population comparability in a study

Performance bias – The influence of

estimate effect (aspects other than the

intervention or exposure of interest)

Reporting bias – Problems with

measurements or classification of

outcome (missing information)

The risk of bias is assessed frequently by

evaluating the study design and the

conduct of the study because it is

impossible to determine a specific study

has been affected by bias or not.

II. DIFFERENT TOOLS USED TO

ASSESS THE RISK OF BIAS IN THE

SYSTEMATIC REVIEW

AMSTAR 2 - A MeaSurement Tool to

Assess systematic Reviews

AHRQ RRB - AHRQ tool for

evaluating the Risk of Reporting Bias

Copyright © 2020 pubrica. All rights reserved 2

AHRQ ORB – AHRQ Outcome and

analysis Reporting Bias framework

GRADE – Grading of

Recommendations Assessment,

Development and Evaluation GRADE-CERQual – GRADE

Confidence in Evidence from Reviews

of Qualitative research

SAQAT – Semi-Automated Quality

Assessment Tool

NMA Quality – Framework for

evaluating the quality of evidence from

a Network Meta-Analysis

NOS - Newcastle-Ottawa Scale

AXIS – Appraisal tool for Cross-

Sectional Studies

QUIPS - QUality In Prognosis Studies

tool

RoB 1.0 – Cochrane Risk of Bias tool

for randomized trials

RoB 2.0 - Revised tool for Assessing

Risk of Bias in randomized trials

SYRCLE RoB – SYstematic Review

Centre for Laboratory animal

Experimentation Risk of Bias tool

SciRAP - Science in Risk Assessment

and Policy

SIGN - The Scottish Intercollegiate

Guidelines Network

ROBIS - Risk Of Bias in Systematic

Reviews

RoBANS - Risk of Bias Assessment

Tool for Nonrandomized Studies

RoBINS I - Risk of Bias In Non-

randomized Studies of Interventions

tool

ORBIT 1 – Outcome Reporting Bias

In Trials classification system for

benefit outcomes

QUADAS 2 - Quality Assessment of

Diagnostic Accuracy Studies

PROBAST - Prediction model Risk

Of Bias ASsessment Tool

ORBIT 2 - Outcome Reporting Bias

In Trials classification system for harm

outcomes

Reid – Selective reporting bias

algorithm

RTI 2012 – RTI Item Bank for

Assessment of Risk of Bias and

Precision for Observational Studies of

Interventions or Exposures

RTI 2013 – RTI Item Bank for

Assessing Risk of Bias and

Confounding for Observational Studies

of Interventions or Exposures

JBI - the Joanna Briggs Institute

OHAT - Office of Health Assessment

and Translation

Downs Black - Downs Black tool

III. HOW TO ASSESS THE RISK OF

BIAS

1. Plan your approach

2. Identifying an appropriate risk of bias

assessment tool 3. Be aware of related issues

4. Appraise each study

5. Report the assessment process

6. Use your appraisals to inform the

guideline

IV. DETERMINING AN APPROPRIATE

RISK OF BIAS ASSESSMENT TOOL

Copyright © 2020 pubrica. All rights reserved 2

Study type Assessment tools Year Source

Systematic reviews

AMSTAR - 2 2017 https://amstar.ca/Amstar-2.php

ROBIS 2016 www.bristol.ac.uk/population-health-

sciences/projects/robis/

SIGN checklist 2014 https://www.sign.ac.uk/checklists-and-

notes.html

Randomized trials

Cochrane RoB 2.0

Tool 2016 www.riskofbias.info

SIGN checklist 2014 https://www.sign.ac.uk/checklists-and-

notes.html

Non-randomised studies of

interventions

(case-control, cohort, etc.)

ROBINS-I 2016 www.riskofbias.info

SIGN checklist 2014 https://www.sign.ac.uk/checklists-and-

notes.html

Prognostic:

Prognostic factors

Risk prediction

modelling

Overall

prognosis/baseline

risk/prevalence/inciden

ce

JBI checklist for

prevalence studies 2017

http://joannabriggs.org/research/critica

l-appraisal-tools.html

PROBAST 2014 www.systematic-reviews.com/probast

QUIPS 2013 https://methods.cochrane.org/prognosi

s/welcome

Diagnostic

SIGN checklist 2014 https://www.sign.ac.uk/checklists-and-

notes.html

QUADAS-2 2011 www.bristol.ac.uk/population-health-

sciences/projects/quadas/quadas-2

Qualitative

CASP Qualitative

Checklist 2018

https://casp-uk.net/wp-

content/uploads/2018/01/CASP-

Qualitative-Checklist-2018.pdf

GRADE-CERQual 2017 https://www.cerqual.org/

JBI Checklist for

Qualitative Research 2017

http://joannabriggs.org/research/critica

l-appraisal-tools.html

Observational studies of

exposures (human

epidemiology, wildlife)

Navigation Guide

risk of bias checklist 2018 https://www.cosmin.nl/

OHAT tool 2015 https://ntp.niehs.nih.gov/ntp/ohat/pubs

/riskofbiastool_508.pdf

In vivo animal studies

Navigation Guide

risk of bias checklist 2018 https://www.cosmin.nl/

SciRAP tool 2018 http://www.scirap.org/

OHAT tool 2015 https://ntp.niehs.nih.gov/ntp/ohat/pubs

/riskofbiastool_508.pdf

CRED 2016 www.ecotoxcentre.ch/projects/risk-

assessment/cred

SYRCLE tool 2014 Systematic Review Centre for

Laboratory Animal Experimentation

In vitro studies OHAT tool 2015 https://ntp.niehs.nih.gov/ntp/ohat/pubs

/riskofbiastool_508.pdf

Obtained from: https://www.nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias#

Copyright © 2020 pubrica. All rights reserved 2

In summary, risk of bias assessment is a

tool used to regulate findings which are

accurate and appropriate, and it is essential

to select the risk of bias tool rightly. There

are many researches available to help, and

it makes things easier to find appropriate

tools for assessing the risk of bias. This

article describes different assessment tools

for a systematic review and the types of

study designs for which the tool is

applicable along with the flow of how to

avoid the risk of bias.

REFERENCE

1. Bero, L. A. Why the Cochrane risk of bias tool

should include funding source as a standard item.

Cochrane Database of Systematic Reviews, 2013.

2. Higgins, J. P., J. Savović, et al., Eds. Revised

Cochrane the risk of bias tool for randomized trials,

20 October 2016.

3. Marshall, I. J., A. Noel-Storr, et al. Machine

learning for identifying Randomized Controlled

Trials: An evaluation and practitioner's guide.

Research Synthesis Methods, 2016.

4. Reeves, et al. AMSTAR 2: a critical appraisal tool

for systematic reviews that include randomized or

non-randomised studies of healthcare interventions,

or both. British Medical Journal 358, 2017.

5. Whiting, P., J. Savović, et al. ROBIS: A new tool to

assess the risk of bias in systematic reviews was

developed. Journal of Clinical Epidemiology 69:

225-234, 2016.