Scoping + Systematic Reviews

Standards for Screening and Reviewing

Now that you have completed your searches of multiple databases, you're left with likely a ton of results. Some of them may be more relevant to your review than others. At this point, you're ready to begin screening and reviewing articles.

In this guide, we will use Screening to refer to title/abstract screening and Reviewing to refer full-text reviewing. 

Here are some guidelines that can help you make these processes run smoothly.

Tracking the Screening and Review Process

Remember the PRISMA guidelines we talked about a few sections ago? There's a diagram component that is helpful (and sometimes necessary) to include in your review, called the PRISMA Flow Diagram (more on that soon!). To complete the diagram, you'll need a listing of:

• how many articles you've found in different databases,
• how many were removed because of duplication,
• how many you screened out during the title/abstract phase,
• how many articles were you unable to find
• how many articles were removed during the full-text screening phase,
• and, finally, how many articles/studies you will be including in the final review.

Using a citation management tool like Zotero, or a review management tool like Covidence, may help you keep track of the counts at the various stages.

Inclusion and Exclusion Criteria

Your search terms and the filters/limits you apply are generally not enough to narrow your results to the most relevant and highest quality studies for your project. The final step to selecting these studies is to apply your inclusion and exclusion criteria. Basically, these are the reasons why you keep (include) or reject (exclude) articles as you look through the results, reading titles and abstracts (and sometimes the whole article)

Examples of types of Inclusion/Exclusion Criteria

• PICO(T) elements - if one of the main elements of your topic does not match those of the study, you may need to exclude it
• Age - if you can't use a filter/limit to exclude studies that do not focus on the age group you require, you may need to exclude those studies yourself.
• Setting - i.e. home, acute care, assisted living facility
• Study Design - sometimes a filter/limit doesn't exist for the study design you're interested in; in that case you'll need to look through articles to find that detail yourself.
• Number of subjects - do you have a minimum study group size?
• Study drop-out rate

Inter-rater Reliability

Why It's Important

Evaluating is a subjective process, so different observers’ perceptions of situations and phenomena naturally differ. Reliable research aims to minimize subjectivity as much as possible so that a different researcher could replicate the same results.

When designing the scale and criteria for data collection, it’s important to make sure that different people will rate the same variable consistently with minimal bias. This is especially important when there are multiple researchers involved in data collection or analysis.

How to get it?

To improve your inter-rater reliability, here's what you do.

1. Select 5 or so articles at random from your results (if you think your results are more varied that 5 articles will represent, go ahead and select more).
2. Ask all of the members of your team who will be screening to evaluate those articles according to your inclusion/exclusion criteria and make a determination of whether to include or reject each article.
3. Compare results and talk about any differences of opinion.
4. Revise your inclusion/exclusion criteria to reflect insights that conversation provided.
5. Repeat this process for the full-text review section of your review.

What Happens if I End Up with Zero Sources?

This may happen from time to time. This is called an empty review, and may actually still be publishable if it highlights a major gap in the research literature. 

You can read more about how to move forward with an empty review:

Checking for Retractions

If an article has been retracted, you will want to remove it from your review until a correction has been issued (if that happens). 

So what is a retraction? Articles can be retracted (or withdrawn from a journal), either temporarily or permanently, for several reasons, including (Wager & Williams, 2011):

• Honest errors – data entry errors, calculation mistakes, statistical errors.
• Non-replicable findings – questionable experimental design, insufficient data, or insufficient sample size.
• Research misconduct – data fabrication, data falsification, image manipulation, not declaring a conflict of interest.
• Redundant publication – publishing the same research in multiple journals without disclosure, self-plagiarism.
• Plagiarism – basically not citing the work of others.

How do I Check my Articles for Retractions?

• You can use a database like Retraction Watch to manually check each citation.
• Zotero, the online citation manager, will do it automatically. If you've been using it all along to capture your articles so that you can easily reference them in your paper later on, you're already doing it. If you haven't been using Zotero, you might want to start. It will save you lots of time when it comes to citing and referencing in your manuscript.