What Makes a Scientific Study "Good"?

There is a huge variation among scientific studies. What this means is we can't possibly assess all of these different study designs by the same metrics. For example, you can't ask if a systematic review is randomized because that makes no sense. To assess the quality of your specific study design, you'll need to locate the right checklist for it. Below are the checklists for common study designs.

• CASP checklists work best for critical appraisal, or when you are assessing evidence that you plan to use in clinical practice.
• JBI checklists are more rigorous and work best when you're conducting a comprehensive review (systematic, scoping, etc.). 

Two groups are studied and compared: the "cases" (those with a certain condition) and the "controls" (those without that condition). This type of study looks backward in time, typically by comparing medical records of the cases and the controls, and therefore is considered to be retrospective. The researchers are trying to figure out differences between the two groups that might have led to the development of the condition. 

Appraisal Checklists

• CASP Case Control Checklist
• JBI Case Control Checklist

Biases

• Recall bias: When study participants misremember and misreport information. 

Case studies or reports detail unusual or unique signs and symptoms in a particular patient. While they are considered a weaker form of evidence than other studies, they oftentimes are the only option for rare diseases that do not have large enough populations to conduct an RCT or other large-scale study. 

Appraisal Checklists

• JBI Case Report Checklist
• JBI Case Series Checklist

Biases

• Admission rate bias: By virtue of being hospitalized, the person being studied in the case report had a higher likelihood of developing this condition. 

Cohort studies follow a group of people (or "cohort") over time, often to see what symptoms will develop or to watch the trajectory of an illness. Because this type of study looks forward in time by following this group of people, it is considered prospective.

Appraisal Checklists

• CASP Cohort Study Checklist
• JBI Cohort Study Checklist

Biases

• Selection bias: Participants in groups are too different to be compared, or their results are not generalizable to the population this phenomena affects. 
• Confounders: A third variable (income level, predisposition to a certain disease, etc.) affected the outcomes of the study participants. 

A cross-sectional study uses surveys or other data collection methods to gather information from a certain population. It is helpful for questions relating to the prevalence of a condition (though not etiology), or the preferences and attitudes of a population. 

Appraisal Checklists

• CASP Cross-Sectional Study Checklist
• JBI Analytical Cross-Sectional Study Checklist

Biases

• Antecedent-consequent bias: When there is uncertainty as to what came first. For example, it's unclear whether the exposure came before the disease development. 

Study participants are recruited and then divided into two groups by random methods (roll of a die, toss of a coin, etc.). Each group receives a different treatment (or one group receives no treatment). These types of studies typically look at the efficacy of a treatment. 

Appraisal Checklists

• CASP RCT Checklist
• JBI RCT Checklist

Biases

• Outcome reporting bias: When researchers selectively omit outcomes from their study to make their results seem more impressive. 
• Non-randomization: Group selection needs to be random. For example, maybe the researchers flipped a coin to see who would go into a group. Conversely, assigning people by their birth date would not be random.
• Masking: Participants and researchers cannot know who is in what group. 
• Drop outs: This concerns how many participants dropped out of the study and whether this is reported and justified. 
• Dissimilar groups: The two groups can only differ in one regard: whether or not they received the experimental treatment. If the groups are too different, this is a problem.

Other Resources

• Lesson 15 of Treadwell Library's "How to Conduct a Systematic, Scoping or Other Reviews" course covers how to assess a randomized trial for quality. Sign in with your MGB username and password to enroll.

Systematic reviews are reviews of the literature. A team conducts a massive search, finding every paper written ever on their topic (i.e., thousands of papers) and winnows them down to a small set that answer their question. They then perform a rigorous quality assessment of their final set of papers. 

Appraisal Checklists

• CASP Systematic Review Checklist 
• JBI Systematic Review Checklist
• AMSTAR Checklist

Biases

• Non-exhaustiveness: The review cannot miss a single paper on its topic because that missing paper could refute the review's results. With the correct selection of databases and grey literature searching (searching of unpublished literature), this is possible. 
• Meta-bias: The review is assessing bias in the papers it finds but it also needs to address how it will address its own bias (publication bias, etc.) 
• Lack of reproducibility: The review needs to produce the same conclusions if it was done again. 
• Publication bias: Studies with positive results are more likely to be published. To account for this, the reviewers should have examined grey literature.

Other Resources

Umbrella Reviews

You can also look up an umbrella review on your topic. Umbrella reviews are systematic reviews of systematic reviews. In other words, they perform quality assessment of systematic reviews. To limit to umbrella reviews, search in PubMed for:

[keyword describing your topic] AND umbrella[ti]

Videos

Our recent presentation at the Slavin Academy gets into the quality issues surrounding systematic reviews: