Evaluating Your Own SLR Process
This section is distinct from Appraise Study Quality, which assesses the rigor of the primary studies you have included. This section asks a different question: how rigorously did you conduct the review itself?
Overview
Quality appraisal tools such as CASP, MMAT, and JBI look outward: they help you evaluate the studies in your dataset. The checklist and scoring rubric on this page look inward: they help you evaluate your own review process against recognized best practices.
The items below are adapted from Petersen, Vakkalanka, and Kuzniarz (2015), who derived them from a systematic mapping study of how SLRs and systematic mapping studies are conducted in practice. Use this rubric in two ways:
- During planning: as a checklist of actions to build into your protocol.
- Before submission: as a retrospective audit to identify gaps in your process and to disclose them transparently in your methods chapter.
This rubric was developed in the context of software engineering research. The core dimensions; motivating the review, search strategy, search evaluation, extraction/classification, and validity; apply equally to business and management SLRs. Items that refer to software-engineering-specific classification schemes may be skipped if they are not relevant to your discipline.
Part 1: Activities Checklist
The table below lists the 26 actions identified by Petersen et al. (2015) as relevant to a rigorous systematic review or mapping study. Work through each row and mark whether the action was taken (✓), partially taken (~), or not taken (✗). This produces a ratio score: count your ✓ marks and divide by 26 (or by the number of applicable items).
| Phase | Action | Taken? |
|---|---|---|
| Motivate the review | Motivate the need and relevance of the review | |
| Define objectives and research questions | ||
| Consult with the target audience (e.g., supervisor, domain expert) to refine questions | ||
| Search strategy | Conduct a database search | |
| Apply snowball sampling (backward and/or forward) | ||
| Conduct a manual search of key journals or conference proceedings | ||
| Develop the search | Use a structured framework (PICO, SPIDER, or PCC) to derive keywords | |
| Consult a librarian or domain expert during search design | ||
| Iteratively refine the search string to improve coverage | ||
| Derive additional keywords from known relevant papers | ||
| Use thesauri, encyclopedias, or controlled vocabularies (e.g., MeSH, EBSCO subject headings) | ||
| Evaluate the search | Test the search against a set of known-relevant papers | |
| Have an expert evaluate the search results | ||
| Check the web pages or profiles of key authors in the field | ||
| Conduct a test–retest to check consistency | ||
| Inclusion and exclusion | Define objective, pre-specified criteria for inclusion and exclusion | |
| Involve a second reviewer; resolve disagreements systematically | ||
| Define and apply explicit decision rules for borderline cases | ||
| Data extraction | Define objective criteria for the extraction process | |
| Blind or obscure information that could bias extraction | ||
| Involve a second reviewer; resolve disagreements in extraction | ||
| Conduct test–retest of extraction on a subset | ||
| Classification | Classify studies by research type (e.g., empirical, conceptual, review) | |
| Classify studies by research method (e.g., case study, survey, experiment) | ||
| Classify studies by venue type (e.g., journal, conference, practitioner publication) | ||
| Validity | Discuss validity threats and limitations of the review process |
Part 2: Scoring Rubrics
After completing the checklist, use the rubrics below to assign a score to each of the five key dimensions. Record these scores in your methods chapter alongside a brief narrative.
Rubric 1 — Motivating the Review
| Score | Label | Description |
|---|---|---|
| 0 | Not described | The review is not motivated and no objectives are stated |
| 1 | Partial | Motivations and research questions are provided |
| 2 | Full | Motivations and questions are provided and have been developed in dialogue with the target audience (supervisor, practitioners, or domain experts) |
Rubric 2 — Search Strategy
| Score | Label | Description |
|---|---|---|
| 0 | Not described | Only one type of search was conducted |
| 1 | Minimal | Two search strategies were used |
| 2 | Full | All three strategies were used: database search, snowball sampling, and manual search |
Rubric 3 — Evaluating the Search
| Score | Label | Description |
|---|---|---|
| 0 | Not described | No actions were taken to improve the reliability of the search or inclusion/exclusion process |
| 1 | Minimal | At least one action was taken to improve either the reliability of the search or the inclusion/exclusion process |
| 2 | Partial | At least one action was taken to improve both the search and the inclusion/exclusion process |
| 3 | Full | All identified actions were taken |
Rubric 4 — Extraction and Classification
| Score | Label | Description |
|---|---|---|
| 0 | Not described | No actions were taken to improve extraction reliability or enable comparability between studies |
| 1 | Minimal | At least one action was taken to increase extraction reliability |
| 2 | Partial | At least one action to increase extraction reliability and studies were classified by research type and method |
| 3 | Full | All identified actions were taken |
Rubric 5 — Study Validity
| Score | Label | Description |
|---|---|---|
| 0 | Not described | No threats or limitations are described |
| 1 | Full | Threats and limitations of the review process are described |
Interpreting Your Scores
No minimum threshold is formally established in the literature for general SLRs; the rubric is a diagnostic tool, not a pass/fail gate. Use the results as follows:
- In your methods chapter: Report your scores and briefly explain any dimension rated 0 or 1. A low score on a dimension is not automatically a fatal flaw, but it must be acknowledged as a limitation.
- In your discussion: Dimensions scored 0 (especially search strategy and validity) should be discussed explicitly when qualifying the strength of your conclusions.
- As a planning aid: If you are still in the protocol stage, any action not yet checked is a concrete item to build into your plan before searching begins.
For more detail on designing and evaluating your search strategy, see the Search Quality Self-Assessment Checklist (adapted from vom Brocke et al., 2015), which provides granular guidance on the search phase specifically.