# Evaluating Your Own SLR Process

<p class="callout info">This section is distinct from [Appraise Study Quality](https://libguides.berlin-international.de/books/research-skills/page/appraise-study-quality), which assesses the rigor of the *primary studies* you have included. This section asks a different question: how rigorously did *you* conduct the review itself?</p>

## Overview

Quality appraisal tools such as [CASP, MMAT, and JBI](https://libguides.berlin-international.de/link/92#bkmrk-selecting-an-apprais) look outward: they help you evaluate the studies in your dataset. The checklist and scoring rubric on this page look *inward*: they help you evaluate your own review process against recognized best practices.

The items below are adapted from [Petersen, Vakkalanka, and Kuzniarz (2015)](https://doi.org/10.1016/j.infsof.2015.03.007), who derived them from a systematic mapping study of how SLRs and systematic mapping studies are conducted in practice. Use this rubric in two ways:

- **[During planning](https://libguides.berlin-international.de/books/research-skills/page/writing-a-protocol "Writing a Protocol"):** as a checklist of actions to build into your protocol.
- **Before submission:** as a retrospective audit to identify gaps in your process and to disclose them transparently in your methods chapter.

<p class="callout warning">This rubric was developed in the context of software engineering research. The core dimensions; motivating the review, search strategy, search evaluation, extraction/classification, and validity; apply equally to business and management SLRs. Items that refer to software-engineering-specific classification schemes may be skipped if they are not relevant to your discipline.</p>

---

## Part 1: Activities Checklist

The table below lists the 26 actions identified by Petersen et al. (2015) as relevant to a rigorous systematic review or mapping study. Work through each row and mark whether the action was taken (✓), partially taken (~), or not taken (✗). This produces a ratio score: count your ✓ marks and divide by 26 (or by the number of applicable items).

<table id="bkmrk-phase-action-taken%3F-"><thead><tr><th>Phase</th><th>Action</th><th>Taken?</th></tr></thead><tbody><tr><td>**Motivate the review**</td><td>Motivate the need and relevance of the review</td><td> </td></tr><tr><td> </td><td>Define objectives and research questions</td><td> </td></tr><tr><td> </td><td>Consult with the target audience (e.g., supervisor, domain expert) to refine questions</td><td> </td></tr><tr><td>**Search strategy**</td><td>Conduct a database search</td><td> </td></tr><tr><td> </td><td>Apply snowball sampling (backward and/or forward)</td><td> </td></tr><tr><td> </td><td>Conduct a manual search of key journals or conference proceedings</td><td> </td></tr><tr><td>**Develop the search**</td><td>Use a structured framework (PICO, SPIDER, or PCC) to derive keywords</td><td> </td></tr><tr><td> </td><td>Consult a librarian or domain expert during search design</td><td> </td></tr><tr><td> </td><td>Iteratively refine the search string to improve coverage</td><td> </td></tr><tr><td> </td><td>Derive additional keywords from known relevant papers</td><td> </td></tr><tr><td> </td><td>Use thesauri, encyclopedias, or controlled vocabularies (e.g., MeSH, EBSCO subject headings)</td><td> </td></tr><tr><td>**Evaluate the search**</td><td>Test the search against a set of known-relevant papers</td><td> </td></tr><tr><td> </td><td>Have an expert evaluate the search results</td><td> </td></tr><tr><td> </td><td>Check the web pages or profiles of key authors in the field</td><td> </td></tr><tr><td> </td><td>Conduct a test–retest to check consistency</td><td> </td></tr><tr><td>**Inclusion and exclusion**</td><td>Define objective, pre-specified criteria for inclusion and exclusion</td><td> </td></tr><tr><td> </td><td>Involve a second reviewer; resolve disagreements systematically</td><td> </td></tr><tr><td> </td><td>Define and apply explicit decision rules for borderline cases</td><td> </td></tr><tr><td>**Data extraction**</td><td>Define objective criteria for the extraction process</td><td> </td></tr><tr><td> </td><td>Blind or obscure information that could bias extraction</td><td> </td></tr><tr><td> </td><td>Involve a second reviewer; resolve disagreements in extraction</td><td> </td></tr><tr><td> </td><td>Conduct test–retest of extraction on a subset</td><td> </td></tr><tr><td>**Classification**</td><td>Classify studies by research type (e.g., empirical, conceptual, review)</td><td> </td></tr><tr><td> </td><td>Classify studies by research method (e.g., case study, survey, experiment)</td><td> </td></tr><tr><td> </td><td>Classify studies by venue type (e.g., journal, conference, practitioner publication)</td><td> </td></tr><tr><td>**Validity**</td><td>Discuss validity threats and limitations of the review process</td><td> </td></tr></tbody></table>

---

## Part 2: Scoring Rubrics

After completing the checklist, use the rubrics below to assign a score to each of the five key dimensions. Record these scores in your methods chapter alongside a brief narrative.

### Rubric 1: Motivating the Review

<table id="bkmrk-score-label-descript"><thead><tr><th>Score</th><th>Label</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>Not described</td><td>The review is not motivated and no objectives are stated</td></tr><tr><td>1</td><td>Partial</td><td>Motivations and research questions are provided</td></tr><tr><td>2</td><td>Full</td><td>Motivations and questions are provided **and** have been developed in dialogue with the target audience (supervisor, practitioners, or domain experts)</td></tr></tbody></table>

### Rubric 2: Search Strategy

<table id="bkmrk-score-label-descript-1"><thead><tr><th>Score</th><th>Label</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>Not described</td><td>Only one type of search was conducted</td></tr><tr><td>1</td><td>Minimal</td><td>Two search strategies were used</td></tr><tr><td>2</td><td>Full</td><td>All three strategies were used: database search, snowball sampling, and manual search</td></tr></tbody></table>

### Rubric 3: Evaluating the Search

<table id="bkmrk-score-label-descript-2"><thead><tr><th>Score</th><th>Label</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>Not described</td><td>No actions were taken to improve the reliability of the search or inclusion/exclusion process</td></tr><tr><td>1</td><td>Minimal</td><td>At least one action was taken to improve *either* the reliability of the search *or* the inclusion/exclusion process</td></tr><tr><td>2</td><td>Partial</td><td>At least one action was taken to improve *both* the search and the inclusion/exclusion process</td></tr><tr><td>3</td><td>Full</td><td>All identified actions were taken</td></tr></tbody></table>

### Rubric 4: Extraction and Classification

<table id="bkmrk-score-label-descript-3"><thead><tr><th>Score</th><th>Label</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>Not described</td><td>No actions were taken to improve extraction reliability or enable comparability between studies</td></tr><tr><td>1</td><td>Minimal</td><td>At least one action was taken to increase extraction reliability</td></tr><tr><td>2</td><td>Partial</td><td>At least one action to increase extraction reliability **and** studies were classified by research type and method</td></tr><tr><td>3</td><td>Full</td><td>All identified actions were taken</td></tr></tbody></table>

### Rubric 5: Study Validity

<table id="bkmrk-score-label-descript-4"><thead><tr><th>Score</th><th>Label</th><th>Description</th></tr></thead><tbody><tr><td>0</td><td>Not described</td><td>No threats or limitations are described</td></tr><tr><td>1</td><td>Full</td><td>Threats and limitations of the review process are described</td></tr></tbody></table>

---

## Interpreting Your Scores

No minimum threshold is formally established in the literature for general SLRs; the rubric is a diagnostic tool, not a pass/fail gate. Use the results as follows:

- **In your methods chapter:** Report your scores and briefly explain any dimension rated 0 or 1. A low score on a dimension is not automatically a fatal flaw, but it must be acknowledged as a limitation.
- **In your discussion:** Dimensions scored 0 (especially search strategy and validity) should be discussed explicitly when qualifying the strength of your conclusions.
- **As a planning aid:** If you are still in the protocol stage, any action not yet checked is a concrete item to build into your plan before searching begins.

<p class="callout info">For more detail on designing and evaluating your search strategy, see the [Search Quality Self-Assessment Checklist ](https://libguides.berlin-international.de/books/research-skills/page/self-assessment-checklist-for-your-slr-search "Self-Assessment Checklist for Your SLR Search")(adapted from vom Brocke et al., 2015), which provides granular guidance on the search phase specifically.</p>