Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research
The pragmatic (i.e., practical) quality of quantitative implementation measures has received increased attention in the implementation science literature in recent years. Implementation measures that are judged to be pragmatic by implementation stakeholders are thought to be more likely to be applied in research and practice. Despite the need for pragmatic implementation measures, ambiguity and uncertainty regarding what constitutes a pragmatic measure remains. This study sought to identify and critically appraise the published literature to understand (i) how pragmatism is defined as a measurement construct/quality of implementation determinants and outcome instruments; (ii) how pragmatic qualities of instruments are evaluated; (iii) identify key gaps and limitations of the current evidence-base and (iv) identify recommendations for future research. We conducted a scoping review of the literature also employing methods of critical review. PubMed and PsycINFO databases, using the OVID interface, were searched for relevant articles published between January 2010 and September 2020. Articles that contained a definition and/or described characteristics of “pragmatism” as a measurement construct of quantitative implementation outcomes (as defined by Proctor’s Implementation Outcomes taxonomy) and/or implementation determinants were eligible for inclusion. Nine articles met inclusion criteria. A degree of overlap in definitions and terms used to describe the pragmatic qualities of quantitative implementation determinant and outcome instruments were found. The most frequently cited descriptors of pragmatism were “not burdensome”, “brief”, “reliable”, “valid” and “sensitive to change”. 3 of the 9 included articles involved international implementation stakeholders in defining and conceptualizing pragmatism and employed specific methods to do so, including a systematic literature review, stakeholder interviews, concept mapping, and a Delphi process. All other articles defined pragmatism, with or without citing relevant literature. One article objectively assessed the pragmatic qualities, above and beyond the psychometric qualities, of implementation measures, using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The evidence base within the implementation instrumentation literature on what pragmatism is and how it might be assessed is limited. Some of the research identified in the review provides a strong foundation to build upon, by testing its applicability in other settings (including healthcare areas and countries) and among a more diverse group of stakeholders. We discuss directions for further development of the concept of pragmatism relating to the measurement of implementation determinants and outcomes.
The need for pragmatic (i.e., practical) measures to evaluate implementation efforts has been repeatedly called for in the implementation science literature. This literature review focuses on understanding how pragmatism, as a measurement construct and quality of implementation measures, is defined, conceptualized and evaluated. We identified few articles (n = 9) that contribute to our understanding of how pragmatism is defined and evaluated. We found that the most frequently used terms to describe pragmatic qualities of implementation measures include “not burdensome”, “brief”, “reliable”, “valid” and “sensitive to change”. We identified one scale, the Psychometric and Pragmatic Evidence Rating Scale (PAPERS), developed to measure the pragmatic quality, as well as psychometric quality, of implementation measures. We identified several gaps and limitations of the current evidence-base and offer directions to further develop the concept and appraisal of pragmatism. Specifically, we recommend that future research focus on engaging and involving a wider diversity of implementation stakeholders in defining and conceptualizing pragmatism as well as subjecting existing pragmatic assessment measures to more rigorous and extensive evaluation.
Hull, L., Boulton, R., Jones, F., Boaz, A., & Sevdalis, N. (2022). Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research. Transl Behav Med, 12(11), 1049-1064. doi:10.1093/tbm/ibac064