Banner

  • Queen's University Library
  • Research Guides

Introduction to Research: Humanities and Social Sciences

  • Critical Reviews
  • Choosing Your Research Topic
  • Finding Background Information
  • Choosing Relevant Keywords
  • Connecting Keywords
  • Citation Searching
  • Articles (General)
  • Scholarly Articles
  • Films & Videos
  • Scholarly vs. Popular Sources
  • Scholarly vs. Non-Scholarly Periodicals
  • Web Resources
  • Annotated Bibliography
  • Reflective Writing
  • Citing Sources
  • Off Campus?

Writing Critical Reviews

What is a Critical Review of a Journal Article?

A critical review of a journal article evaluates the strengths and weaknesses of an article's ideas and content. It provides description, analysis and interpretation that allow readers to assess the article's value.

Before You Read the Article

  • What does the title lead you to expect about the article?
  • Study any sub-headings to understand how the author organized the content.
  • Read the abstract for a summary of the author's arguments.
  • Study the list of references to determine what research contributed to the author's arguments. Are the references recent? Do they represent important work in the field?
  • If possible, read about the author to learn what authority he or she has to write about the subject.
  • Consult Web of Science to see if other writers have cited the author's work. (Please see 'How to use E-Indexes'.) Has the author made an important contribution to the field of study?

Reading the Article: Points to Consider

Read the article carefully. Record your impressions and note sections suitable for quoting.

  • Who is the intended audience?
  • What is the author's purpose? To survey and summarize research on a topic? To present an argument that builds on past research? To refute another writer's argument?
  • Does the author define important terms?
  • Is the information in the article fact or opinion? (Facts can be verified, while opinions arise from interpretations of facts.) Does the information seem well-researched or is it unsupported?
  • What are the author's central arguments or conclusions? Are they clearly stated? Are they supported by evidence and analysis?
  • If the article reports on an experiment or study, does the author clearly outline methodology and the expected result?
  • Is the article lacking information or argumentation that you expected to find?
  • Is the article organized logically and easy to follow?
  • Does the writer's style suit the intended audience? Is the style stilted or unnecessarily complicated?
  • Is the author's language objective or charged with emotion and bias?
  • If illustrations or charts are used, are they effective in presenting information?

Prepare an Outline

Read over your notes. Choose a statement that expresses the central purpose or thesis of your review. When thinking of a thesis, consider the author's intentions and whether or not you think those intentions were successfully realized. Eliminate all notes that do not relate to your thesis. Organize your remaining points into separate groups such as points about structure, style, or argument. Devise a logical sequence for presenting these ideas. Remember that all of your ideas must support your central thesis.

Write the First Draft

The review should begin with a complete citation of the article. For example:

Platt, Kevin M.F. "History and Despotism, or: Hayden White vs. Ivan the Terrible  and Peter the Great." Rethinking History 3:3 (1999) : 247-269.

NOTE: Use the same bibliographic citation format as you would for any bibliography, works cited or reference list. It will follow a standard documentation style such as MLA or APA.

Be sure to ask your instructor which citation style to use. For frequently used style guides consult Queen's University Library's Citing Sources guide.

The first paragraph may contain:

  • a statement of your thesis
  • the author's purpose in writing the article
  • comments on how the article relates to other work on the same subject
  • information about the author's reputation or authority in the field

The body of the review should:

  • state your arguments in support of your thesis
  • follow the logical development of ideas that you mapped out in your outline
  • include quotations from the article which illustrate your main ideas

The concluding paragraph may:

  • summarize your review
  • restate your thesis

Revise the First Draft

Ideally, you should leave your first draft for a day or two before revising. This allows you to gain a more objective perspective on your ideas. Check for the following when revising:

  • grammar and punctuation errors
  • organization, logical development and solid support of your thesis
  • errors in quotations or in references

You may make major revisions in the organization or content of your review during the revision process. Revising can even lead to a radical change in your central thesis.

NOTE: Prepared by University of Toronto Mississauga Library, Hazel McCallion Academic Learning Centre.

  • << Previous: Writing Resources
  • Next: Annotated Bibliography >>

Additional Resources

Writing a Critical Review (Allyson Skene, The Writing Centre, U of Toronto at Scarborough)

The Book Review or Article Critique (Margaret Procter, Writing Support, University of Toronto)

Critical Reviews of Journal Articles (Herbert Coutts, University of Alberta)

Writing a Critical Review (The Writing Centre, Queen's University)

  • Last Updated: Jun 17, 2024 10:52 AM
  • Subjects: Multidisciplinary

Banner

Write a Critical Review of a Scientific Journal Article

What is a scientific journal, what is a critical review, more help with science writing.

  • Analyzing the Text
  • Writing Your Critique

Ask Us: Chat, email, visit or call

Click to chat: contact the library

More writing resources

  • Check out our full list of online writing resources These guides, templates, and videos are designed to help academic writers at various stages of their writing process, including the pre-writing and revising stages.

Get assistance

The library offers a range of helpful services.  All of our appointments are free of charge and confidential.

  • Book an appointment

A scientific journal is a publication intended to further the progress of scientific discovery by reporting novel research. Scientific journals publish both original research articles and review articles.

A critical review is an assessment of an original research article.  Writing a critical review of a journal article can help you improve your research skills. By assessing the work of others, you develop skills as a critical reader and become familiar with the types of evaluation criteria that will be applied to research in your field.

You are expected to read the article carefully, analyze it, and evaluate the quality and originality of the research, as well as its relevance and presentation. You should assess its strengths and weaknesses, followed by its overall value.

  • Do not be confused by the term critique: it does not mean that you only look at the negative aspects of what the researchers have done. You should address both the positive and negative aspects of the journal article.
  • If your instructor has given you specific advice on how to write a critical review, follow that advice. If not, the following steps may help you. 

This guide is divided into two parts. The first part, "Analyzing the Text," outlines the steps involved in evaluating a research article. The second part, "Writing Your Critique," discusses two possible ways to structure your review.​

  • Developing a Research Question + Worksheet Use this worksheet to develop, assess, and refine your research questions. There is also a downloadable PDF version.
  • Research Article Mapping Template This workbook provides writers with a mapping template and fillable worksheets to begin organizing and drafting sections of a research article.
  • Organizing your Research Proposal - Template This 6-page fillable pdf handout provides writers with a template to begin outlining sections of their own research proposal.
  • WriteOnline.ca Case Study Report; Lab Report; Literature Review; Reflective Writing
  • Library Help Videos On YouTube
  • Next: Analyzing the Text >>
  • Last Updated: Jan 11, 2024 12:42 PM
  • URL: https://guides.lib.uoguelph.ca/WriteCriticalReview

Suggest an edit to this guide

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

  • Advanced Search
  • Journal List
  • J Indian Prosthodont Soc
  • v.21(1); Jan-Mar 2021

Critical evaluation of publications

N. gopi chander.

Editor, The Journal of Indian Prosthodontic Society, Chennai, Tamil Nadu, India

An external file that holds a picture, illustration, etc.
Object name is JIPS-21-1-g001.jpg

Critical evaluation is the process of examining the research for the strength or weakness of the findings, validity, relevance, and usefulness of the research findings.[ 1 ] The availability of extensive information and the difficulty in differentiating the relevant information obligate the primary need of critical appraisal. In addition, it establishes superior evidence and increases the application to clinical practice.[ 2 ] More importantly, it differentiates between significant and/or insignificant data in the literature and aids in providing the updated information. The purpose of critical appraisal shall help in informed decision and improve the quality of healthcare provided to patients.[ 1 , 2 , 3 ]

The research data have three possible outcomes – true findings, random variation that occurs due to chance, and biased results due to systematic error.[ 4 ] The true findings can be of positive or negative results, but it shall be highly recognized. The random error or actual result deviation occurs due to the uncontrollable factors such as smaller sample size and confounding factors. The random error does not alter the measured value, but it is an imperfect error caused due to study design inconsistencies. These errors are unpredictable and cannot be repeated again by repeating the analysis. The biased results are deliberate deviation in the study design, methodology, or investigations. The deviations in the result can be due to poor designing, to the methodology, or in the analysis. It will be difficult to differentiate these findings without critical analysis of the literature.[ 5 , 6 ]

There are various guidelines and tools proposed to critically evaluate the literature.[ 7 , 8 , 9 ] Since the scientific literature is in constant evolution, no one guidelines or checklist is considered to be gold standard. Moreover, the appraisal varies with the type of research. The checklist provided by various organizations for designing or structuring manuscripts - case report, reviews, and original research - cannot be combined or generalized for use. Similarly, it varies with the types of study design - randomized clinical trials and observational studies –case–control, cohort, and cross-sectional studies. The methodological guidelines such as consort statements, CARE guidelines, PROSPERO, or Cochrane checklists can significantly aid in the evaluation of different types of research data.[ 10 ] The structured approach and checklists provided by the organizations can be a valuable aid to conduct research as well as critically evaluate the manuscripts. In addition to the guidelines, the simplified checklists proposed by Young and Solomon can be of adjuvant tool in critical assessment of the literature.[ 1 ] It consists of 10 simple rules. That includes relevance of study question, new information to existing literature, type of research question, appropriateness of study design, bias appraisal, adherence of study protocol, hypothesis testing, check or estimation of statistical analysis, validation of conclusion, and identification of conflicts of interest. These checklists along with updated methodological guidelines for different types of study designs can be a valuable tool for critical appraisal of the literature.[ 1 , 10 ]

Most of the tools assess the validity, reliability, bias, and clinical application of the research data. The validity aids in determining the accuracy of the results, and the reliability establishes the consistency of the results. The bias is systemic deviation of results. The bias is of many types: it can be of from the initiation of the study to manuscript publication. Various assessment tools have been proposed to determine the bias. More commonly employed are the GRADE, Grade pro, Newcastle Ottawa, jaded, ROB 2, and ARRIVE 2.[ 11 ] The bias tools vary with the type of study design, and it is significant to use the appropriate tool. The tools assess and grade the quality of bias in the manuscript. These tools are majorly used for evaluating randomized control trial employed for systematic review and meta-analysis but can be suitably employed to different study designs. These tools provide the grading of bias and provide useful data that are essential for clinical application.[ 11 , 12 ]

Rapid appraisal can be done with merit trials/rapid critical appraisal tool.[ 6 ] It is a compressed tool that basically assesses on the validity, reliability, and clinical use of the study. This is a simplified checklist for quicker assessment; however, for more accurate assessment, it is essential to appraise the entire manuscript from introduction till the conclusion. This mandates a detailed check for every component of the literature in accordance to the standard guidelines. In addition, the journal indexing and metrics can play a significant role in estimation. Higher metric journal shall possess more rigorous peer-review process that reduces the significant errors in the manuscript.[ 3 , 4 ]

The major contents to be generally assessed in the introduction of the manuscript are type and contents of research question, justification of purpose/background of the study with articles published in the last 5 years, or older articles that possess significant influences, citations of peer-reviewed journal, defined objective, and hypothesis statement. In methodology, the parameter of appraisal parameters should be on study design, inclusion and exclusion criteria, care in reduction of bias, following the acceptable procedures, control on confounding variables, and valid outcome measures. The result section should be checked for the subject and baseline - demographic, relevant statistical tests, and statistical significance. The discussion should possess adequate literature substantiation for results, study limitations, and declarations on conflicts of interest.[ 6 ]

In the prosthodontic literature, extensive reports of similar nature exist; critical analysis of the literature is a necessary skill to be mastered by researchers and clinicians.[ 10 ] It helps clinicians to make quality evidenced healthcare decisions by extensive evaluation of the literature.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • Published: 31 January 2022

The fundamentals of critically appraising an article

  • Sneha Chotaliya 1  

BDJ Student volume  29 ,  pages 12–13 ( 2022 ) Cite this article

1965 Accesses

Metrics details

Sneha Chotaliya

We are often surrounded by an abundance of research and articles, but the quality and validity can vary massively. Not everything will be of a good quality - or even valid. An important part of reading a paper is first assessing the paper. This is a key skill for all healthcare professionals as anything we read can impact or influence our practice. It is also important to stay up to date with the latest research and findings.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

We are sorry, but there is no personal subscription option available for your country.

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Chambers R, 'Clinical Effectiveness Made Easy', Oxford: Radcliffe Medical Press , 1998

Loney P L, Chambers L W, Bennett K J, Roberts J G and Stratford P W. Critical appraisal of the health research literature: prevalence or incidence of a health problem. Chronic Dis Can 1998; 19 : 170-176.

Brice R. CASP CHECKLISTS - CASP - Critical Appraisal Skills Programme . 2021. Available at: https://casp-uk.net/casp-tools-checklists/ (Accessed 22 July 2021).

White S, Halter M, Hassenkamp A and Mein G. 2021. Critical Appraisal Techniques for Healthcare Literature . St George's, University of London.

Download references

Author information

Authors and affiliations.

Academic Foundation Dentist, London, UK

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sneha Chotaliya .

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Chotaliya, S. The fundamentals of critically appraising an article. BDJ Student 29 , 12–13 (2022). https://doi.org/10.1038/s41406-021-0275-6

Download citation

Published : 31 January 2022

Issue Date : 31 January 2022

DOI : https://doi.org/10.1038/s41406-021-0275-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

critical evaluation of research article example

  • Writing Rules
  • Running Head & Page numbers
  • Using Quotations
  • Citing Sources
  • Reference List
  • General Reference List Principles
  • Structure of the Report

Introduction

  • References & Appendices
  • Unpacking the Assignment Topic
  • Planning and Structuring the Assignment
  • Writing the Assignment
  • Writing Concisely
  • Developing Arguments

Critically Evaluating Research

  • Editing the Assignment
  • Writing in the Third Person
  • Directive Words
  • Before You Submit
  • Cover Sheet & Title Page
  • Academic Integrity
  • Marking Criteria
  • Word Limit Rules
  • Submitting Your Work
  • Writing Effective E-mails
  • Writing Concisely Exercise
  • About Redbook

Some research reports or assessments will require you critically evaluate a journal article or piece of research. Below is a guide with examples of how to critically evaluate research and how to communicate your ideas in writing.

To develop the skill of being able to critically evaluate, when reading research articles in psychology read with an open mind and be active when reading. Ask questions as you go and see if the answers are provided. Initially skim through the article to gain an overview of the problem, the design, methods, and conclusions. Then read for details and consider the questions provided below for each section of a journal article.

  • Did the title describe the study?
  • Did the key words of the title serve as key elements of the article?
  • Was the title concise, i.e., free of distracting or extraneous phrases?
  • Was the abstract concise and to the point?
  • Did the abstract summarise the study’s purpose/research problem, the independent and dependent variables under study, methods, main findings, and conclusions?
  • Did the abstract provide you with sufficient information to determine what the study is about and whether you would be interested in reading the entire article?
  • Was the research problem clearly identified?
  • Is the problem significant enough to warrant the study that was conducted?
  • Did the authors present an appropriate theoretical rationale for the study?
  • Is the literature review informative and comprehensive or are there gaps?
  • Are the variables adequately explained and operationalised?
  • Are hypotheses and research questions clearly stated? Are they directional? Do the author’s hypotheses and/or research questions seem logical in light of the conceptual framework and research problem?
  • Overall, does the literature review lead logically into the Method section?
  • Is the sample clearly described, in terms of size, relevant characteristics (gender, age, SES, etc), selection and assignment procedures, and whether any inducements were used to solicit subjects (payment, subject credit, free therapy, etc)?
  • What population do the subjects represent (external validity)?
  • Are there sufficient subjects to produce adequate power (statistical validity)?
  • Have the variables and measurement techniques been clearly operationalised?
  • Do the measures/instruments seem appropriate as measures of the variables under study (construct validity)?
  • Have the authors included sufficient information about the psychometric properties (eg. reliability and validity) of the instruments?
  • Are the materials used in conducting the study or in collecting data clearly described?
  • Are the study’s scientific procedures thoroughly described in chronological order?
  • Is the design of the study identified (or made evident)?
  • Do the design and procedures seem appropriate in light of the research problem, conceptual framework, and research questions/hypotheses?
  • Are there other factors that might explain the differences between groups (internal validity)?
  • Were subjects randomly assigned to groups so there was no systematic bias in favour of one group? Was there a differential drop-out rate from groups so that bias was introduced (internal validity and attrition)?
  • Were all the necessary control groups used? Were participants in each group treated identically except for the administration of the independent variable?
  • Were steps taken to prevent subject bias and/or experimenter bias, eg, blind or double blind procedures?
  • Were steps taken to control for other possible confounds such as regression to the mean, history effects, order effects, etc (internal validity)?
  • Were ethical considerations adhered to, eg, debriefing, anonymity, informed consent, voluntary participation?
  • Overall, does the method section provide sufficient information to replicate the study?
  • Are the findings complete, clearly presented, comprehensible, and well organised?
  • Are data coding and analysis appropriate in light of the study’s design and hypotheses? Are the statistics reported correctly and fully, eg. are degrees of freedom and p values given?
  • Have the assumptions of the statistical analyses been met, eg. does one group have very different variance to the others?
  • Are salient results connected directly to hypotheses? Are there superfluous results presented that are not relevant to the hypotheses or research question?
  • Are tables and figures clearly labelled? Well-organised? Necessary (non-duplicative of text)?
  • If a significant result is obtained, consider effect size. Is the finding meaningful? If a non-significant result is found, could low power be an issue? Were there sufficient levels of the IV?
  • If necessary have appropriate post-hoc analyses been performed? Were any transformations performed; if so, were there valid reasons? Were data collapsed over any IVs; if so, were there valid reasons? If any data was eliminated, were valid reasons given?

Discussion and Conclusion

  • Are findings adequately interpreted and discussed in terms of the stated research problem, conceptual framework, and hypotheses?
  • Is the interpretation adequate? i.e., does it go too far given what was actually done or not far enough? Are non-significant findings interpreted inappropriately?
  • Is the discussion biased? Are the limitations of the study delineated?
  • Are implications for future research and/or practical application identified?
  • Are the overall conclusions warranted by the data and any limitations in the study? Are the conclusions restricted to the population under study or are they generalised too widely?
  • Is the reference list sufficiently specific to the topic under investigation and current?
  • Are citations used appropriately in the text?

General Evaluation

  • Is the article objective, well written and organised?
  • Does the information provided allow you to replicate the study in all its details?
  • Was the study worth doing? Does the study provide an answer to a practical or important problem? Does it have theoretical importance? Does it represent a methodological or technical advance? Does it demonstrate a previously undocumented phenomenon? Does it explore the conditions under which a phenomenon occurs?

How to turn your critical evaluation into writing

Example from a journal article.

critical evaluation of research article example

  • Subscribe to journal Subscribe
  • Get new issue alerts Get alerts

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

A guide to critical appraisal of evidence

Fineout-Overholt, Ellen PhD, RN, FNAP, FAAN

Ellen Fineout-Overholt is the Mary Coulter Dowdy Distinguished Professor of Nursing at the University of Texas at Tyler School of Nursing, Tyler, Tex.

The author has disclosed no financial relationships related to this article.

Critical appraisal is the assessment of research studies' worth to clinical practice. Critical appraisal—the heart of evidence-based practice—involves four phases: rapid critical appraisal, evaluation, synthesis, and recommendation. This article reviews each phase and provides examples, tips, and caveats to help evidence appraisers successfully determine what is known about a clinical issue. Patient outcomes are improved when clinicians apply a body of evidence to daily practice.

How do nurses assess the quality of clinical research? This article outlines a stepwise approach to critical appraisal of research studies' worth to clinical practice: rapid critical appraisal, evaluation, synthesis, and recommendation. When critical care nurses apply a body of valid, reliable, and applicable evidence to daily practice, patient outcomes are improved.

FU1-4

Critical care nurses can best explain the reasoning for their clinical actions when they understand the worth of the research supporting their practices. In c ritical appraisal , clinicians assess the worth of research studies to clinical practice. Given that achieving improved patient outcomes is the reason patients enter the healthcare system, nurses must be confident their care techniques will reliably achieve best outcomes.

Nurses must verify that the information supporting their clinical care is valid, reliable, and applicable. Validity of research refers to the quality of research methods used, or how good of a job researchers did conducting a study. Reliability of research means similar outcomes can be achieved when the care techniques of a study are replicated by clinicians. Applicability of research means it was conducted in a similar sample to the patients for whom the findings will be applied. These three criteria determine a study's worth in clinical practice.

Appraising the worth of research requires a standardized approach. This approach applies to both quantitative research (research that deals with counting things and comparing those counts) and qualitative research (research that describes experiences and perceptions). The word critique has a negative connotation. In the past, some clinicians were taught that studies with flaws should be discarded. Today, it is important to consider all valid and reliable research informative to what we understand as best practice. Therefore, the author developed the critical appraisal methodology that enables clinicians to determine quickly which evidence is worth keeping and which must be discarded because of poor validity, reliability, or applicability.

Evidence-based practice process

The evidence-based practice (EBP) process is a seven-step problem-solving approach that begins with data gathering (see Seven steps to EBP ). During daily practice, clinicians gather data supporting inquiry into a particular clinical issue (Step 0). The description is then framed as an answerable question (Step 1) using the PICOT question format ( P opulation of interest; I ssue of interest or intervention; C omparison to the intervention; desired O utcome; and T ime for the outcome to be achieved). 1 Consistently using the PICOT format helps ensure that all elements of the clinical issue are covered. Next, clinicians conduct a systematic search to gather data answering the PICOT question (Step 2). Using the PICOT framework, clinicians can systematically search multiple databases to find available studies to help determine the best practice to achieve the desired outcome for their patients. When the systematic search is completed, the work of critical appraisal begins (Step 3). The known group of valid and reliable studies that answers the PICOT question is called the body of evidence and is the foundation for the best practice implementation (Step 4). Next, clinicians evaluate integration of best evidence with clinical expertise and patient preferences and values to determine if the outcomes in the studies are realized in practice (Step 5). Because healthcare is a community of practice, it is important that experiences with evidence implementation be shared, whether the outcome is what was expected or not. This enables critical care nurses concerned with similar care issues to better understand what has been successful and what has not (Step 6).

Critical appraisal of evidence

The first phase of critical appraisal, rapid critical appraisal, begins with determining which studies will be kept in the body of evidence. All valid, reliable, and applicable studies on the topic should be included. This is accomplished using design-specific checklists with key markers of good research. When clinicians determine a study is one they want to keep (a “keeper” study) and that it belongs in the body of evidence, they move on to phase 2, evaluation. 2

In the evaluation phase, the keeper studies are put together in a table so that they can be compared as a body of evidence, rather than individual studies. This phase of critical appraisal helps clinicians identify what is already known about a clinical issue. In the third phase, synthesis, certain data that provide a snapshot of a particular aspect of the clinical issue are pulled out of the evaluation table to showcase what is known. These snapshots of information underpin clinicians' decision-making and lead to phase 4, recommendation. A recommendation is a specific statement based on the body of evidence indicating what should be done—best practice. Critical appraisal is not complete without a specific recommendation. Each of the phases is explained in more detail below.

Phase 1: Rapid critical appraisal . Rapid critical appraisal involves using two tools that help clinicians determine if a research study is worthy of keeping in the body of evidence. The first tool, General Appraisal Overview for All Studies (GAO), covers the basics of all research studies (see Elements of the General Appraisal Overview for All Studies ). Sometimes, clinicians find gaps in knowledge about certain elements of research studies (for example, sampling or statistics) and need to review some content. Conducting an internet search for resources that explain how to read a research paper, such as an instructional video or step-by-step guide, can be helpful. Finding basic definitions of research methods often helps resolve identified gaps.

To accomplish the GAO, it is best to begin with finding out why the study was conducted and how it answers the PICOT question (for example, does it provide information critical care nurses want to know from the literature). If the study purpose helps answer the PICOT question, then the type of study design is evaluated. The study design is compared with the hierarchy of evidence for the type of PICOT question. The higher the design falls within the hierarchy or levels of evidence, the more confidence nurses can have in its finding, if the study was conducted well. 3,4 Next, find out what the researchers wanted to learn from their study. These are called the research questions or hypotheses. Research questions are just what they imply; insufficient information from theories or the literature are available to guide an educated guess, so a question is asked. Hypotheses are reasonable expectations guided by understanding from theory and other research that predicts what will be found when the research is conducted. The research questions or hypotheses provide the purpose of the study.

Next, the sample size is evaluated. Expectations of sample size are present for every study design. As an example, consider as a rule that quantitative study designs operate best when there is a sample size large enough to establish that relationships do not exist by chance. In general, the more participants in a study, the more confidence in the findings. Qualitative designs operate best with fewer people in the sample because these designs represent a deeper dive into the understanding or experience of each person in the study. 5 It is always important to describe the sample, as clinicians need to know if the study sample resembles their patients. It is equally important to identify the major variables in the study and how they are defined because this helps clinicians best understand what the study is about.

The final step in the GAO is to consider the analyses that answer the study research questions or confirm the study hypothesis. This is another opportunity for clinicians to learn, as learning about statistics in healthcare education has traditionally focused on conducting statistical tests as opposed to interpreting statistical tests. Understanding what the statistics indicate about the study findings is an imperative of critical appraisal of quantitative evidence.

The second tool is one of the variety of rapid critical appraisal checklists that speak to validity, reliability, and applicability of specific study designs, which are available at varying locations (see Critical appraisal resources ). When choosing a checklist to implement with a group of critical care nurses, it is important to verify that the checklist is complete and simple to use. Be sure to check that the checklist has answers to three key questions. The first question is: Are the results of the study valid? Related subquestions should help nurses discern if certain markers of good research design are present within the study. For example, identifying that study participants were randomly assigned to study groups is an essential marker of good research for a randomized controlled trial. Checking these essential markers helps clinicians quickly review a study to check off these important requirements. Clinical judgment is required when the study lacks any of the identified quality markers. Clinicians must discern whether the absence of any of the essential markers negates the usefulness of the study findings. 6-9

TU1

The second question is: What are the study results? This is answered by reviewing whether the study found what it was expecting to and if those findings were meaningful to clinical practice. Basic knowledge of how to interpret statistics is important for understanding quantitative studies, and basic knowledge of qualitative analysis greatly facilitates understanding those results. 6-9

The third question is: Are the results applicable to my patients? Answering this question involves consideration of the feasibility of implementing the study findings into the clinicians' environment as well as any contraindication within the clinicians' patient populations. Consider issues such as organizational politics, financial feasibility, and patient preferences. 6-9

When these questions have been answered, clinicians must decide about whether to keep the particular study in the body of evidence. Once the final group of keeper studies is identified, clinicians are ready to move into the phase of critical appraisal. 6-9

Phase 2: Evaluation . The goal of evaluation is to determine how studies within the body of evidence agree or disagree by identifying common patterns of information across studies. For example, an evaluator may compare whether the same intervention is used or if the outcomes are measured in the same way across all studies. A useful tool to help clinicians accomplish this is an evaluation table. This table serves two purposes: first, it enables clinicians to extract data from the studies and place the information in one table for easy comparison with other studies; and second, it eliminates the need for further searching through piles of periodicals for the information. (See Bonus Content: Evaluation table headings .) Although the information for each of the columns may not be what clinicians consider as part of their daily work, the information is important for them to understand about the body of evidence so that they can explain the patterns of agreement or disagreement they identify across studies. Further, the in-depth understanding of the body of evidence from the evaluation table helps with discussing the relevant clinical issue to facilitate best practice. Their discussion comes from a place of knowledge and experience, which affords the most confidence. The patterns and in-depth understanding are what lead to the synthesis phase of critical appraisal.

The key to a successful evaluation table is simplicity. Entering data into the table in a simple, consistent manner offers more opportunity for comparing studies. 6-9 For example, using abbreviations versus complete sentences in all columns except the final one allows for ease of comparison. An example might be the dependent variable of depression defined as “feelings of severe despondency and dejection” in one study and as “feeling sad and lonely” in another study. 10 Because these are two different definitions, they need to be different dependent variables. Clinicians must use their clinical judgment to discern that these different dependent variables require different names and abbreviations and how these further their comparison across studies.

TU2

Sample and theoretical or conceptual underpinnings are important to understanding how studies compare. Similar samples and settings across studies increase agreement. Several studies with the same conceptual framework increase the likelihood of common independent variables and dependent variables. The findings of a study are dependent on the analyses conducted. That is why an analysis column is dedicated to recording the kind of analysis used (for example, the name of the statistical analyses for quantitative studies). Only statistics that help answer the clinical question belong in this column. The findings column must have a result for each of the analyses listed; however, in the actual results, not in words. For example, a clinician lists a t -test as a statistic in the analysis column, so a t -value should reflect whether the groups are different as well as probability ( P -value or confidence interval) that reflects statistical significance. The explanation for these results would go in the last column that describes worth of the research to practice. This column is much more flexible and contains other information such as the level of evidence, the studies' strengths and limitations, any caveats about the methodology, or other aspects of the study that would be helpful to its use in practice. The final piece of information in this column is a recommendation for how this study would be used in practice. Each of the studies in the body of evidence that addresses the clinical question is placed in one evaluation table to facilitate the ease of comparing across the studies. This comparison sets the stage for synthesis.

Phase 3: Synthesis . In the synthesis phase, clinicians pull out key information from the evaluation table to produce a snapshot of the body of evidence. A table also is used here to feature what is known and help all those viewing the synthesis table to come to the same conclusion. A hypothetical example table included here demonstrates that a music therapy intervention is effective in reducing the outcome of oxygen saturation (SaO 2 ) in six of the eight studies in the body of evidence that evaluated that outcome (see Sample synthesis table: Impact on outcomes ). Simply using arrows to indicate effect offers readers a collective view of the agreement across studies that prompts action. Action may be to change practice, affirm current practice, or conduct research to strengthen the body of evidence by collaborating with nurse scientists.

When synthesizing evidence, there are at least two recommended synthesis tables, including the level-of-evidence table and the impact-on-outcomes table for quantitative questions, such as therapy or relevant themes table for “meaning” questions about human experience. (See Bonus Content: Level of evidence for intervention studies: Synthesis of type .) The sample synthesis table also demonstrates that a final column labeled synthesis indicates agreement across the studies. Of the three outcomes, the most reliable for clinicians to see with music therapy is SaO 2 , with positive results in six out of eight studies. The second most reliable outcome would be reducing increased respiratory rate (RR). Parental engagement has the least support as a reliable outcome, with only two of five studies showing positive results. Synthesis tables make the recommendation clear to all those who are involved in caring for that patient population. Although the two synthesis tables mentioned are a great start, the evidence may require more synthesis tables to adequately explain what is known. These tables are the foundation that supports clinically meaningful recommendations.

Phase 4: Recommendation . Recommendations are definitive statements based on what is known from the body of evidence. For example, with an intervention question, clinicians should be able to discern from the evidence if they will reliably get the desired outcome when they deliver the intervention as it was in the studies. In the sample synthesis table, the recommendation would be to implement the music therapy intervention across all settings with the population, and measure SaO 2 and RR, with the expectation that both would be optimally improved with the intervention. When the synthesis demonstrates that studies consistently verify an outcome occurs as a result of an intervention, however that intervention is not currently practiced, care is not best practice. Therefore, a firm recommendation to deliver the intervention and measure the appropriate outcomes must be made, which concludes critical appraisal of the evidence.

A recommendation that is off limits is conducting more research, as this is not the focus of clinicians' critical appraisal. In the case of insufficient evidence to make a recommendation for practice change, the recommendation would be to continue current practice and monitor outcomes and processes until there are more reliable studies to be added to the body of evidence. Researchers who use the critical appraisal process may indeed identify gaps in knowledge, research methods, or analyses, for example, that they then recommend studies that would fill in the identified gaps. In this way, clinicians and nurse scientists work together to build relevant, efficient bodies of evidence that guide clinical practice.

Evidence into action

Critical appraisal helps clinicians understand the literature so they can implement it. Critical care nurses have a professional and ethical responsibility to make sure their care is based on a solid foundation of available evidence that is carefully appraised using the phases outlined here. Critical appraisal allows for decision-making based on evidence that demonstrates reliable outcomes. Any other approach to the literature is likely haphazard and may lead to misguided care and unreliable outcomes. 11 Evidence translated into practice should have the desired outcomes and their measurement defined from the body of evidence. It is also imperative that all critical care nurses carefully monitor care delivery outcomes to establish that best outcomes are sustained. With the EBP paradigm as the basis for decision-making and the EBP process as the basis for addressing clinical issues, critical care nurses can improve patient, provider, and system outcomes by providing best care.

Seven steps to EBP

Step 0–A spirit of inquiry to notice internal data that indicate an opportunity for positive change.

Step 1– Ask a clinical question using the PICOT question format.

Step 2–Conduct a systematic search to find out what is already known about a clinical issue.

Step 3–Conduct a critical appraisal (rapid critical appraisal, evaluation, synthesis, and recommendation).

Step 4–Implement best practices by blending external evidence with clinician expertise and patient preferences and values.

Step 5–Evaluate evidence implementation to see if study outcomes happened in practice and if the implementation went well.

Step 6–Share project results, good or bad, with others in healthcare.

Adapted from: Steps of the evidence-based practice (EBP) process leading to high-quality healthcare and best patient outcomes. © Melnyk & Fineout-Overholt, 2017. Used with permission.

Critical appraisal resources

  • The Joanna Briggs Institute http://joannabriggs.org/research/critical-appraisal-tools.html
  • Critical Appraisal Skills Programme (CASP) www.casp-uk.net/casp-tools-checklists
  • Center for Evidence-Based Medicine www.cebm.net/critical-appraisal
  • Melnyk BM, Fineout-Overholt E. Evidence-Based Practice in Nursing and Healthcare: A Guide to Best Practice . 3rd ed. Philadelphia, PA: Wolters Kluwer; 2015.

A full set of critical appraisal checklists are available in the appendices.

Bonus content!

This article includes supplementary online-exclusive material. Visit the online version of this article at www.nursingcriticalcare.com to access this content.

critical appraisal; decision-making; evaluation of research; evidence-based practice; synthesis

  • + Favorites
  • View in Gallery

Readers Of this Article Also Read

Determining the level of evidence: experimental research appraisal, nurse-driven protocols, delirium in critical care patients, recognizing and preventing drug diversion, pulmonary embolism: prevention, recognition, and treatment.

The Tech Edvocate

  • Advertisement
  • Home Page Five (No Sidebar)
  • Home Page Four
  • Home Page Three
  • Home Page Two
  • Icons [No Sidebar]
  • Left Sidbear Page
  • Lynch Educational Consulting
  • My Speaking Page
  • Newsletter Sign Up Confirmation
  • Newsletter Unsubscription
  • Page Example
  • Privacy Policy
  • Protected Content
  • Request a Product Review
  • Shortcodes Examples
  • Terms and Conditions
  • The Edvocate
  • The Tech Edvocate Product Guide
  • Write For Us
  • Dr. Lynch’s Personal Website
  • The Edvocate Podcast
  • Assistive Technology
  • Child Development Tech
  • Early Childhood & K-12 EdTech
  • EdTech Futures
  • EdTech News
  • EdTech Policy & Reform
  • EdTech Startups & Businesses
  • Higher Education EdTech
  • Online Learning & eLearning
  • Parent & Family Tech
  • Personalized Learning
  • Product Reviews
  • Tech Edvocate Awards
  • School Ratings

Awards Season Calendar (2024-2025): Key Dates And Timeline For Oscars, Emmys, Golden Globes, Grammys, SAG And More

Want to vacation like the rich and famous stay at one of these turks and caicos resorts, bob’s burgers actor pleads guilty to interfering with police during capitol riot, chinese self-driving cars have quietly traveled 1.8 million miles on us roads, how i’ve learned to live with a nonexistent working memory, some florida cities are living off red light cameras, how to convert live photos to videos or gifs on your iphone, to elect trump, republicans are hiding their extreme anti-abortion plans, ai stack attack: navigating the generative tech maze, steelseries arctis nova 5x wireless review: the best universal gaming headset, how to write an article review (with sample reviews)  .

critical evaluation of research article example

An article review is a critical evaluation of a scholarly or scientific piece, which aims to summarize its main ideas, assess its contributions, and provide constructive feedback. A well-written review not only benefits the author of the article under scrutiny but also serves as a valuable resource for fellow researchers and scholars. Follow these steps to create an effective and informative article review:

1. Understand the purpose: Before diving into the article, it is important to understand the intent of writing a review. This helps in focusing your thoughts, directing your analysis, and ensuring your review adds value to the academic community.

2. Read the article thoroughly: Carefully read the article multiple times to get a complete understanding of its content, arguments, and conclusions. As you read, take notes on key points, supporting evidence, and any areas that require further exploration or clarification.

3. Summarize the main ideas: In your review’s introduction, briefly outline the primary themes and arguments presented by the author(s). Keep it concise but sufficiently informative so that readers can quickly grasp the essence of the article.

4. Evaluate the strengths and weaknesses: In subsequent paragraphs, assess the strengths and limitations of the article based on factors such as methodology, quality of evidence presented, coherence of arguments, and alignment with existing literature in the field. Be fair and objective while providing your critique.

5. Discuss any implications: Deliberate on how this particular piece contributes to or challenges existing knowledge in its discipline. You may also discuss potential improvements for future research or explore real-world applications stemming from this study.

6. Provide recommendations: Finally, offer suggestions for both the author(s) and readers regarding how they can further build on this work or apply its findings in practice.

7. Proofread and revise: Once your initial draft is complete, go through it carefully for clarity, accuracy, and coherence. Revise as necessary, ensuring your review is both informative and engaging for readers.

Sample Review:

A Critical Review of “The Effects of Social Media on Mental Health”

Introduction:

“The Effects of Social Media on Mental Health” is a timely article which investigates the relationship between social media usage and psychological well-being. The authors present compelling evidence to support their argument that excessive use of social media can result in decreased self-esteem, increased anxiety, and a negative impact on interpersonal relationships.

Strengths and weaknesses:

One of the strengths of this article lies in its well-structured methodology utilizing a variety of sources, including quantitative surveys and qualitative interviews. This approach provides a comprehensive view of the topic, allowing for a more nuanced understanding of the effects of social media on mental health. However, it would have been beneficial if the authors included a larger sample size to increase the reliability of their conclusions. Additionally, exploring how different platforms may influence mental health differently could have added depth to the analysis.

Implications:

The findings in this article contribute significantly to ongoing debates surrounding the psychological implications of social media use. It highlights the potential dangers that excessive engagement with online platforms may pose to one’s mental well-being and encourages further research into interventions that could mitigate these risks. The study also offers an opportunity for educators and policy-makers to take note and develop strategies to foster healthier online behavior.

Recommendations:

Future researchers should consider investigating how specific social media platforms impact mental health outcomes, as this could lead to more targeted interventions. For practitioners, implementing educational programs aimed at promoting healthy online habits may be beneficial in mitigating the potential negative consequences associated with excessive social media use.

Conclusion:

Overall, “The Effects of Social Media on Mental Health” is an important and informative piece that raises awareness about a pressing issue in today’s digital age. Given its minor limitations, it provides valuable

3 Ways to Make a Mini Greenhouse ...

3 ways to teach yourself to play ....

' src=

Matthew Lynch

Related articles more from author.

critical evaluation of research article example

3 Ways to Clean Red Brick

critical evaluation of research article example

3 Ways to Clean a Rusty Razor Blade

critical evaluation of research article example

How to Dry Hair with a Round Brush

critical evaluation of research article example

3 Ways to Clean a Wool Hat

critical evaluation of research article example

3 Ways to Make Ballet Flats Not Hurt Your Feet

critical evaluation of research article example

5 Ways to Fix Bad Breath on the Spot

  • Jump to menu
  • Student Home
  • Accept your offer
  • How to enrol
  • Student ID card
  • Set up your IT
  • Orientation Week
  • Fees & payment
  • Academic calendar
  • Special consideration
  • Transcripts
  • The Nucleus: Student Hub
  • Referencing
  • Essay writing
  • Learning abroad & exchange
  • Professional development & UNSW Advantage
  • Employability
  • Financial assistance
  • International students
  • Equitable learning
  • Postgraduate research
  • Health Service
  • Events & activities
  • Emergencies
  • Volunteering
  • Clubs and societies
  • Accommodation
  • Health services
  • Sport and gym
  • Arc student organisation
  • Security on campus
  • Maps of campus
  • Careers portal
  • Change password

Structure of a Critical Review

Critical reviews, both short (one page) and long (four pages), usually have a similar structure. Check your assignment instructions for formatting and structural specifications. Headings are usually optional for longer reviews and can be helpful for the reader.

Introduction

The length of an introduction is usually one paragraph for a journal article review and two or three paragraphs for a longer book review. Include a few opening sentences that announce the author(s) and the title, and briefly explain the topic of the text. Present the aim of the text and summarise the main finding or key argument. Conclude the introduction with a brief statement of your evaluation of the text. This can be a positive or negative evaluation or, as is usually the case, a mixed response.

Present a summary of the key points along with a limited number of examples. You can also briefly explain the author’s purpose/intentions throughout the text and you may briefly describe how the text is organised. The summary should only make up about a third of the critical review.

The critique should be a balanced discussion and evaluation of the strengths, weakness and notable features of the text. Remember to base your discussion on specific criteria. Good reviews also include other sources to support your evaluation (remember to reference).

You can choose how to sequence your critique. Here are some examples to get you started:

  • Most important to least important conclusions you make about the text.
  • If your critique is more positive than negative, then present the negative points first and the positive last.
  • If your critique is more negative than positive, then present the positive points first and the negative last.
  • If there are both strengths and weakness for each criterion you use, you need to decide overall what your judgement is. For example, you may want to comment on a key idea in the text and have both positive and negative comments. You could begin by stating what is good about the idea and then concede and explain how it is limited in some way. While this example shows a mixed evaluation, overall you are probably being more negative than positive.
  • In long reviews, you can address each criterion you choose in a paragraph, including both negative and positive points. For very short critical reviews (one page or less), where your comments will be briefer, include a paragraph of positive aspects  and another of negative.
  • You can also include recommendations for how the text can be improved in terms of ideas, research approach; theories or frameworks used can also be included in the critique section.

Conclusion & References

This is usually a very short paragraph.

  • Restate your overall opinion of the text.
  • Briefly present recommendations.
  • If necessary, some further qualification or explanation of your judgement can be included. This can help your critique sound fair and reasonable.

If you have used other sources in you review you should also include a list of references at the end of the review.

Summarising and paraphrasing for the critical review

The best way to summarise

  • Scan the text. Look for information that can be deduced from the introduction, conclusion, title, and headings. What do these tell you about the main points of the article?
  • Locate the topic sentences and highlight the main points as you read.
  • Reread the text and make separate notes of the main points. Examples and evidence do not need to be included at this stage. Usually they are used selectively in your critique.

Paraphrasing means putting it into your own words. Paraphrasing offers an alternative to using direct quotations in your summary (and the critique) and can be an efficient way to integrate your summary notes.

The best way to paraphrase

  • Review your summary notes
  • Rewrite them in your own words and in complete sentences
  • Use reporting verbs and phrases, e.g. 'The author describes…', 'Smith argues that …'.
  • Use quotation marks if If you include unique or specialist phrases from the text.

  Next: Some general criteria for evaluating texts

Essay and assignment writing guide.

  • Essay writing basics
  • Essay and assignment planning
  • Answering assignment questions
  • Editing checklist
  • Structure of a critical review
  • General criteria for evaluating
  • Sample extracts
  • Annotated bibliography
  • Reflective writing
  • ^ More support

Term 2, 2024 - Provisional Exam timetable released (T2) 10 Jul 2024

Term 2, 2024 - Final Exam timetable released (T2) 15 Jul 2024

Study Hacks Workshops | All the hacks you need! 28 May – 25 Jul 2024

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base
  • Working with sources
  • Evaluating Sources | Methods & Examples

Evaluating Sources | Methods & Examples

Published on June 2, 2022 by Eoghan Ryan . Revised on May 31, 2023.

The sources you use are an important component of your research. It’s important to evaluate the sources you’re considering using, in order to:

  • Ensure that they’re credible
  • Determine whether they’re relevant to your topic
  • Assess the quality of their arguments

Table of contents

Evaluating a source’s credibility, evaluating a source’s relevance, evaluating a source’s arguments, other interesting articles, frequently asked questions about evaluating sources.

Evaluating the credibility of a source is an important way of sifting out misinformation and determining whether you should use it in your research. Useful approaches include the CRAAP test and lateral reading .

One of the best ways to evaluate source credibility is the CRAAP test . This stands for:

  • Currency: Does the source reflect recent research?
  • Relevance: Is the source related to your research topic?
  • Authority: Is it a respected publication? Is the author an expert in their field?
  • Accuracy: Does the source support its arguments and conclusions with evidence?
  • Purpose: What is the author’s intention?

How you evaluate a source using these criteria will depend on your subject and focus. It’s important to understand the types of sources and how you should use them in your field of research.

Lateral reading

Lateral reading is the act of evaluating the credibility of a source by comparing it to other sources. This allows you to:

  • Verify evidence
  • Contextualize information
  • Find potential weaknesses

If a source is using methods or drawing conclusions that are incompatible with other research in its field, it may not be reliable.

Rather than taking these figures at face value, you decide to determine the accuracy of the source’s claims by cross-checking them with official statistics such as census reports and figures compiled by the Department of Homeland Security’s Office of Immigration Statistics.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

critical evaluation of research article example

Try for free

How you evaluate the relevance of a source will depend on your topic, and on where you are in the research process . Preliminary evaluation helps you to pick out relevant sources in your search, while in-depth evaluation allows you to understand how they’re related.

Preliminary evaluation

As you cannot possibly read every source related to your topic, you can use preliminary evaluation to determine which sources might be relevant. This is especially important when you’re surveying a large number of sources (e.g., in a literature review or systematic review ).

One way to do this is to look at paratextual material, or the parts of a work other than the text itself.

  • Look at the table of contents to determine the scope of the work.
  • Consult the index for key terms or the names of important scholars.

You can also read abstracts , prefaces , introductions , and conclusions . These will give you a clear idea of the author’s intentions, the parameters of the research, and even the conclusions they draw.

Preliminary evaluation is useful as it allows you to:

  • Determine whether a source is worth examining in more depth
  • Quickly move on to more relevant sources
  • Increase the quality of the information you consume

While this preliminary evaluation is an important step in the research process, you should engage with sources more deeply in order to adequately understand them.

In-depth evaluation

Begin your in-depth evaluation with any landmark studies in your field of research, or with sources that you’re sure are related to your research topic.

As you read, try to understand the connections between the sources. Look for:

  • Key debates: What topics or questions are currently influencing research? How does the source respond to these key debates?
  • Major publications or critics: Are there any specific texts or scholars that have greatly influenced the field? How does the source engage with them?
  • Trends: Is the field currently dominated by particular theories or research methods ? How does the source respond to these?
  • Gaps: Are there any oversights or weaknesses in the research?

Even sources whose conclusions you disagree with can be relevant, as they can strengthen your argument by offering alternative perspectives.

Every source should contribute to the debate about its topic by taking a clear position. This position and the conclusions the author comes to should be supported by evidence from direct observation or from other sources.

Most sources will use a mix of primary and secondary sources to form an argument . It is important to consider how the author uses these sources. A good argument should be based on analysis and critique, and there should be a logical relationship between evidence and conclusions.

To assess an argument’s strengths and weaknesses, ask:

  • Does the evidence support the claim?
  • How does the author use evidence? What theories, methods, or models do they use?
  • Could the evidence be used to draw other conclusions? Can it be interpreted differently?
  • How does the author situate their argument in the field? Do they agree or disagree with other scholars? Do they confirm or challenge established knowledge?

Situating a source in relation to other sources ( lateral reading ) can help you determine whether the author’s arguments and conclusions are reliable and how you will respond to them in your own writing.

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

  • ChatGPT vs human editor
  • ChatGPT citations
  • Is ChatGPT trustworthy?
  • Using ChatGPT for your studies
  • What is ChatGPT?
  • Chicago style
  • Paraphrasing

 Plagiarism

  • Types of plagiarism
  • Self-plagiarism
  • Avoiding plagiarism
  • Academic integrity
  • Consequences of plagiarism
  • Common knowledge

As you cannot possibly read every source related to your topic, it’s important to evaluate sources to assess their relevance. Use preliminary evaluation to determine whether a source is worth examining in more depth.

This involves:

  • Reading abstracts , prefaces, introductions , and conclusions
  • Looking at the table of contents to determine the scope of the work
  • Consulting the index for key terms or the names of important scholars

Lateral reading is the act of evaluating the credibility of a source by comparing it with other sources. This allows you to:

A credible source should pass the CRAAP test  and follow these guidelines:

  • The information should be up to date and current.
  • The author and publication should be a trusted authority on the subject you are researching.
  • The sources the author cited should be easy to find, clear, and unbiased.
  • For a web source, the URL and layout should signify that it is trustworthy.

The CRAAP test is an acronym to help you evaluate the credibility of a source you are considering using. It is an important component of information literacy .

The CRAAP test has five main components:

  • Currency: Is the source up to date?
  • Relevance: Is the source relevant to your research?
  • Authority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
  • Accuracy: Is the source supported by evidence? Are the claims cited correctly?
  • Purpose: What was the motive behind publishing this source?

Scholarly sources are written by experts in their field and are typically subjected to peer review . They are intended for a scholarly audience, include a full bibliography, and use scholarly or technical language. For these reasons, they are typically considered credible sources .

Popular sources like magazines and news articles are typically written by journalists. These types of sources usually don’t include a bibliography and are written for a popular, rather than academic, audience. They are not always reliable and may be written from a biased or uninformed perspective, but they can still be cited in some contexts.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Ryan, E. (2023, May 31). Evaluating Sources | Methods & Examples. Scribbr. Retrieved July 8, 2024, from https://www.scribbr.com/working-with-sources/evaluating-sources/

Is this article helpful?

Eoghan Ryan

Eoghan Ryan

Other students also liked, student guide: information literacy | meaning & examples, types of sources explained | examples & tips, what are credible sources & how to spot them | examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

  • Systematic review
  • Open access
  • Published: 24 June 2024

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

  • Laura Ellen Ashcraft   ORCID: orcid.org/0000-0001-9957-0617 1 , 2 ,
  • David E. Goodrich 3 , 4 , 5 ,
  • Joachim Hero 6 ,
  • Angela Phares 3 ,
  • Rachel L. Bachrach 7 , 8 ,
  • Deirdre A. Quinn 3 , 4 ,
  • Nabeel Qureshi 6 ,
  • Natalie C. Ernecoff 6 ,
  • Lisa G. Lederer 5 ,
  • Leslie Page Scheunemann 9 , 10 ,
  • Shari S. Rogal 3 , 11   na1 &
  • Matthew J. Chinman 3 , 4 , 6   na1  

Implementation Science volume  19 , Article number:  43 ( 2024 ) Cite this article

2434 Accesses

19 Altmetric

Metrics details

Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes.

We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms “implementation strategy”, “intervention”, “bundle”, “support”, and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes.

Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness ( n =82; 64%) and Implementation ( n =73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials ( n =99), Conduct Educational Meetings ( n =96), Audit and Provide Feedback ( n =76), and External Facilitation ( n =59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions.

This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

Peer Review reports

Contributions to the literature

While many implementation strategies exist, it has been challenging to compare their effectiveness across a wide range of trial designs and practice settings

This systematic review provides a transdisciplinary evaluation of implementation strategies across population, practice setting, and evidence-based interventions using a standardized taxonomy of strategies and outcomes.

Educational strategies were employed ubiquitously; nineteen other commonly used implementation strategies, including External Facilitation and Audit and Provide Feedback, were associated with positive outcomes in these experimental trials.

This review offers guidance for scholars and practitioners alike in selecting implementation strategies and suggests a roadmap for future evidence generation.

Implementation strategies are “methods or techniques used to enhance the adoption, implementation, and sustainment of evidence-based practices or programs” (EBPs) [ 1 ]. In 2015, the Expert Recommendations for Implementing Change (ERIC) study organized a panel of implementation scientists to compile a standardized set of implementation strategy terms and definitions [ 2 , 3 , 4 ]. These 73 strategies were then organized into nine “clusters” [ 5 ]. The ERIC taxonomy has been widely adopted and further refined [ 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ]. However, much of the evidence for individual or groups of ERIC strategies remains narrowly focused. Prior systematic reviews and meta-analyses have assessed strategy effectiveness, but have generally focused on a specific strategy, (e.g., Audit and Provide Feedback) [ 14 , 15 , 16 ], subpopulation, disease (e.g., individuals living with dementia) [ 16 ], outcome [ 15 ], service setting (e.g., primary care clinics) [ 17 , 18 , 19 ] or geography [ 20 ]. Given that these strategies are intended to have broad applicability, there remains a need to understand how well implementation strategies work across EBPs and settings and the extent to which implementation knowledge is generalizable.

There are challenges in assessing the evidence of implementation strategies across many EBPs, populations, and settings. Heterogeneity in population characteristics, study designs, methods, and outcomes have made it difficult to quantitatively compare which strategies work and under which conditions [ 21 ]. Moreover, there remains significant variability in how researchers operationalize, apply, and report strategies (individually or in combination) and outcomes [ 21 , 22 ]. Still, synthesizing data related to using individual strategies would help researchers replicate findings and better understand possible mediating factors including the cost, timing, and delivery by specific types of health providers or key partners [ 23 , 24 , 25 ]. Such an evidence base would also aid practitioners with implementation planning such as when and how to deploy a strategy for optimal impact.

Building upon previous efforts, we therefore conducted a systematic review to evaluate the level of evidence supporting the ERIC implementation strategies across a broad array of health and human service settings and outcomes, as organized by the evaluation framework, RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) [ 26 , 27 , 28 ]. A secondary aim of this work was to identify patterns in scientific reporting of strategy use that could not only inform reporting standards for strategies but also the methods employed in future. The current study was guided by the following research questions Footnote 1 :

What implementation strategies have been most commonly and rigorously tested in health and human service settings?

Which implementation strategies were commonly paired?

What is the evidence supporting commonly tested implementation strategies?

We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA-P) model [ 29 , 30 , 31 ] to develop and report on the methods for this systematic review (Additional File 1). This study was considered to be non-human subjects research by the RAND institutional review board.

Registration

The protocol was registered with PROSPERO (PROSPERO 2021 CRD42021235592).

Eligibility criteria

This review sought to synthesize evidence for implementation strategies from research studies conducted across a wide range of health-related settings and populations. Inclusion criteria required studies to: 1) available in English; 2) published between January 1, 2010 and September 20, 2022; 3) based on experimental research (excluded protocols, commentaries, conference abstracts, or proposed frameworks); 4) set in a health or human service context (described below); 5) tested at least one quantitative outcome that could be mapped to the RE-AIM evaluation framework [ 26 , 27 , 28 ]; and 6) evaluated the impact of an implementation strategy that could be classified using the ERIC taxonomy [ 2 , 32 ]. We defined health and human service setting broadly, including inpatient and outpatient healthcare settings, specialty clinics, mental health treatment centers, long-term care facilities, group homes, correctional facilities, child welfare or youth services, aging services, and schools, and required that the focus be on a health outcome. We excluded hybrid type I trials that primarily focused on establishing EBP effectiveness, qualitative studies, studies that described implementation barriers and facilitators without assessing implementation strategy impact on an outcome, and studies not meeting standardized rigor criteria defined below.

Information sources

Our three-pronged search strategy included searching academic databases (i.e., CINAHL, PubMed, and Web of Science for replicability and transparency), seeking recommendations from expert implementation scientists, and assessing existing, relevant systematic reviews and meta-analyses.

Search strategy

Search terms included “implementation strateg*” OR “implementation intervention*” OR “implementation bundl*” OR “implementation support*.” The search, conducted on September 20, 2022, was limited to English language and publication between 2010 and 2022, similar to other recent implementation science reviews [ 22 ]. This timeframe was selected to coincide with the advent of Implementation Science and when the term “implementation strategy” became conventionally used [ 2 , 4 , 33 ]. A full search strategy can be found in Additional File 2.

Title and abstract screening process

Each study’s title and abstract were read by two reviewers, who dichotomously scored studies on each of the six eligibility criteria described above as yes=1 or no=0, resulting in a score ranging from 1 to 6. Abstracts receiving a six from both reviewers were included in the full text review. Those with only one score of six were adjudicated by a senior member of the team (MJC, SSR, DEG). The study team held weekly meetings to troubleshoot and resolve any ongoing issues noted through the abstract screening process.

Full text screening

During the full text screening process, we reviewed, in pairs, each article that had progressed through abstract screening. Conflicts between reviewers were adjudicated by a senior member of the team for a final inclusion decision (MJC, SSR, DEG).

Review of study rigor

After reviewing published rigor screening tools [ 34 , 35 , 36 ], we developed an assessment of study rigor that was appropriate for the broad range of reviewed implementation studies. Reviewers evaluated studies on the following: 1) presence of a concurrent comparison or control group (=2 for traditional randomized controlled trial or stepped wedge cluster randomized trial and =1 for pseudo-randomized and other studies with concurrent control); 2) EBP standardization by protocol or manual (=1 if present); 3) EBP fidelity tracking (=1 if present); 4) implementation strategy standardization by operational description, standard training, or manual (=1 if present); 5) length of follow-up from full implementation of intervention (=2 for twelve months or longer, =1 for six to eleven months, or =0 for less than six months); and 6) number of sites (=1 for more than one site). Rigor scores ranged from 0 to 8, with 8 indicating the most rigorous. Articles were included if they 1) included a concurrent control group, 2) had an experimental design, and 3) received a score of 7 or 8 from two independent reviewers.

Outside expert consultation

We contacted 37 global implementation science experts who were recognized by our study team as leaders in the field or who were commonly represented among first or senior authors in the included abstracts. We asked each expert for recommendations of publications meeting study inclusion criteria (i.e., quantitatively evaluating the effectiveness of an implementation strategy). Recommendations were recorded and compared to the full abstract list.

Systematic reviews

Eighty-four systematic reviews were identified through the initial search strategy (See Additional File 3). Systematic reviews that examined the effectiveness of implementation strategies were reviewed in pairs for studies that were not found through our initial literature search.

Data abstraction and coding

Data from the full text review were abstracted in pairs, with conflicts resolved by senior team members (DEG, MJC) using a standard Qualtrics abstraction form. The form captured the setting, number of sites and participants studied, evidence-based practice/program of focus, outcomes assessed (based on RE-AIM), strategies used in each study arm, whether the study took place in the U.S. or outside of the U.S., and the findings (i.e., was there significant improvement in the outcome(s)?). We coded implementation strategies used in the Control and Experimental Arms. We defined the Control Arm as receiving the lowest number of strategies (which could mean zero strategies or care as usual) and the Experimental Arm as the most intensive arm (i.e., receiving the highest number of strategies). When studies included multiple Experimental Arms, the Experimental Arm with the least intensive implementation strategy(ies) was classified as “Control” and the Experimental Arm with the most intensive implementation strategy(ies) was classified as the “Experimental” Arm.

Implementation strategies were classified using standard definitions (MJC, SSR, DEG), based on minor modifications to the ERIC taxonomy [ 2 , 3 , 4 ]. Modifications resulted in 70 named strategies and were made to decrease redundancy and improve clarity. These modifications were based on input from experts, cognitive interview data, and team consensus [ 37 ] (See Additional File 4). Outcomes were then coded into RE-AIM outcome domains following best practices as recommended by framework experts [ 26 , 27 , 28 ]. We coded the RE-AIM domain of Effectiveness as either an assessment of the effectiveness of the EBP or the implementation strategy. We did not assess implementation strategy fidelity or effects on health disparities as these are recently adopted reporting standards [ 27 , 28 ] and not yet widely implemented in current publications. Further, we did not include implementation costs as an outcome because reporting guidelines have not been standardized [ 38 , 39 ].

Assessment and minimization of bias

Assessment and minimization of bias is an important component of high-quality systematic reviews. The Cochrane Collaboration guidance for conducting high-quality systematic reviews recommends including a specific assessment of bias for individual studies by assessing the domains of randomization, deviations of intended intervention, missing data, measurement of the outcome, and selection of the reported results (e.g., following a pre-specified analysis plan) [ 40 , 41 ]. One way we addressed bias was by consolidating multiple publications from the same study into a single finding (i.e., N =1), so-as to avoid inflating estimates due to multiple publications on different aspects of a single trial. We also included high-quality studies only, as described above. However, it was not feasible to consistently apply an assessment of bias tool due to implementation science’s broad scope and the heterogeneity of study design, context, outcomes, and variable measurement, etc. For example, most implementation studies reviewed had many outcomes across the RE-AIM framework, with no one outcome designated as primary, precluding assignment of a single score across studies.

We used descriptive statistics to present the distribution of health or healthcare area, settings, outcomes, and the median number of included patients and sites per study, overall and by country (classified as U.S. vs. non-U.S.). Implementation strategies were described individually, using descriptive statistics to summarize the frequency of strategy use “overall” (in any study arm), and the mean number of strategies reported in the Control and Experimental Arms. We additionally described the strategies that were only in the experimental (and not control) arm, defining these as strategies that were “tested” and may have accounted for differences in outcomes between arms.

We described frequencies of pair-wise combinations of implementation strategies in the Experimental Arm. To assess the strength of the evidence supporting implementation strategies that were used in the Experimental Arm, study outcomes were categorized by RE-AIM and coded based on whether the association between use of the strategies resulted in a significantly positive effect (yes=1; no=0). We then created an indicator variable if at least one RE-AIM outcome in the study was significantly positive (yes=1; no=0). We plotted strategies on a graph with quadrants based on the combination of median number of studies in which a strategy appears and the median percent of studies in which a strategy was associated with at least one positive RE-AIM outcome. The upper right quadrant—higher number of studies overall and higher percent of studies with a significant RE-AIM outcome—represents a superior level of evidence. For implementation strategies in the upper right quadrant, we describe each RE-AIM outcome and the proportion of studies which have a significant outcome.

Search results

We identified 14,646 articles through the initial literature search, 17 articles through expert recommendation (three of which were not included in the initial search), and 1,942 articles through reviewing prior systematic reviews (Fig. 1 ). After removing duplicates, 9,399 articles were included in the initial abstract screening. Of those, 48% ( n =4,075) abstracts were reviewed in pairs for inclusion. Articles with a score of five or six were reviewed a second time ( n =2,859). One quarter of abstracts that scored lower than five were reviewed for a second time at random. We screened the full text of 1,426 articles in pairs. Common reasons for exclusion were 1) study rigor, including no clear delineation between the EBP and implementation strategy, 2) not testing an implementation strategy, and 3) article type that did not meet inclusion criteria (e.g., commentary, protocol, etc.). Six hundred seventeen articles were reviewed for study rigor with 385 excluded for reasons related to study design and rigor, and 86 removed for other reasons (e.g., not a research article). Among the three additional expert-recommended articles, one met inclusion criteria and was added to the analysis. The final number of studies abstracted was 129 representing 143 publications.

figure 1

Expanded PRISMA Flow Diagram

The expanded PRISMA flow diagram provides a description of each step in the review and abstraction process for the systematic review

Descriptive results

Of 129 included studies (Table 1 ; see also Additional File 5 for Summary of Included Studies), 103 (79%) were conducted in a healthcare setting. EBP health care setting varied and included primary care ( n =46; 36%), specialty care ( n =27; 21%), mental health ( n =11; 9%), and public health ( n =30; 23%), with 64 studies (50%) occurring in an outpatient health care setting. Studies included a median of 29 sites and 1,419 target population (e.g., patients or students). The number of strategies varied widely across studies, with Control Arms averaging approximately two strategies (Range = 0-20, including studies with no strategy in the comparison group) and Experimental Arms averaging eight strategies (Range = 1-21). Non-US studies ( n =73) included more sites and target population on average, with an overall median of 32 sites and 1,531 patients assessed in each study.

Organized by RE-AIM, the most evaluated outcomes were Effectiveness ( n = 82, 64%) and Implementation ( n = 73, 56%); followed by Maintenance ( n =40; 31%), Adoption ( n =33; 26%), and Reach ( n =31; 24%). Most studies ( n = 98, 76%) reported at least one significantly positive outcome. Adoption and Implementation outcomes showed positive change in three-quarters of studies ( n =78), while Reach ( n =18; 58%), Effectiveness ( n =44; 54%), and Maintenance ( n =23; 58%) outcomes evidenced positive change in approximately half of studies.

The following describes the results for each research question.

Table 2 shows the frequency of studies within which an implementation strategy was used in the Control Arm, Experimental Arm(s), and tested strategies (those used exclusively in the Experimental Arm) grouped by strategy type, as specified by previous ERIC reports [ 2 , 6 ].

Control arm

In about half the studies (53%; n =69), the Control Arms were “active controls” that included at least one strategy, with an average of 1.64 (and up to 20) strategies reported in control arms. The two most common strategies used in Control Arms were: Distribute Educational Materials ( n =52) and Conduct Educational Meetings ( n =30).

Experimental arm

Experimental conditions included an average of 8.33 implementation strategies per study (Range = 1-21). Figure 2 shows a heat map of the strategies that were used in the Experimental Arms in each study. The most common strategies in the Experimental Arm were Distribute Educational Materials ( n =99), Conduct Educational Meetings ( n =96), Audit and Provide Feedback ( n =76), and External Facilitation ( n =59).

figure 2

Implementation strategies used in the Experimental Arm of included studies. Explore more here: https://public.tableau.com/views/Figure2_16947070561090/Figure2?:language=en-US&:display_count=n&:origin=viz_share_link

Tested strategies

The average number of implementation strategies that were included in the Experimental Arm only (and not in the Control Arm) was 6.73 (Range = 0-20). Footnote 2 Overall, the top 10% of tested strategies included Conduct Educational Meetings ( n =68), Audit and Provide Feedback ( n =63), External Facilitation ( n =54), Distribute Educational Materials ( n =49), Tailor Strategies ( n =41), Assess for Readiness and Identify Barriers and Facilitators ( n =38) and Organize Clinician Implementation Team Meetings ( n =37). Few studies tested a single strategy ( n =9). These strategies included, Audit and Provide Feedback, Conduct Educational Meetings, Conduct Ongoing Training, Create a Learning Collaborative, External Facilitation ( n =2), Facilitate Relay of Clinical Data To Providers, Prepare Patients/Consumers to be Active Participants, and Use Other Payment Schemes. Three implementation strategies were included in the Control or Experimental Arms but were not Tested including, Use Mass Media, Stage Implementation Scale Up, and Fund and Contract for the Clinical Innovation.

Table 3  shows the five most used strategies in Experimental Arms with their top ten most frequent pairings, excluding Distribute Educational Materials and Conduct Educational Meetings, as these strategies were included in almost all Experimental and half of Control Arms. The five most used strategies in the Experimental Arm included Audit and Provide Feedback ( n =76), External Facilitation ( n =59), Tailor Strategies ( n =43), Assess for Readiness and Identify Barriers and Facilitators ( n =43), and Organize Implementation Teams ( n =42).

Strategies frequently paired with these five strategies included two educational strategies: Distribute Educational Materials and Conduct Educational Meetings. Other commonly paired strategies included Develop a Formal Implementation Blueprint, Promote Adaptability, Conduct Ongoing Training, Purposefully Reexamine the Implementation, and Develop and Implement Tools for Quality Monitoring.

We classified the strength of evidence for each strategy by evaluating both the number of studies in which each strategy appeared in the Experimental Arm and the percentage of times there was at least one significantly positive RE-AIM outcome. Using these factors, Fig. 3 shows the number of studies in which individual strategies were evaluated (on the y axis) compared to the percentage of times that studies including those strategies had at least one positive outcome (on the x axis). Due to the non-normal distribution of both factors, we used the median (rather than the mean) to create four quadrants. Strategies in the lower left quadrant were tested in fewer than the median number of studies (8.5) and were less frequently associated with a significant RE-AIM outcome (75%). The upper right quadrant included strategies that occurred in more than the median number of studies (8.5) and had more than the median percent of studies with a significant RE-AIM outcome (75%); thus those 19 strategies were viewed as having stronger evidence. Of those 19 implementation strategies, Conduct Educational Meetings, Distribute Educational Materials, External Facilitation, and Audit and Provide Feedback continued to occur frequently, appearing in 59-99 studies.

figure 3

Experimental Arm Implementation Strategies with significant RE-AIM outcome. Explore more here: https://public.tableau.com/views/Figure3_16947017936500/Figure3?:language=en-US&publish=yes&:display_count=n&:origin=viz_share_link

Figure 4 graphically illustrates the proportion of significant outcomes for each RE-AIM outcome for the 19 commonly used and evidence-based implementation strategies in the upper right quadrant. These findings again show the widespread use of Conduct Educational Meetings and Distribute Educational Materials. Implementation and Effectiveness outcomes were assessed most frequently, with Implementation being the mostly commonly reported significantly positive outcome.

figure 4

RE-AIM outcomes for the 19 Top-Right Quadrant Implementation Strategies . The y-axis is the number of studies and the x-axis is a stacked bar chart for each RE-AIM outcome with R=Reach, E=Effectiveness, A=Adoption, I=Implementation, M=Maintenance. Blue denotes at least one significant RE-AIM outcome; Light blue denotes studies which used the given implementation strategy and did not have a significant RE-AIM . Explore more here: https://public.tableau.com/views/Figure4_16947017112150/Figure4?:language=en-US&publish=yes&:display_count=n&:origin=viz_share_link

This systematic review identified 129 experimental studies examining the effectiveness of implementation strategies across a broad range of health and human service studies. Overall, we found that evidence is lacking for most ERIC implementation strategies, that most studies employed combinations of strategies, and that implementation outcomes, categorized by RE-AIM dimensions, have not been universally defined or applied. Accordingly, other researchers have described the need for universal outcomes definitions and descriptions across implementation research studies [ 28 , 42 ]. Our findings have important implications not only for the current state of the field but also for creating guidance to help investigators determine which strategies and in what context to examine.

The four most evaluated strategies were Distribute Educational Materials, Conduct Educational Meetings, External Facilitation, and Audit and Provide Feedback. Conducting Educational Meetings and Distributing Educational Materials were surprisingly the most common. This may reflect the fact that education strategies are generally considered to be “necessary but not sufficient” for successful implementation [ 43 , 44 ]. Because education is often embedded in interventions, it is critical to define the boundary between the innovation and the implementation strategies used to support the innovation. Further specification as to when these strategies are EBP core components or implementation strategies (e.g., booster trainings or remediation) is needed [ 45 , 46 ].

We identified 19 implementation strategies that were tested in at least 8 studies (more than the median) and were associated with positive results at least 75% of the time. These strategies can be further categorized as being used in early or pre-implementation versus later in implementation. Preparatory activities or pre-implementation, strategies that had strong evidence included educational activities (Meetings, Materials, Outreach visits, Train for Leadership, Use Train the Trainer Strategies) and site diagnostic activities (Assess for Readiness, Identify Barriers and Facilitators, Conduct Local Needs Assessment, Identify and Prepare Champions, and Assess and Redesign Workflows). Strategies that target the implementation phase include those that provide coaching and support (External and Internal Facilitation), involve additional key partners (Intervene with Patients to Enhance Uptake and Adherence), and engage in quality improvement activities (Audit and Provide Feedback, Facilitate the Relay of Clinical Data to Providers, Purposefully Reexamine the Implementation, Conduct Cyclical Small Tests of Change, Develop and Implement Tools for Quality Monitoring).

There were many ERIC strategies that were not represented in the reviewed studies, specifically the financial and policy strategies. Ten strategies were not used in any studies, including: Alter Patient/Consumer Fees, Change Liability Laws, Change Service Sites, Develop Disincentives, Develop Resource Sharing Agreements, Identify Early Adopters, Make Billing Easier, Start a Dissemination Organization, Use Capitated Payments, and Use Data Experts. One of the limitations of this investigation was that not all individual strategies or combinations were investigated. Reasons for the absence of these strategies in our review may include challenges with testing certain strategies experimentally (e.g., changing liability laws), limitations in our search terms, and the relative paucity of implementation strategy trials compared to clinical trials. Many “untested” strategies require large-scale structural changes with leadership support (see [ 47 ] for policy experiment example). Recent preliminary work has assessed the feasibility of applying policy strategies and described the challenges with doing so [ 48 , 49 , 50 ]. While not impossible in large systems like VA (for example: the randomized evaluation of the VA Stratification Tool for Opioid Risk Management) the large size, structure, and organizational imperative makes these initiatives challenging to experimentally evaluate. Likewise, the absence of these ten strategies may have been the result of our inclusion criteria, which required an experimental design. Thus, creative study designs may be needed to test high-level policy or financial strategies experimentally.

Some strategies that were likely under-represented in our search strategy included electronic medical record reminders and clinical decision support tools and systems. These are often considered “interventions” when used by clinical trialists and may not be indexed as studies involving ‘implementation strategies’ (these tools have been reviewed elsewhere [ 51 , 52 , 53 ]). Thus, strategies that are also considered interventions in the literature (e.g., education interventions) were not sought or captured. Our findings do not imply that these strategies are ineffective, rather that more study is needed. Consistent with prior investigations [ 54 ], few studies meeting inclusion criteria tested financial strategies. Accordingly, there are increasing calls to track and monitor the effects of financial strategies within implementation science to understand their effectiveness in practice [ 55 , 56 ]. However, experts have noted that the study of financial strategies can be a challenge given that they are typically implemented at the system-level and necessitate research designs for studying policy-effects (e.g., quasi-experimental methods, systems-science modeling methods) [ 57 ]. Yet, there have been some recent efforts to use financial strategies to support EBPs that appear promising [ 58 ] and could be a model for the field moving forward.

The relationship between the number of strategies used and improved outcomes has been described inconsistently in the literature. While some studies have found improved outcomes with a bundle of strategies that were uniquely combined or a standardized package of strategies (e.g., Replicating Effective Programs [ 59 , 60 ] and Getting To Outcomes [ 61 , 62 ]), others have found that “more is not always better” [ 63 , 64 , 65 ]. For example, Rogal and colleagues documented that VA hospitals implementing a new evidence-based hepatitis C treatment chose >20 strategies, when multiple years of data linking strategies to outcomes showed that 1-3 specific strategies would have yielded the same outcome [ 39 ]. Considering that most studies employed multiple or multifaceted strategies, it seems that there is a benefit of using a targeted bundle of strategies that are purposefully aligns with site/clinic/population norms, rather than simply adding more strategies [ 66 ].

It is difficult to assess the effectiveness of any one implementation strategy in bundles where multiple strategies are used simultaneously. Even a ‘single’ strategy like External Facilitation is, in actuality, a bundle of narrowly constructed strategies (e.g., Conduct Educational Meetings, Identify and Prepare Champions, and Develop a Formal Implementation Blueprint). Thus, studying External Facilitation does not allow for a test of the individual strategies that comprise it, potentially masking the effectiveness of any individual strategy. While we cannot easily disaggregate the effects of multifaceted strategies, doing so may not yield meaningful results. Because strategies often synergize, disaggregated results could either underestimate the true impact of individual strategies or conversely, actually undermine their effectiveness (i.e., when their effectiveness comes from their combination with other strategies). The complexity of health and human service settings, imperative to improve public health outcomes, and engagement with community partners often requires the use of multiple strategies simultaneously. Therefore, the need to improve real-world implementation may outweigh the theoretical need to identify individual strategy effectiveness. In situations where it would be useful to isolate the impact of single strategies, we suggest that the same methods for documenting and analyzing the critical components (or core functions) of complex interventions [ 67 , 68 , 69 , 70 ] may help to identify core components of multifaceted implementation strategies [ 71 , 72 , 73 , 74 ].

In addition, to truly assess the impacts of strategies on outcomes, it may be necessary to track fidelity to implementation strategies (not just the EBPs they support). While this can be challenging, without some degree of tracking and fidelity checks, one cannot determine whether a strategy’s apparent failure to work was because it 1) was ineffective or 2) was not applied well. To facilitate this tracking there are pragmatic tools to support researchers. For example, the Longitudinal Implementation Strategy Tracking System (LISTS) offers a pragmatic and feasible means to assess fidelity to and adaptations of strategies [ 75 ].

Implications for implementation science: four recommendations

Based on our findings, we offer four recommended “best practices” for implementation studies.

Prespecify strategies using standard nomenclature. This study reaffirmed the need to apply not only a standard naming convention (e.g., ERIC) but also a standard reporting of for implementation strategies. While reporting systems like those by Proctor [ 1 ] or Pinnock [ 75 ] would optimize learning across studies, few manuscripts specify strategies as recommended [ 76 , 77 ]. Pre-specification allows planners and evaluators to assess the feasibility and acceptability of strategies with partners and community members [ 24 , 78 , 79 ] and allows evaluators and implementers to monitor and measure the fidelity, dose, and adaptations to strategies delivered over the course of implementation [ 27 ]. In turn, these data can be used to assess the costs, analyze their effectiveness [ 38 , 80 , 81 ], and ensure more accurate reporting [ 82 , 83 , 84 , 85 ]. This specification should include, among other data, the intensity, stage of implementation, and justification for the selection. Information regarding why strategies were selected for specific settings would further the field and be of great use to practitioners. [ 63 , 65 , 69 , 79 , 86 ].

Ensure that standards for measuring and reporting implementation outcomes are consistently applied and account for the complexity of implementation studies. Part of improving standardized reporting must include clearly defining outcomes and linking each outcome to particular implementation strategies. It was challenging in the present review to disentangle the impact of the intervention(s) (i.e., the EBP) versus the impact of the implementation strategy(ies) for each RE-AIM dimension. For example, often fidelity to the EBP was reported but not for the implementation strategies. Similarly, Reach and Adoption of the intervention would be reported for the Experimental Arm but not for the Control Arm, prohibiting statistical comparisons of strategies on the relative impact of the EBP between study arms. Moreover, there were many studies evaluating numerous outcomes, risking data dredging. Further, the significant heterogeneity in the ways in which implementation outcomes are operationalized and reported is a substantial barrier to conducting large-scale meta-analytic approaches to synthesizing evidence for implementation strategies [ 67 ]. The field could look to others in the social and health sciences for examples in how to test, validate, and promote a common set of outcome measures to aid in bringing consistency across studies and real-world practice (e.g., the NIH-funded Patient-Reported Outcomes Measurement Information System [PROMIS], https://www.healthmeasures.net/explore-measurement-systems/promis ).

Develop infrastructure to learn cross-study lessons in implementation science. Data repositories, like those developed by NCI for rare diseases, U.S. HIV Implementation Science Coordination Initiative [ 87 ], and the Behavior Change Technique Ontology [ 88 ], could allow implementation scientists to report their findings in a more standardized manner, which would promote ease of communication and contextualization of findings across studies. For example, the HIV Implementation Science Coordination Initiative requested all implementation projects use common frameworks, developed user friendly databases to enable practitioners to match strategies to determinants, and developed a dashboard of studies that assessed implementation determinants [ 89 , 90 , 91 , 92 , 93 , 94 ].

Develop and apply methods to rigorously study common strategies and bundles. These findings support prior recommendations for improved empirical rigor in implementation studies [ 46 , 95 ]. Many studies were excluded from our review based on not meeting methodological rigor standards. Understanding the effectiveness of discrete strategies deployed alone or in combination requires reliable and low burden tracking methods to collect information about strategy use and outcomes. For example, frameworks like the Implementation Replication Framework [ 96 ] could help interpret findings across studies using the same strategy bundle. Other tracking approaches may leverage technology (e.g., cell phones, tablets, EMR templates) [ 78 , 97 ] or find novel, pragmatic approaches to collect recommended strategy specifications over time (e.g.., dose, deliverer, and mechanism) [ 1 , 9 , 27 , 98 , 99 ]. Rigorous reporting standards could inform more robust analyses and conclusions (e.g., moving toward the goal of understanding causality, microcosting efforts) [ 24 , 38 , 100 , 101 ]. Such detailed tracking is also required to understand how site-level factors moderate implementation strategy effects [ 102 ]. In some cases, adaptive trial designs like sequential multiple assignment randomized trials (SMARTs) and just-in-time adaptive interventions (JITAIs) can be helpful for planning strategy escalation.

Limitations

Despite the strengths of this review, there were certain notable limitations. For one, we only included experimental studies, omitting many informative observational investigations that cover the range of implementation strategies. Second, our study period was centered on the creation of the journal Implementation Science and not on the standardization and operationalization of implementation strategies in the publication of the ERIC taxonomy (which came later). This, in conjunction with latency in reporting study results and funding cycles, means that the employed taxonomy was not applied in earlier studies. To address this limitation, we retroactively mapped strategies to ERIC, but it is possible that some studies were missed. Additionally, indexing approaches used by academic databases may have missed relevant studies. We addressed this particular concern by reviewing other systematic reviews of implementation strategies and soliciting recommendations from global implementation science experts.

Another potential limitation comes from the ERIC taxonomy itself—i.e., strategy listings like ERIC are only useful when they are widely adopted and used in conjunction with guidelines for specifying and reporting strategies [ 1 ] in protocol and outcome papers. Although the ERIC paper has been widely cited (over three thousand times, accessed about 186 thousand times), it is still not universally applied, making tracking the impact of specific strategies more difficult. However, our experience with this review seemed to suggest that ERIC’s use was increasing over time. Also, some have commented that ERIC strategies can be unclear and are missing key domains. Thus, researchers are making definitions clearer for lay users [ 37 , 103 ], increasing the number of discrete strategies for specific domains like HIV treatment, acknowledging strategies for new functions (e.g., de-implementation [ 104 ], local capacity building), accounting for phases of implementation (dissemination, sustainment [ 13 ], scale-up), addressing settings [ 12 , 20 ], actors roles in the process, and making mechanisms of change to select strategies more user-friendly through searchable databases [ 9 , 10 , 54 , 73 , 104 , 105 , 106 ]. In sum, we found the utility of the ERIC taxonomy to outweigh any of the taxonomy’s current limitations.

As with all reviews, the search terms influenced our findings. As such, the broad terms for implementation strategies (e.g., “evidence-based interventions”[ 7 ] or “behavior change techniques” [ 107 ]) may have led to inadvertent omissions of studies of specific strategies. For example, the search terms may not have captured tests of policies, financial strategies, community health promotion initiatives, or electronic medical record reminders, due to differences in terminology used in corresponding subfields of research (e.g., health economics, business, health information technology, and health policy). To manage this, we asked experts to inform us about any studies that they would include and cross-checked their lists with what was identified through our search terms, which yielded very few additional studies. We included standard coding using the ERIC taxonomy, which was a strength, but future work should consider including the additional strategies that have been recommended to augment ERIC, around sustainment [ 13 , 79 , 106 , 108 ], community and public health research [ 12 , 109 , 110 , 111 ], consumer or service user engagement [ 112 ], de-implementation [ 104 , 113 , 114 , 115 , 116 , 117 ] and related terms [ 118 ].

We were unable to assess the bias of studies due to non-standard reporting across the papers and the heterogeneity of study designs, measurement of implementation strategies and outcomes, and analytic approaches. This could have resulted in over- or underestimating the results of our synthesis. We addressed this limitation by being cautious in our reporting of findings, specifically in identifying “effective” implementation strategies. Further, we were not able to gather primary data to evaluate effect sizes across studies in order to systematically evaluate bias, which would be fruitful for future study.

Conclusions

This novel review of 129 studies summarized the body of evidence supporting the use of ERIC-defined implementation strategies to improve health or healthcare. We identified commonly occurring implementation strategies, frequently used bundles, and the strategies with the highest degree of supportive evidence, while simultaneously identifying gaps in the literature. Additionally, we identified several key areas for future growth and operationalization across the field of implementation science with the goal of improved reporting and assessment of implementation strategies and related outcomes.

Availability and materials

All data for this study are included in this published article and its supplementary information files.

We modestly revised the following research questions from our PROSPERO registration after reading the articles and better understanding the nature of the literature: 1) What is the available evidence regarding the effectiveness of implementation strategies in supporting the uptake and sustainment of evidence intended to improve health and healthcare outcomes? 2) What are the current gaps in the literature (i.e., implementation strategies that do not have sufficient evidence of effectiveness) that require further exploration?

Tested strategies are those which exist in the Experimental Arm but not in the Control Arm. Comparative effectiveness or time staggered trials may not have any unique strategies in the Experimental Arm and therefore in our analysis would have no Tested Strategies.

Abbreviations

Centers for Disease Control

Cumulated Index to Nursing and Allied Health Literature

Dissemination and Implementation

Evidence-based practices or programs

Expert Recommendations for Implementing Change

Multiphase Optimization Strategy

National Cancer Institute

National Institutes of Health

The Pittsburgh Dissemination and Implementation Science Collaborative

Sequential Multiple Assignment Randomized Trial

United States

Department of Veterans Affairs

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

Article   PubMed   PubMed Central   Google Scholar  

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.

Waltz TJ, Powell BJ, Chinman MJ, Smith JL, Matthieu MM, Proctor EK, et al. Expert recommendations for implementing change (ERIC): protocol for a mixed methods study. Implement Sci IS. 2014;9:39.

Article   PubMed   Google Scholar  

Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A Compilation of Strategies for Implementing Clinical Innovations in Health and Mental Health. Med Care Res Rev. 2012;69:123–57.

Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10:109.

Perry CK, Damschroder LJ, Hemler JR, Woodson TT, Ono SS, Cohen DJ. Specifying and comparing implementation strategies across seven large implementation interventions: a practical application of theory. Implement Sci. 2019;14(1):32.

Community Preventive Services Task Force. Community Preventive Services Task Force: All Active Findings June 2023 [Internet]. 2023 [cited 2023 Aug 7]. Available from: https://www.thecommunityguide.org/media/pdf/CPSTF-All-Findings-508.pdf

Solberg LI, Kuzel A, Parchman ML, Shelley DR, Dickinson WP, Walunas TL, et al. A Taxonomy for External Support for Practice Transformation. J Am Board Fam Med JABFM. 2021;34:32–9.

Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond “implementation strategies”: classifying the full range of strategies used in implementation science and practice. Implement Sci. 2017;12:1–9.

Article   Google Scholar  

Leeman J, Calancie L, Hartman MA, Escoffery CT, Herrmann AK, Tague LE, et al. What strategies are used to build practitioners’ capacity to implement community-based interventions and are they effective?: a systematic review. Implement Sci. 2015;10:1–15.

Nathan N, Shelton RC, Laur CV, Hailemariam M, Hall A. Editorial: Sustaining the implementation of evidence-based interventions in clinical and community settings. Front Health Serv. 2023;3:1176023.

Balis LE, Houghtaling B, Harden SM. Using implementation strategies in community settings: an introduction to the Expert Recommendations for Implementing Change (ERIC) compilation and future directions. Transl Behav Med. 2022;12:965–78.

Nathan N, Powell BJ, Shelton RC, Laur CV, Wolfenden L, Hailemariam M, et al. Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment? Front Health Serv. 2022;2:905909.

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;6:CD000259.

Google Scholar  

Moore L, Guertin JR, Tardif P-A, Ivers NM, Hoch J, Conombo B, et al. Economic evaluations of audit and feedback interventions: a systematic review. BMJ Qual Saf. 2022;31:754–67.

Sykes MJ, McAnuff J, Kolehmainen N. When is audit and feedback effective in dementia care? A systematic review. Int J Nurs Stud. 2018;79:27–35.

Barnes C, McCrabb S, Stacey F, Nathan N, Yoong SL, Grady A, et al. Improving implementation of school-based healthy eating and physical activity policies, practices, and programs: a systematic review. Transl Behav Med. 2021;11:1365–410.

Tomasone JR, Kauffeldt KD, Chaudhary R, Brouwers MC. Effectiveness of guideline dissemination and implementation strategies on health care professionals’ behaviour and patient outcomes in the cancer care context: a systematic review. Implement Sci. 2020;15:1–18.

Seda V, Moles RJ, Carter SR, Schneider CR. Assessing the comparative effectiveness of implementation strategies for professional services to community pharmacy: A systematic review. Res Soc Adm Pharm. 2022;18:3469–83.

Lovero KL, Kemp CG, Wagenaar BH, Giusto A, Greene MC, Powell BJ, et al. Application of the Expert Recommendations for Implementing Change (ERIC) compilation of strategies to health intervention implementation in low- and middle-income countries: a systematic review. Implement Sci. 2023;18:56.

Chapman A, Rankin NM, Jongebloed H, Yoong SL, White V, Livingston PM, et al. Overcoming challenges in conducting systematic reviews in implementation science: a methods commentary. Syst Rev. 2023;12:1–6.

Article   CAS   Google Scholar  

Proctor EK, Bunger AC, Lengnick-Hall R, Gerke DR, Martin JK, Phillips RJ, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18:1–19.

Michaud TL, Pereira E, Porter G, Golden C, Hill J, Kim J, et al. Scoping review of costs of implementation strategies in community, public health and healthcare settings. BMJ Open. 2022;12:e060785.

Sohn H, Tucker A, Ferguson O, Gomes I, Dowdy D. Costing the implementation of public health interventions in resource-limited settings: a conceptual framework. Implement Sci. 2020;15:1–8.

Peek C, Glasgow RE, Stange KC, Klesges LM, Purcell EP, Kessler RS. The 5 R’s: an emerging bold standard for conducting relevant research in a changing world. Ann Fam Med. 2014;12:447–55.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89:1322–7.

Shelton RC, Chambers DA, Glasgow RE. An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time. Front Public Health. 2020;8:134.

Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: Clarifications and resources. J Clin Transl Sci. 2021;5:e126.

Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1.

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;349:g7647.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ [Internet]. 2021;372. Available from: https://www.bmj.com/content/372/bmj.n71

Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A Glossary for Dissemination and Implementation Research in Health. J Public Health Manag Pract. 2008;14:117–23.

Eccles MP, Mittman BS. Welcome to Implementation Science. Implement Sci. 2006;1:1.

Article   PubMed Central   Google Scholar  

Miller WR, Wilbourne PL. Mesa Grande: a methodological analysis of clinical trials of treatments for alcohol use disorders. Addict Abingdon Engl. 2002;97:265–77.

Miller WR, Brown JM, Simpson TL, Handmaker NS, Bien TH, Luckie LF, et al. What works? A methodological analysis of the alcohol treatment outcome literature. Handb Alcohol Treat Approaches Eff Altern 2nd Ed. Needham Heights, MA, US: Allyn & Bacon; 1995:12–44.

Wells S, Tamir O, Gray J, Naidoo D, Bekhit M, Goldmann D. Are quality improvement collaboratives effective? A systematic review BMJ Qual Saf. 2018;27:226–40.

Yakovchenko V, Chinman MJ, Lamorte C, Powell BJ, Waltz TJ, Merante M, et al. Refining Expert Recommendations for Implementing Change (ERIC) strategy surveys using cognitive interviews with frontline providers. Implement Sci Commun. 2023;4:1–14.

Wagner TH, Yoon J, Jacobs JC, So A, Kilbourne AM, Yu W, et al. Estimating costs of an implementation intervention. Med Decis Making. 2020;40:959–67.

Gold HT, McDermott C, Hoomans T, Wagner TH. Cost data in implementation science: categories and approaches to costing. Implement Sci. 2022;17:11.

Boutron I, Page MJ, Higgins JP, Altman DG, Lundh A, Hróbjartsson A. Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions. 2019. https://doi.org/10.1002/9781119536604.ch7 . 

Higgins JP, Savović J, Page MJ, Elbers RG, Sterne J. Assessing risk of bias in a randomized trial. Cochrane Handb Syst Rev Interv. 2019;6:205–28.

Reilly KL, Kennedy S, Porter G, Estabrooks P. Comparing, Contrasting, and Integrating Dissemination and Implementation Outcomes Included in the RE-AIM and Implementation Outcomes Frameworks. Front Public Health [Internet]. 2020 [cited 2024 Apr 24];8. Available from: https://www.frontiersin.org/journals/public-health/articles/ https://doi.org/10.3389/fpubh.2020.00430/full

Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess Winch Engl. 2004;8:iii–iv 1-72.

CAS   Google Scholar  

Beidas RS, Kendall PC. Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. Clin Psychol Publ Div Clin Psychol Am Psychol Assoc. 2010;17:1–30.

Powell BJ, Beidas RS, Lewis CC, Aarons GA, McMillen JC, Proctor EK, et al. Methods to Improve the Selection and Tailoring of Implementation Strategies. J Behav Health Serv Res. 2017;44:177–94.

Powell BJ, Fernandez ME, Williams NJ, Aarons GA, Beidas RS, Lewis CC, et al. Enhancing the Impact of Implementation Strategies in Healthcare: A Research Agenda. Front Public Health [Internet]. 2019 [cited 2021 Mar 31];7. Available from: https://www.frontiersin.org/articles/ https://doi.org/10.3389/fpubh.2019.00003/full

Frakt AB, Prentice JC, Pizer SD, Elwy AR, Garrido MM, Kilbourne AM, et al. Overcoming Challenges to Evidence-Based Policy Development in a Large. Integrated Delivery System Health Serv Res. 2018;53:4789–807.

PubMed   Google Scholar  

Crable EL, Lengnick-Hall R, Stadnick NA, Moullin JC, Aarons GA. Where is “policy” in dissemination and implementation science? Recommendations to advance theories, models, and frameworks: EPIS as a case example. Implement Sci. 2022;17:80.

Crable EL, Grogan CM, Purtle J, Roesch SC, Aarons GA. Tailoring dissemination strategies to increase evidence-informed policymaking for opioid use disorder treatment: study protocol. Implement Sci Commun. 2023;4:16.

Bond GR. Evidence-based policy strategies: A typology. Clin Psychol Sci Pract. 2018;25:e12267.

Loo TS, Davis RB, Lipsitz LA, Irish J, Bates CK, Agarwal K, et al. Electronic Medical Record Reminders and Panel Management to Improve Primary Care of Elderly Patients. Arch Intern Med. 2011;171:1552–8.

Shojania KG, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J. Effect of point-of-care computer reminders on physician behaviour: a systematic review. CMAJ Can Med Assoc J. 2010;182:E216-25.

Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A Randomized Trial of Electronic Clinical Reminders to Improve Quality of Care for Diabetes and Coronary Artery Disease. J Am Med Inform Assoc JAMIA. 2005;12:431–7.

Dopp AR, Kerns SEU, Panattoni L, Ringel JS, Eisenberg D, Powell BJ, et al. Translating economic evaluations into financing strategies for implementing evidence-based practices. Implement Sci. 2021;16:1–12.

Dopp AR, Hunter SB, Godley MD, Pham C, Han B, Smart R, et al. Comparing two federal financing strategies on penetration and sustainment of the adolescent community reinforcement approach for substance use disorders: protocol for a mixed-method study. Implement Sci Commun. 2022;3:51.

Proctor EK, Toker E, Tabak R, McKay VR, Hooley C, Evanoff B. Market viability: a neglected concept in implementation science. Implement Sci. 2021;16:98.

Dopp AR, Narcisse M-R, Mundey P, Silovsky JF, Smith AB, Mandell D, et al. A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: State of the literature and future directions. Implement Res Pract. 2020;1:2633489520939980.

PubMed   PubMed Central   Google Scholar  

Dopp AR, Kerns SEU, Panattoni L, Ringel JS, Eisenberg D, Powell BJ, et al. Translating economic evaluations into financing strategies for implementing evidence-based practices. Implement Sci IS. 2021;16:66.

Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care:application of the replicating effective programs framework. Implement Sci. 2007;2:42–51.

Kegeles SM, Rebchook GM, Hays RB, Terry MA, O’Donnell L, Leonard NR, et al. From science to application: the development of an intervention package. AIDS Educ Prev Off Publ Int Soc AIDS Educ. 2000;12:62–74.

Wandersman A, Imm P, Chinman M, Kaftarian S. Getting to outcomes: a results-based approach to accountability. Eval Program Plann. 2000;23:389–95.

Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation support for implementing innovations with quality: Tools, training, technical assistance, and quality assurance/quality improvement. Am J Community Psychol. 2012;50:445–59.

Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Kirchner JE, Proctor EK, et al. The association between implementation strategy use and the uptake of hepatitis C treatment in a national sample. Implement Sci. 2017;12:1–13.

Smith SN, Almirall D, Prenovost K, Liebrecht C, Kyle J, Eisenberg D, et al. Change in patient outcomes after augmenting a low-level implementation strategy in community practices that are slow to adopt a collaborative chronic care model: a cluster randomized implementation trial. Med Care. 2019;57:503.

Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Gonzalez R, Park A, et al. Longitudinal assessment of the association between implementation strategy use and the uptake of hepatitis C treatment: Year 2. Implement Sci. 2019;14:1–12.

Harvey G, Kitson A. Translating evidence into healthcare policy and practice: Single versus multi-faceted implementation strategies – is there a simple answer to a complex question? Int J Health Policy Manag. 2015;4:123–6.

Engell T, Stadnick NA, Aarons GA, Barnett ML. Common Elements Approaches to Implementation Research and Practice: Methods and Integration with Intervention Science. Glob Implement Res Appl. 2023;3:1–15.

Michie S, Fixsen D, Grimshaw JM, Eccles MP. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci IS. 2009;4:40.

Smith JD, Li DH, Rafferty MR. The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci IS. 2020;15:84.

Perez Jolles M, Lengnick-Hall R, Mittman BS. Core Functions and Forms of Complex Health Interventions: a Patient-Centered Medical Home Illustration. JGIM J Gen Intern Med. 2019;34:1032–8.

Schroeck FR, Ould Ismail AA, Haggstrom DA, Sanchez SL, Walker DR, Zubkoff L. Data-driven approach to implementation mapping for the selection of implementation strategies: a case example for risk-aligned bladder cancer surveillance. Implement Sci IS. 2022;17:58.

Frank HE, Kemp J, Benito KG, Freeman JB. Precision Implementation: An Approach to Mechanism Testing in Implementation Research. Adm Policy Ment Health. 2022;49:1084–94.

Lewis CC, Klasnja P, Lyon AR, Powell BJ, Lengnick-Hall R, Buchanan G, et al. The mechanics of implementation strategies and measures: advancing the study of implementation mechanisms. Implement Sci Commun. 2022;3:114.

Geng EH, Baumann AA, Powell BJ. Mechanism mapping to advance research on implementation strategies. PLoS Med. 2022;19:e1003918.

Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356:i6795.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health Ment Health Serv Res. 2011;38:65–76.

Hooley C, Amano T, Markovitz L, Yaeger L, Proctor E. Assessing implementation strategy reporting in the mental health literature: a narrative review. Adm Policy Ment Health Ment Health Serv Res. 2020;47:19–35.

Proctor E, Ramsey AT, Saldana L, Maddox TM, Chambers DA, Brownson RC. FAST: a framework to assess speed of translation of health innovations to practice and policy. Glob Implement Res Appl. 2022;2:107–19.

Cullen L, Hanrahan K, Edmonds SW, Reisinger HS, Wagner M. Iowa Implementation for Sustainability Framework. Implement Sci IS. 2022;17:1.

Saldana L, Ritzwoller DP, Campbell M, Block EP. Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers. Implement Sci Commun. 2022;3:40.

Eisman AB, Kilbourne AM, Dopp AR, Saldana L, Eisenberg D. Economic evaluation in implementation science: making the business case for implementation strategies. Psychiatry Res. 2020;283:112433.

Akiba CF, Powell BJ, Pence BW, Nguyen MX, Golin C, Go V. The case for prioritizing implementation strategy fidelity measurement: benefits and challenges. Transl Behav Med. 2022;12:335–42.

Akiba CF, Powell BJ, Pence BW, Muessig K, Golin CE, Go V. “We start where we are”: a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity. Implement Sci Commun. 2022;3:117.

Rudd BN, Davis M, Doupnik S, Ordorica C, Marcus SC, Beidas RS. Implementation strategies used and reported in brief suicide prevention intervention studies. JAMA Psychiatry. 2022;79:829–31.

Painter JT, Raciborski RA, Matthieu MM, Oliver CM, Adkins DA, Garner KK. Engaging stakeholders to retrospectively discern implementation strategies to support program evaluation: Proposed method and case study. Eval Program Plann. 2024;103:102398.

Bunger AC, Powell BJ, Robertson HA, MacDowell H, Birken SA, Shea C. Tracking implementation strategies: a description of a practical approach and early findings. Health Res Policy Syst. 2017;15:1–12.

Mustanski B, Smith JD, Keiser B, Li DH, Benbow N. Supporting the growth of domestic HIV implementation research in the united states through coordination, consultation, and collaboration: how we got here and where we are headed. JAIDS J Acquir Immune Defic Syndr. 2022;90:S1-8.

Marques MM, Wright AJ, Corker E, Johnston M, West R, Hastings J, et al. The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1. Wellcome Open Res. 2023;8:308.

Merle JL, Li D, Keiser B, Zamantakis A, Queiroz A, Gallo CG, et al. Categorising implementation determinants and strategies within the US HIV implementation literature: a systematic review protocol. BMJ Open. 2023;13:e070216.

Glenshaw MT, Gaist P, Wilson A, Cregg RC, Holtz TH, Goodenow MM. Role of NIH in the Ending the HIV Epidemic in the US Initiative: Research Improving Practice. J Acquir Immune Defic Syndr. 1999;2022(90):S9-16.

Purcell DW, Namkung Lee A, Dempsey A, Gordon C. Enhanced Federal Collaborations in Implementation Science and Research of HIV Prevention and Treatment. J Acquir Immune Defic Syndr. 1999;2022(90):S17-22.

Queiroz A, Mongrella M, Keiser B, Li DH, Benbow N, Mustanski B. Profile of the Portfolio of NIH-Funded HIV Implementation Research Projects to Inform Ending the HIV Epidemic Strategies. J Acquir Immune Defic Syndr. 1999;2022(90):S23-31.

Zamantakis A, Li DH, Benbow N, Smith JD, Mustanski B. Determinants of Pre-exposure Prophylaxis (PrEP) Implementation in Transgender Populations: A Qualitative Scoping Review. AIDS Behav. 2023;27:1600–18.

Li DH, Benbow N, Keiser B, Mongrella M, Ortiz K, Villamar J, et al. Determinants of Implementation for HIV Pre-exposure Prophylaxis Based on an Updated Consolidated Framework for Implementation Research: A Systematic Review. J Acquir Immune Defic Syndr. 1999;2022(90):S235-46.

Chambers DA, Emmons KM. Navigating the field of implementation science towards maturity: challenges and opportunities. Implement Sci. 2024;19:26, s13012-024-01352–0.

Chinman M, Acosta J, Ebener P, Shearer A. “What we have here, is a failure to [replicate]”: Ways to solve a replication crisis in implementation science. Prev Sci. 2022;23:739–50.

Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117.

Lengnick-Hall R, Gerke DR, Proctor EK, Bunger AC, Phillips RJ, Martin JK, et al. Six practical recommendations for improved implementation outcomes reporting. Implement Sci. 2022;17:16.

Miller CJ, Barnett ML, Baumann AA, Gutner CA, Wiltsey-Stirman S. The FRAME-IS: a framework for documenting modifications to implementation strategies in healthcare. Implement Sci IS. 2021;16:36.

Xu X, Lazar CM, Ruger JP. Micro-costing in health and medicine: a critical appraisal. Health Econ Rev. 2021;11:1.

Barnett ML, Dopp AR, Klein C, Ettner SL, Powell BJ, Saldana L. Collaborating with health economists to advance implementation science: a qualitative study. Implement Sci Commun. 2020;1:82.

Lengnick-Hall R, Williams NJ, Ehrhart MG, Willging CE, Bunger AC, Beidas RS, et al. Eight characteristics of rigorous multilevel implementation research: a step-by-step guide. Implement Sci. 2023;18:52.

Riley-Gibson E, Hall A, Shoesmith A, Wolfenden L, Shelton RC, Doherty E, et al. A systematic review to determine the effect of strategies to sustain chronic disease prevention interventions in clinical and community settings: study protocol. Res Sq [Internet]. 2023 [cited 2024 Apr 19]; Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10312971/

Ingvarsson S, Hasson H, von Thiele Schwarz U, Nilsen P, Powell BJ, Lindberg C, et al. Strategies for de-implementation of low-value care—a scoping review. Implement Sci IS. 2022;17:73.

Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ Open. 2021;11:e053474.

Hailemariam M, Bustos T, Montgomery B, Barajas R, Evans LB, Drahota A. Evidence-based intervention sustainability strategies: a systematic review. Implement Sci. 2019;14:N.PAG-N.PAG.

Michie S, Atkins L, West R. The behaviour change wheel. Guide Des Interv 1st Ed G B Silverback Publ. 2014;1003:1010.

Birken SA, Haines ER, Hwang S, Chambers DA, Bunger AC, Nilsen P. Advancing understanding and identifying strategies for sustaining evidence-based practices: a review of reviews. Implement Sci IS. 2020;15:88.

Metz A, Jensen T, Farley A, Boaz A, Bartley L, Villodas M. Building trusting relationships to support implementation: A proposed theoretical model. Front Health Serv. 2022;2:894599.

Rabin BA, Cain KL, Watson P, Oswald W, Laurent LC, Meadows AR, et al. Scaling and sustaining COVID-19 vaccination through meaningful community engagement and care coordination for underserved communities: hybrid type 3 effectiveness-implementation sequential multiple assignment randomized trial. Implement Sci IS. 2023;18:28.

Gyamfi J, Iwelunmor J, Patel S, Irazola V, Aifah A, Rakhra A, et al. Implementation outcomes and strategies for delivering evidence-based hypertension interventions in lower-middle-income countries: Evidence from a multi-country consortium for hypertension control. PLOS ONE. 2023;18:e0286204.

Woodward EN, Ball IA, Willging C, Singh RS, Scanlon C, Cluck D, et al. Increasing consumer engagement: tools to engage service users in quality improvement or implementation efforts. Front Health Serv. 2023;3:1124290.

Norton WE, Chambers DA. Unpacking the complexities of de-implementing inappropriate health interventions. Implement Sci IS. 2020;15:2.

Norton WE, McCaskill-Stevens W, Chambers DA, Stella PJ, Brawley OW, Kramer BS. DeImplementing Ineffective and Low-Value Clinical Practices: Research and Practice Opportunities in Community Oncology Settings. JNCI Cancer Spectr. 2021;5:pkab020.

McKay VR, Proctor EK, Morshed AB, Brownson RC, Prusaczyk B. Letting Go: Conceptualizing Intervention De-implementation in Public Health and Social Service Settings. Am J Community Psychol. 2018;62:189–202.

Patey AM, Grimshaw JM, Francis JJ. Changing behaviour, ‘more or less’: do implementation and de-implementation interventions include different behaviour change techniques? Implement Sci IS. 2021;16:20.

Rodriguez Weno E, Allen P, Mazzucca S, Farah Saliba L, Padek M, Moreland-Russell S, et al. Approaches for Ending Ineffective Programs: Strategies From State Public Health Practitioners. Front Public Health. 2021;9:727005.

Gnjidic D, Elshaug AG. De-adoption and its 43 related terms: harmonizing low-value care terminology. BMC Med. 2015;13:273.

Download references

Acknowledgements

The authors would like to acknowledge the early contributions of the Pittsburgh Dissemination and Implementation Science Collaborative (Pitt DISC). LEA would like to thank Dr. Billie Davis for analytical support. The authors would like to acknowledge the implementation science experts who recommended articles for our review, including Greg Aarons, Mark Bauer, Rinad Beidas, Geoffrey Curran, Laura Damschroder, Rani Elwy, Amy Kilbourne, JoAnn Kirchner, Jennifer Leeman, Cara Lewis, Dennis Li, Aaron Lyon, Gila Neta, and Borsika Rabin.

Dr. Rogal’s time was funded in part by a University of Pittsburgh K award (K23-DA048182) and by a VA Health Services Research and Development grant (PEC 19-207). Drs. Bachrach and Quinn were supported by VA HSR Career Development Awards (CDA 20-057, PI: Bachrach; CDA 20-224, PI: Quinn). Dr. Scheunemann’s time was funded by the US Agency for Healthcare Research and Quality (K08HS027210). Drs. Hero, Chinman, Goodrich, Ernecoff, and Mr. Qureshi were funded by the Patient-Centered Outcomes Research Institute (PCORI) AOSEPP2 Task Order 12 to conduct a landscape review of US studies on the effectiveness of implementation strategies with results reported here ( https://www.pcori.org/sites/default/files/PCORI-Implementation-Strategies-for-Evidence-Based-Practice-in-Health-and-Health-Care-A-Review-of-the-Evidence-Full-Report.pdf and https://www.pcori.org/sites/default/files/PCORI-Implementation-Strategies-for-Evidence-Based-Practice-in-Health-and-Health-Care-Brief-Report-Summary.pdf ). Dr. Ashcraft and Ms. Phares were funded by the Center for Health Equity Research and Promotion, (CIN 13-405). The funders had no involvement in this study.

Author information

Shari S. Rogal and Matthew J. Chinman are co-senior authors.

Authors and Affiliations

Center for Health Equity Research and Promotion, Corporal Michael Crescenz VA Medical Center, Philadelphia, PA, USA

Laura Ellen Ashcraft

Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA

Center for Health Equity Research and Promotion, VA Pittsburgh Healthcare System, Pittsburgh, PA, USA

David E. Goodrich, Angela Phares, Deirdre A. Quinn, Shari S. Rogal & Matthew J. Chinman

Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, PA, USA

David E. Goodrich, Deirdre A. Quinn & Matthew J. Chinman

Clinical & Translational Science Institute, University of Pittsburgh, Pittsburgh, PA, USA

David E. Goodrich & Lisa G. Lederer

RAND Corporation, Pittsburgh, PA, USA

Joachim Hero, Nabeel Qureshi, Natalie C. Ernecoff & Matthew J. Chinman

Center for Clinical Management Research, VA Ann Arbor Healthcare System, Ann Arbor, Michigan, USA

Rachel L. Bachrach

Department of Psychiatry, University of Michigan Medical School, Ann Arbor, MI, USA

Division of Geriatric Medicine, University of Pittsburgh, Department of Medicine, Pittsburgh, PA, USA

Leslie Page Scheunemann

Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, University of Pittsburgh, Department of Medicine, Pittsburgh, PA, USA

Departments of Medicine and Surgery, University of Pittsburgh, Pittsburgh, Pennsylvania, USA

Shari S. Rogal

You can also search for this author in PubMed   Google Scholar

Contributions

LEA, SSR, and MJC conceptualized the study. LEA, SSR, MJC, and JOH developed the study design. LEA and JOH acquired the data. LEA, DEG, AP, RLB, DAQ, LGL, LPS, SSR, NQ, and MJC conducted the abstract, full text review, and rigor assessment. LEA, DEG, JOH, AP, RLB, DAQ, NQ, NCE, SSR, and MJC conducted the data abstraction. DEG, SSR, and MJC adjudicated conflicts. LEA and SSR analyzed the data. LEA, SSR, JOH, and MJC interpreted the data. LEA, SSR, and MJC drafted the work. All authors substantially revised the work. All authors approved the submitted version and agreed to be personally accountable for their contributions and the integrity of the work.

Corresponding author

Correspondence to Laura Ellen Ashcraft .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

The manuscript does not contain any individual person’s data.

Competing interests

Additional information, publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., supplementary material 5., supplementary material 6., supplementary material 7., supplementary material 8., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ashcraft, L.E., Goodrich, D.E., Hero, J. et al. A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022. Implementation Sci 19 , 43 (2024). https://doi.org/10.1186/s13012-024-01369-5

Download citation

Received : 09 November 2023

Accepted : 27 May 2024

Published : 24 June 2024

DOI : https://doi.org/10.1186/s13012-024-01369-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Implementation strategy
  • Health-related outcomes

Implementation Science

ISSN: 1748-5908

  • Submission enquiries: Access here and click Contact Us
  • General enquiries: [email protected]

critical evaluation of research article example

  • Open access
  • Published: 08 July 2024

Women’s engagement with community perinatal mental health services: a realist evaluation

  • L. Fisher 1 ,
  • A. Davey 2 ,
  • G. Wong 3 ,
  • S. Morgan-Trimmer 4 ,
  • L. M. Howard 5 ,
  • H. Sharp 1 ,
  • K. H. Atmore 6 ,
  • J. Brook 7 ,
  • G. Collins 2 ,
  • J. Domoney 5 ,
  • E. Makinde 2 ,
  • C. McCree 8 &
  • Heather A O’Mahen 2  

BMC Psychiatry volume  24 , Article number:  492 ( 2024 ) Cite this article

Metrics details

In recognition of the burden of Perinatal Mental Health problems, NHS England invested £365 million to transform women’s access to mental health care, including investment in Community Perinatal Mental Health Services. This study examined how elements of provider care affected women’s engagement with these services.

Semi-structured interviews were conducted with 139 women and explored their experiences of care from 10 different Community Perinatal Mental Health Teams; including which service components participants believed made a difference to their initial and continued engagement. Realist analysis was used to create context-mechanism-outcome configurations (CMOCs) across interviews, since not all parts of the configurations were always articulated within singular interviews.

Four key pillars for engagement were identified: perinatal competence, relationship building, accurate reassurance, and reliability. The way perinatal competencies were relayed to women mattered; compassion, understanding and consistency were critical interactional styles. The extent to which these factors affected women’s engagement varied by their context and personal characteristics.

Conclusions

As mental health problems increase, disproportionately affecting vulnerable populations, it is critical to continue to ensure support is not only available, but appropriately meets the needs of those individuals. Our findings suggest that key staff behaviours applied at the right time can support women’s engagement and potentially contribute to better treatment outcomes.

Peer Review reports

Introduction

Rationale for evaluation.

Perinatal mental health disorders are a serious public health issue, associated with poor outcomes for women (psychological and physical morbidity including suicide and life-threatening complications) [ 1 ], their foetus (preterm birth, low birth weight) and their infants/children [ 2 , 3 , 4 ]. In the UK, perinatal mental health (PMH) disorders are estimated to cost the economy £8.1 billion for each one-year cohort of births [ 5 ], with 72% of these costs relating to the longitudinal impacts on the social, emotional and cognitive development of children and adolescents.

There are effective psychiatric and psychological treatments for perinatal mental health problems [ 6 ] and recent research suggests that the children of women with postnatal depression will have fewer developmental problems at age two when their mothers receive psychological treatment and additional parenting support [ 7 ]. In total, the evidence suggests that effective services could reduce the costs of PMH disorders. However, in the UK, as in other areas of the world, few women with PMH disorders and their infants (8–30%) access mental health care during the perinatal period. These rates are in notable contrast to rates of women’s mental health treatment outside the perinatal period (50%) [ 8 ]. It is therefore important to understand factors that promote perinatal women’s initial and ongoing engagement with appropriate mental health care.

Recent meta-syntheses of qualitative data suggest that lower rates of mental health treatment receipt during the perinatal period may be due to the complex barriers mothers face when trying to access appropriate, acceptable care [ 9 ]. Key barriers include a lack of service provision that is relevant to perinatal problems affecting mental health, delivered in accessible and flexible ways that meet the challenging and quickly-changing needs of parents navigating pregnancy and caring for an infant [ 10 ].

Programme initiative and environment surrounding evaluation

In England, in recognition of the costs and burden of PMH problems and their low service access rates, NHS England invested £365 million to transform women’s access to mental health care, including investment in Community Perinatal Mental Health Teams (CPMHTs). These secondary care mental health services provide treatment for women with moderate to severe and/or complex mental health disorders during pregnancy and up to 24 months postpartum. CPMHTs are multidisciplinary, comprising of psychiatrists, mental health nurses, pharmacists, psychologists, occupational therapists, nursery nurses and peer support workers. CPMHTs include; perinatal specialist assessment, psychiatric and psychological treatment, parent-infant interventions, and advice/liaison for primary care/maternity/secondary generic mental health services on identifying, preventing and caring for mild to severe PMH problems [ 11 ]. With the substantial growth in these services, which aim to see 66,000 women/year nationally, key questions arise about whether they are improving women and babies’ access to care, and what factors promote women’s successful initial engagement and ongoing adherence with mental health treatment.

Although previous literature has highlighted both generic and perinatal specific factors that promote engagement and adherence with mental health interventions [ 12 , 13 , 14 , 15 ], there has not yet been an evaluation of whether CPMHTs, which are designed to target multiple perinatal barriers and facilitators to accessing care at once, are successful in these aims. In the perinatal period, the experience of childbearing, changes in relationships, and exposure to scenarios in which a person may experience a lack of control mean this is a vulnerable time for women [ 16 ]. This may be especially true for those with histories of mental health difficulties, trauma, abuse and discrimination, experiences that are common amongst female mental health service users [ 17 , 18 , 19 , 20 ]. Early research into engagement with PMH care found that women reported improved willingness to engage with services when providers were relatable, had expert knowledge about the ways in which pregnancy and parenting in the postnatal period influence mental health, and provided women with the time to discuss their needs and the ability to choose between treatment options [ 21 ]. What is less clear, however, is when and why these elements might be important for particular populations and/or presenting problems, and how they might be modified to meet the specific needs of women in the perinatal period. These questions are critical to understanding which factors ensure CPMHTs successfully deliver care.

Rationale for using realist evaluation and research question

In this paper, we report a realist evaluation that examined how elements of CPMHTs provider care affected women’s engagement with services. The data reported in this paper come from a larger mixed-methods study examining the effectiveness and cost effectiveness of community perinatal mental health services in England (ESMI-II). The ESMI-II study was funded to produce evidence about the effectiveness of CPMHTs in improving mother and infant outcomes. This included undertaking a realist evaluation to investigate how, why and for whom CPMHTs configurations and components are effective at improving acceptability and engagement with care and consequently improving outcomes. This paper extends the literature that outlines core generic provider elements supporting engagement with mental health treatment by asking, “what perinatal specific elements of service provider care make a difference to women’s engagement and adherence, for whom and why?”

Ethical approval

The study was approved by the Southwest - Central Bristol Research Ethics Committee (Reference: 19/SW/0218).

Study design

The ESMI-III study consisted of 4 overarching work packages focused around different aspects of CPMHTs, from the characterisation of service variations seen within CPMHT, the reliability, validity and clinical feasibility of observational measures for assessing parent-infant interaction quality within CPMHTs and the exploration of linked health data in relation to access to secondary care mental health services. The final work package was the evaluation of the effectiveness and cost effectiveness of the CPMHTs through research interviews with 139 women, 55 sources of significant support and 80 health and social care practitioners, across 10 different CPMHTs. This is the work package and data which this paper is based upon, with focus only on the women’s interview data.

Participants

A total of 10 CPMHTs were purposefully selected for variations in different components and configurations including: level of mother-infant and/or psychological interventions, collaborative care (e.g. integrated with universal and adjunctive services), quality of integration (e.g. communication, mental health co-located with maternity care), delivering perinatally tailored care, including care coordination vs. co-working with generic mental health teams, meeting women’s needs (flexible, remote/in home, de-stigmatising, family centred). A purposive sampling approach (with maximum variation in characteristics) was taken to identify up to 10 women who had received care in each service to help us refine and test aspects of our initial programme theory. Where possible, women were purposively sampled with different types of mental health problems (i.e., Emotionally Unstable Personality Disorder (EUPD), Anxiety Disorder, Trauma/Post Traumatic Stress Disorder (PTSD), Obsessive Compulsive Disorder (OCD), Severe Depression and Severe Mental Illness (SMI; Psychosis, Bipolar I) and across diverse sociodemographic groups. The current study focused on women’s experiences of CPMHTs and what they reported worked for them in relation to their engagement with the service.

Recruitment

Recruitment of women took place between April 2020 and June 2021 and included those who experienced care prior to and during the COVID 19 pandemic. A member of the direct care team approached potential participants via phone or email and gave a brief overview of the study and an opt-out option. A link to more detailed information and the information sheet was then emailed to interested participants, including a consent to be contacted by the research team. Interested participants were then contacted by the researcher via telephone to discuss the study further. Participants were interviewed towards the end of their treatment or after they had completed their treatment, to avoid Hawthorne effects.

Data collection

Semi-structured interviews lasting approximately 90 min, were conducted via telephone or videoconference with participants. Interviews followed a guide that was developed in collaboration with clinicians, policy makers and persons with lived experience. The interviews explored women’s experiences of care including their mental health history, access to service, and which service components participants believed made a difference to their access, engagement and adherence with treatment and the impact (if any) on their mental health/well-being, functioning and their relationship with their family, including their infant.

Data analysis

Interviews were audio-recorded, transcribed verbatim and analysed using NVivo 12 software. Initially, interview data was coded using thematic analysis, focusing around broad areas of interest from the initial programme theory, including aspects such as referral process, access to care and quality of care. Further analysis involved the use of a realist logic of analysis with the goal of using the collected data to develop and refine the initial programme theory (see additional file 1 ) into a more refined realist programme theory. Data coding was deductive (informed by our initial programme theory), inductive (from the data within transcripts) and retroductive (where inferences were made based on interpretations of the data within sources about underlying causal processes – i.e. mechanisms). Through this approach, context-mechanism-outcome configurations (CMOCs) were developed and data to inform our interpretation of the relationships between contexts, mechanisms and outcomes was sought across interviews (e.g. mechanisms inferred from one interview could help explain the way contexts influenced outcomes in a different interview). Synthesising data from different interviews was often necessary to compile CMOCs, since not all parts of the configurations were always articulated in a single interview. We moved iteratively between the analysis of specific examples, refinement of programme theory, and any data we had collected to test particular theories.

Details of participants

Characteristics of the women are described in Table  1 .

Main findings

An initial programme theory was designed prior to data collection to explain and account for engagement and its association with perceived outcomes. Based on collected data this was gradually refined into Fig.  1 . In this figure, using the hypothesised elements from the initial programme theory, we mapped high level patterns observed within the data that were associated with women’s engagement with the service.

figure 1

Overall programme theory of women’s engagement with CPMHTs

Figures  2 and 3 explain the engagement processes in more detail. We report the processes and effects of engagement in three stages: Fig.  2 illustrates initial engagement, which we defined as the process of referral, the first contact, the woman’s assessment appointment with the service and the first contact with their respective key worker(s). We also included transitory engagement in Fig.  2 , which we defined as the process between the initial assessment appointment and when the woman agrees an initial treatment plan with the service. Figure  3 (presented later in the findings) illustrates continued engagement/adherence, which we defined as the woman’s ongoing subsequent treatment contacts with the service. In Figs.  2 and 3 we portray both the direct primary relationships between these high-level factors, and the indirect factors supporting these relationships as they relate to engagement. The boxes highlighted in green are components which appear within and are taken from Fig.  1 and the unfilled boxes are additional details not previously covered within the overall programme theory. Within the following findings, we present knowledge claims supported by CMO configurations numbered 1–19 [e.g., CMO-C1]. See Tables  2 , 3 , 4 , 5 , 6 , 7 and 8 .

figure 2

Programme Theory specific to the initial engagement phase

Initial engagement- providing reassurance

Before accessing CPMHTs, most women described feeling vulnerable, confused about what kind of support might be helpful, and frustrated with the help-seeking process. They reported struggling with significant concerns about stigma and feeling they were failures as mothers. A significant proportion of women in the study stated they were afraid that having mental health difficulties put them at risk of losing their children. In that context, mothers reported that their fears of judgement by providers were a key barrier to engagement with mental health services.

Fear of judgement was especially acute in mothers who had had contact with generic mental health services prior to accessing the CPMHTs. These women reported they were particularly wary of mental health services because they had encountered poor understanding of their perinatal specific needs, judgement, and a lack of perinatally-tailored interventions in generic mental health services. For example, women reported receiving incorrect guidance about medication usage from GPs or psychiatrists in generic adult community mental health teams. They also reported that the psychological interventions they received from generic services lacked a focus on perinatal specific mental health difficulties.

“[Adult Mental Health] took me off of my medication…and put me on a different one that they classed as safer to be on when pregnant. But since being in perinatal, the doctor there said that they should not have done that…I could have stayed on the medication that I was on throughout the pregnancy and it would have actually [been] safer” [2M5, EUPD, OCD, Anxiety & Depression]. “I went to [primary care mental health, Improving Access to Psychological Therapy; IAPT] groups, like a panic group…just trying to help with like different strategies and stuff like that but it didn’t really apply to me because I was pregnant, a lot of my worries, panics, fears were about were around pregnancy…it was really hard for me to actually engage with the service because it wasn’t for pregnancy, it was just for people who weren’t pregnant.” [6-M2, Anxiety & Depression].

Some women consequently approached CPMHTs apprehensively but were positively and powerfully impacted by initial conversations with CPMHTs staff who took the time to hear their worries and problems, were perceived as knowledgeable in addressing their perinatal specific needs and normalised the purpose of the service. [CMO-C1] [CMO-C2] are presented in Table  2 .

“They gave me so much like advice…like other mums go through this…they kind of went through stories like confidentially… ‘this mum does this, or this mum was with us for this long time and here they are now’ and that kind of made it better” [9-M6, Depression and bonding difficulties].

Relieved that they were in the right place and with the right providers, many women expressed gratitude when they perceived they had accurate reassurance that they were doing the right thing for their mental health and their baby by accessing the service. Critically, women said that a key factor supporting their engagement was when staff addressed their most significant fears - that they were not a terrible mother or person, that their baby wouldn’t be taken away (when this was the case) - and when staff highlighted what women were doing well. [CMO-C3] as shown in Table  3 .

“I think my problem is I’m a lot more capable than I think I am and the anxiety just completely muddles my brain and what [perinatal nursery nurse] and [perinatal mental health nurse] did was allow me to access the tools I already have, you know, I am a really good mum, I am switched on, you know, I had already done a lot of the prep for welcoming a new baby and [perinatal nursery nurse] just, you know, gently guided, reassured me that, you know, ‘hey, look at everything you’re doing’.” [5-M10, Anxiety & Depression].

Women’s rapidly growing confidence in both the expert perinatal competence of staff and their ability to address problems non-judgementally allowed them to engage openly with staff if they struggled with challenging issues. [CMO-4] [CMO-5] as presented in Table  3 . For example, women who had safeguarding issues reported they appreciated when staff openly and honestly acknowledged that social care might need to become involved, explaining how the CPMHTs and social care would be there to act in the mother and baby’s best interests.

“you are slightly guarded about how mental you tell someone you are [laughs], because you do worry about, you know, will they take my children away from me, you know, what intervention is going to happen next? but … I’m not in that position where I ever felt that, ‘oh, I can’t say that’… [perinatal nursery nurse] was incredibly helpful and supportive through that process” [1-M8, Depression]. “[scan practitioner] said to me, you know, ‘I’m going to put a yellow flag up there for you, just because of your mental health’…she made me almost feel like I needed social services involved because I was mentally not well, whereas, when [perinatal mental health nurse] come round, she explained to me that if social services were to be involved, it would be for the benefit of me, not to try and rip my child away from me.” [7-M10, Anxiety & Depression].

In sum, perceived staff expertise and perinatal competence [CMO-C6] (see Table  4 ), combined with their non-judgemental reassurance, gave many women a sense of feeling understood and safety, which provided them with a foundation of trust and confidence in the service. [CMO-2] as presented in Table  2 previously. Though still wary and vulnerable, women described this initial confidence as key in their willingness to “take a leap of faith” and engage openly with the CPMHTs. Some of these women reported they were surprised at how open they were able to be with perinatal staff, because of the way staff modelled honest and non-judgmental communication. [CMO-C7][CMO-C8] as presented in Table  4 .

“[perinatal psychologist] said to me ‘you know I am not here to judge you, because other people think the same as you’ … after that conversation I never once thought ‘I can’t possibly tell anyone that because they’re going to judge me.’ After that conversation I was completely honest with [perinatal mental health nurse] and [perinatal psychologist] which I never thought … I would be.” [8-M13, Bipolar & Psychosis, Postnatal Depression & Anxiety].

Women’s ability to be open allowed staff to better understand the causes and contexts in which their mental health problems were occurring and how they were affecting women and their families, and increased the likelihood staff would correctly identify which treatments would be beneficial for their needs. [CMO-C9], as presented in Table  4 .

“I think it’s definitely someone listening to what the problem is and thinking about what might help you the most. And then with the acceptance and commitment therapy, yeah, that was the same, actually, thinking about it, [perinatal practitioner] really listening to me, hearing what I needed, where I was coming from… she had actually given me all the tools that I needed… I have felt completely supported, I know that I’ve always had someone to turn to, they’ve always been fighting my corner if I need anything, they’re amazing” [1-M13, Bipolar II].

Transitory Engagement .

In a cycle of growing trust, women stated that receiving treatment offers that matched their needs helped them to feel as if they had a greater degree of choice and control in their treatment, by virtue of being heard and recognised. [CMO-C10] as shown in Table  5 .

“That would always be a question in every consultation ‘well what do you think we can do to help you… Make you feel a bit better?’ …And sometimes there wasn’t anything and sometimes perhaps I’d bring something up that I think they could help me with and they’d say, ‘well we’ve got this person that might be able to help you’ and then obviously that was my choice.” [9-M9, Postpartum Psychosis].

In turn, they were grateful for staff members perceived as competent in perinatal mental health, who were able to confidently deliver on treatment planning. This produced a sense of trusted security in the service, prompting greater readiness for continued engagement with the service. [CMO-C11] see Table  5 .

“I thought they were really good…experienced and understanding people…having people that are specifically…this is what they do for this specific situation…I felt like, you know, if there was something going on, then they’d be able to spot it a mile away, so I felt quite…reassured by that.” [8-M17, EUPD, Psychosis].

In contrast, some women reported that they had less positive early interactions with staff. When staff failed to take the time to have open, non-judgmental discussions, women reported they felt coerced to follow treatment plans. This had a negative impact on their immediate mental health and resulted in reticence and anxiety about future contact. Many of these women, faced with few other appropriate treatment options, continued to guardedly engage with the service, though this was marked by avoidance. Women had poorer attendance at appointments or attended appointments alone, kept appointments brief and interacted no more than they felt they needed to. As a consequence, women failed to develop close and trusting relationships and services had to work harder to keep them engaged.

“I didn’t see talking to her as a solution. It was more of a problem, because my first contact with her was about like trying to get me on this other medication. So, I’d got anxieties over talking to her again, because that’s how I’d kind of remembered her, from then on. So, I never really wanted to hang around with her, but I knew that if I needed help, I’d get it.” [3-M7, Anxiety & Depression].

Family inclusive approach

Women reported that when the perinatal team considered their needs within a family context this supported their engagement. Women noted they benefitted from getting help with parenting support in the context of struggling with their own mental health problems. Women also commented on how including a loved one in their care (if they wanted it) supported them to attend appointments, and aided and encouraged them to be honest and open about their experiences. [CMO-C12] presented in Table  6 .

“Sometimes, when I was struggling, it felt a bit hard to be open with people…So, having [Partner] there as support he kind of, you know, encourages me to be honest. And, because I’ve been honest with [Partner] about how I feel, he knows if I am struggling to say something, or explain something, [Partner] will like help to explain it, or…will, you know, reiterate what I’ve told him…So, it was helpful to have him there at times.” [3-M10, EUPD, PTSD].

Family members also acted as valuable co-historians, and supported women in their care outside of the service. Critically, significant others could be an additional source of information and communication when the woman might be critically unwell and unable to make contact and engage with the service by themselves.

“I felt a bit better after the meeting, knowing there were people, sort of in the background, I could contact if there was anything, or… my husband would contact them, ‘cos I wasn’t in a place where I would speak to people or ring them up” [EX-BM2, Severe Depression].

This specialist ability to work closely with their loved ones and in a family context, helped women to feel that staff had their family’s best interests at heart and further reduced their feelings of shame about experiencing mental health difficulties as a parent.

[Interviewer: what’s important? “being sort of family-focused, or patient focused in the needs of the mother and the child and the family, I suppose. You know like how they’ve extended things for me, because of my situation…That’s been a massive one for me. Advocating and sort of just helping build confidence…empowering me really.” [3-M11 EUPD, Postnatal Depression with Psychosis] .

Some women, however, stated that they preferred not to have loved ones included in their treatment; these were often women who were experiencing bonding difficulties or suicidal ideation. Fearful of judgment and ashamed of their feelings, women described the appeal of seeking support from the CPMHTs as being something that was separate and confidential from their day-to-day lives. In being giving the choice to attend appointments alone, they felt more comfortable to be open about their experiences and needs. [CMO-C13] see Table  6 .

“I wouldn’t necessarily have wanted him in my appointments because I think that a lot of the things that I was feeling and saying at the time were really hurtful to (Husband). So, I don’t think that would have been helpful for either of us. I would have felt, you know, judged by him for saying things which of course he would be and he would have felt very uncomfortable about the way I was talking about our baby.” [3-M6, Anxiety & Depression].

Continuing engagement

As women continued in their treatment with the service, their focus shifted from factors that supported their initial engagement to factors that helped sustain them in a mental health journey that was for many a challenging, non-linear path, complicated by the challenges of adapting to a changing pregnancy and/or rapidly developing infant development across the first postnatal year. Against this background, women highlighted how service reliability and consistency, flexibility, and having a key identified member of staff in the team were critical to keeping them engaged with the service and their treatment.

figure 3

Programme Theory specific to continued engagement with CPMHTs

Reliability and consistency

Women reported that the maintenance of trust in the service was dependent on the reliability of staff, who needed to be consistently compassionate, non-judgemental, and deliver consistent messages both across time and between providers in the service (e.g. about medication use, treatment approaches). [CMO-C14] presented in Table  7 . This reliability was particularly valued when women faced challenges in their own relationships that left them feeling vulnerable and distrustful. Reliability and the team’s shared, perinatal knowledge helped increase women’s confidence in the competence of staff and continued to reassure them that they were getting the right help for their difficulties. In turn, they noted that this contributed positively to their ongoing engagement with the service. [CMO-C15] see Table  7 .

“you didn’t lose your trust, you know, she made sure that the communication lines were open… that she followed through with what she was going to say and what she was going to do…you knew what to expect, there weren’t any surprises” [5-M12, Psychosis].

Frequent contact also helped with engagement and was perceived to be a very different approach to how other services operate. Women who felt they had little or no external support, or who felt unable to discuss their difficulties with support networks, especially appreciated regular check-ins from staff, which they interpreted as a sign that staff cared about them, their needs and progress.

“I think definitely with the phone calls it was definitely like a massive thing…It’s the frequency…it’s not like someone’s just coming in seeing you once a month, like you’re having [weekly contact], you’re building that relationship up with somebody and the only way that you can get, I think, very into a person’s mind is by having regular contact. They open up to your more and you find it more [a] trusting … relationship otherwise you don’t build it, … then people aren’t honest.” [LV-NWM4, EUPD, Depression].

However, instances in which staff were not consistent or reliable, cancelling appointments or not showing up to appointments on time or at all, women felt frustrated and this eroded their confidence with the service or practitioner. Others at times felt ignored by practitioners. Women described some staff as being ‘dismissive,’ and ‘not passing along’ pertinent information about their care to other practitioners and noted how this negatively impacted their engagement with the service. [CMO-C16] see Table  7 .

“whenever I did seek help it was brushed off. So, in the end I did start to learn to like keep my mouth closed a lot of the time and not tell [perinatal practitioner] that I was struggling or anything.” [3-M2, Anxiety & Depression, PTSD].

Key staff connection

For these reasons, women found having at least one person in the service whom they felt was their main worker helped them to feel safe, connected to the service, understood, and accurately represented to the rest of the service. Women who experienced a change of main practitioner during their care reflected on the negative impact this had on their recovery as they had to shift focus on their treatment to build trust and relationship with a new person. Some women who experienced multiple changes in staff described having lower and more guarded levels of engagement as a result. [CMO-C17] presented in Table  8 . This was at times detrimental to a woman’s care, as a few described being less likely to contact the service even when their own safety and wellbeing was at risk.

“I was just kind of feeling comfortable opening up to one person and then to be handed to somebody else and then again to somebody else who I never actually met. So, that kind of trust was never really built and …that shift has hindered me somewhat… I felt like I couldn’t really call anybody if I needed to.” [6-M7, Postpartum Depression].

Some services adopted a different model, where women were assigned a small core group of practitioners involved in their care. Women in this service model reflected that they were less affected if they experienced changes in practitioners because they had good existing relationships with other practitioners in their core group. [CMO-C17] see Table  8 .

“I do think it’s helped that every time [perinatal mental health nurse]’s been off it’s always been [Occupational Therapist]. I think you feel a bit more supported than somebody just turning up that you don’t know.” [9-M11, Bipolar I].

Flexible delivery

While women valued consistency and reliability, many also reported that their ongoing ability to engage in the service required a flexible delivery approach. Women, faced in pregnancy with managing fatigue, feeling physically unwell, and multiple medical appointments, and postnatally with changeable infant care schedules and rapid developmental changes, found that the rigidity of generic mental health services clashed with their ever changing and unpredictable parenting demands and therefore hampered their engagement. In contrast, women described perinatal services as providing a more flexible, perinatally-informed approach that supported their ability to remain engaged with treatment. [CMO-C18] see Table  8 . For example, services provided women with the means to contact individual staff members, and ensured appointment locations took place in environments in which the mother felt comfortable. Women reported this approach helped them to feel as if the team was exerting a real effort to meet their needs and to give them reasonable opportunities to receive the support they needed.

“I was supposed to attend a course, which was for people just like myself, going through the perinatal situation, but I couldn’t make it and I was heartbroken. [Occupational Therapist] didn’t judge me…she did it with me, like, remotely… I was so much better than when I started. I actually felt it, as well, I felt supported, I felt sort of, like, ready to put my foot forward” [10-M1, Postnatal Depression] .

Impacts of engagement

Women described that they benefitted from services in two ways. Firstly, when they had honest relationships with staff and were able to get the right treatment [CMO-C9] (as presented in Table  4 previously), and secondly, when they formed trusting relationships with staff that they were then able to use as models to help them build trust with others in their support system and thereby indirectly get appropriate help in their day-to-day lives. [CMO-C19] as presented in Table  8 . Strikingly, women reported that getting the right treatment and feeling able to engage in that treatment not only helped their wellbeing and functioning, but in some cases, saved their lives.

“if I hadn’t had the perinatal mental health team, I think I would have probably been dead, or (baby) would have been dead to be honest with you. I was that ill.” [2-M11, EUPD, Postnatal Depression with Psychosis]. “the … therapeutic process meant that I began to understand how unboundaried I was and how that wasn’t really my fault and that actually, even though it felt really, really difficult, if I hadn’t gone through this process with them then my relationships with my family would probably be worse…it completely changed the relationship that I have with my mum, which I can just about handle now.” [5-M3, Anxiety & Depression].

Further, drawing on their faith in perinatal mental health services, women reported that in the future they would be more likely to seek support earlier on when they needed it, critically supporting ongoing mental health.

“I used to be very guarded about my mental health. So, I would not openly say to someone, ‘I’m a mental health patient’. No way. If I had another child I absolutely would and that kind of confidence and ability to admit that is down to the perinatal mental health intervention that I had. I’m not guarded about it anymore.” [6-M1, Bipolar I].

Summary of findings

This study demonstrated four key pillars in the foundations of women’s engagement with perinatal mental health services. It demonstrated that women’s engagement was underpinned by their perceptions of service providers’ perinatal competence. The way perinatal competencies were relayed to women also mattered; compassion, accurate understanding and reliability and consistency were all critical interactional styles that helped build strong therapeutic relationships. The extent to which these factors affected women’s engagement varied by their context and personal characteristics.

Comparison with existing literature and recommendations

We found that in perinatal mental health, first impressions count. Women were initially cautious about engaging with CPMHTs, though we found their reasons for wariness varied by personal characteristics. Stigma and fear of having one’s child removed from their care, common barriers to perinatal mental health service engagement [ 22 ], were significant for many women. However, these fears were compounded in two groups of women: those who did not have significant previous contact with mental health services, and those who had negative previous experiences with mental health services. The first group’s lack of knowledge of CPMHTs magnified their fears of stigmatization in what they identified as an already vulnerable period of life. The second group described many of their previous experiences of mental health care in negative terms and were guarded and sceptical of CPMHTs as a result. First contacts with the CPMHTs were therefore a key turning point for many women. What staff did in those initial contacts mattered.

Consistent with previous literature on therapeutic behaviours [ 13 ], a welcoming and non-judgemental approach was critical to women’s engagement both initially and beyond, but we found that this stance needed to be underpinned by a firm base of perinatal knowledge and expertise . With these elements in place, women reported they experienced staff as balanced, reflective and non-reactive. This was in contrast to the judgement and unpleasant experiences women reported they feared or had experienced from clinicians in generic mental health and health services. Non-judgemental and perinatally competent approaches helped women build confidence and trust in staff and reassured them the CPMHTs could help them. These key components promoted women’s hope for improvement and their willingness to engage openly and honestly with the service, mechanisms consistently highlighted across the therapy literature as essential for positive outcomes [ 22 , 23 ].

Previous literature has noted the importance of reassurance in building patient engagement [ 24 ]. We found, however, that the quality and accuracy of reassurance mattered. Though accurate reassurance might include negative information (e.g., hospitalisation, social services involvement), women who trusted staff who were perinatally competent and had their best interests in mind reported they appreciated this honesty. It reduced uncertainty about their circumstances and gave them clarity about what steps they needed to take towards improvement.

There is increasing recognition of the importance of continuity of care in both health [ 25 , 26 ] and mental health [ 27 ], with maternity services in many countries adopting small team, “case loading” approaches [ 28 , 29 ]. Women in this study also described how having a continuous provider, or a core set of providers, was critical to their engagement. It ensured ongoing trust in the CPMHTs, patient openness and a sense that they had the agency to undertake the right set of treatments for them. Women in this study also noted that it was critical for staff to be flexible in how they delivered treatment and to undertake outreach, because of the uncertainties and pressures associated with pregnancy and caring for an infant [ 17 ]. These behaviours not only helped women to remain engaged in their treatment, but also provided them with stable interpersonal models. Consistent with the literature on transference in psychotherapy [ 30 ], some women reported that these interpersonal models also helped them to rebalance their own relationships and this supported their ongoing recovery.

Not all women interviewed had these experiences, however, and when elements in this unfolding set of staff behaviours were missing, women were less likely to actively engage with services [ 23 , 31 ]. Notably, a number of women reported they still remained with the service, but had only minimal and guarded interactions with it.

Together, this study provides clarity on what CPMHTs can do to effectively engage perinatal women as shown in Table  9 .

Strengths and limitations

This is one of the largest and most comprehensive qualitative studies of perinatal treatment engagement and outcomes. We sampled a range of CPMHTs across England and a broad range of women with different mental health problems. By taking a realist approach, this study extended previous literature in this domain by examining causal factors underlying women’s engagement with perinatal mental health services, examining what worked for whom and when.

Although the sample here is based on women who were referred to the service and opted to attend at least an initial appointment, we did have accounts from women who initially engaged well, but then disengaged, and women who didn’t initially engage well with the service but returned at a later date. These women’s accounts helped us to distinguish between what worked and what didn’t work for women once they started receiving services from CPMHTs.

Our sample was socioeconomically and ethnically representative of the range of women seen in the CPMHTs we drew from. However, like most mental health services, CPMHTs disproportionally treat White women. To fully understand the needs of ethnically diverse populations, additional research using different recruitment approaches is needed.

In this study four key pillars for engagement emerged: perinatal competence, relationship building, accurate reassurance, and reliability. Consistent with Chorpita’s guidelines [ 32 , 33 ] on evidence-based approaches for addressing problems in engagement, these findings suggest that key staff behaviours applied at the right time can support continued engagement and potentially contribute to better treatment outcomes. Prior to the investment in perinatal mental health services in England, women’s engagement with mental health services was low with fewer than 8 to 30% of eligible women receiving support. Getting the right, perinatally skilled, care matters, and recent results from national linked data show the CPMHTs improve women’s access to timely mental health treatment (ESMI-II unpublished results). Encouragingly, many women in this study finally found safe, trustworthy and non-stigmatising care in CPMHTs, pointing to its perinatal specific nature as a key deciding factor in their decision to engage. As mental health problems increase, disproportionately affecting vulnerable populations, it is critical to continue to ensure support is not only available, but appropriately meets the needs of those individuals.

Data availability

This study collected detailed qualitative data on a sensitive topic which included personally-identifiable information, in a limited number of identified sites. For this reason, it is not possible to anonymise the data. Therefore, the qualitative data from this study are not publicly available, but are available from the corresponding author on reasonable request.

Abbreviations

Community Perinatal Mental Health Teams

Context-Mechanism-Outcome Configurations

Emotionally Unstable Personality Disorder

Obsessive Compulsive Disorder

  • Perinatal Mental Health

Post Traumatic Stress Disorder

Severe Mental Illness

Easter A, Sandall J, Howard LM. Obstetric near misses among women with serious mental illness: data linkage cohort study. Br J Psychiatry. 2021;219(3):494–500.

Article   PubMed   PubMed Central   Google Scholar  

Howard LM, Molyneaux E, Dennis C-L, Rochat T, Stein A, Milgrom J. Non-psychotic mental disorders in the perinatal period. Lancet (London England). 2014;384(9956):1775–88.

Article   PubMed   Google Scholar  

Pearson RM, Campbell A, Howard LM, Bornstein MH, O’Mahen H, Mars B, et al. Impact of dysfunctional maternal personality traits on risk of offspring depression, anxiety and self-harm at age 18 years: a population-based longitudinal study. Psychol Med. 2018;48(1):50–60.

Article   CAS   PubMed   Google Scholar  

Stein A, Pearson RM, Goodman SH, Rapa E, Rahman A, McCallum M, et al. Effects of perinatal mental disorders on the fetus and child. Lancet (London England). 2014;384(9956):1800–19.

Bauer A, Parsonage M, Knapp M, Iemmi V, Adelaja B. The costs of perinatal mental health problems. Centre for Mental Health - London School of Economics; 2014.

Howard LM, Khalifeh H. Perinatal mental health: a review of progress and challenges. World Psychiatry. 2020;19(3):313–27.

Stein A, Netsi E, Lawrence PJ, Granger C, Kempton C, Craske MG, et al. Mitigating the effect of persistent postnatal depression on child outcomes through an intervention to treat depression and improve parenting: a randomised controlled trial. Lancet Psychiatry. 2018;5(2):134–44.

Lee-Carbon L, Nath S, Trevillion K, Byford S, Howard LM, Challacombe FL, et al. Mental health service use among pregnant and early postpartum women. Soc Psychiatry Psychiatr Epidemiol. 2022;57(11):2229–40.

Smith MS, Lawrence V, Sadler E, Easter A. Barriers to accessing mental health services for women with perinatal mental illness: systematic review and meta-synthesis of qualitative studies in the UK. BMJ Open. 2019;9(1):1–9.

Google Scholar  

Buist A, O’Mahen H, Rooney R, Acceptability. Attitudes, and overcoming Stigma. Identifying Perinatal Depression and anxiety: evidence-based practice in Screening, Psychosocial Assessment, and management. Chichester, UK: John Wiley & Sons, Ltd; 2015. pp. 51–62.

Chapter   Google Scholar  

Royal College of Psychiatrists. College Report CR232: Perinatal Maternal Mental Health Services: Recommendations for Provision of Services for Childbearing Women [Internet]. 2021 [cited 2023 Mar 12]. https://www.rcpsych.ac.uk/improving-care/campaigning-for-better-mental-health-policy/college-reports/2021-college-reports/perinatal-mental-health-services-CR232 .

Ayres A, Chen R, Mackle T, Ballard E, Patterson S, Bruxner G, et al. Engagement with perinatal mental health services: a cross-sectional questionnaire survey. BMC Pregnancy Childbirth. 2019;19(1):1–11.

Article   Google Scholar  

Dixon LB, Holoshitz Y, Nossel I. Treatment engagement of individuals experiencing mental illness: review and update. World Psychiatry. 2016;15(1):13–20.

Pilav S, De Backer K, Easter A, Silverio SA, Sundaresh S, Roberts S, et al. A qualitative study of minority ethnic women’s experiences of access to and engagement with perinatal mental health care. BMC Pregnancy Childbirth. 2022;22(1):1–13.

Thompson L, McCabe R. The effect of clinician-patient alliance and communication on treatment adherence in mental health care: a systematic review. BMC Psychiatry. 2012;12.

Stephenson LA, Beck K, Busuulwa P, Rosan C, Pariante CM, Pawlby S, et al. Perinatal interventions for mothers and fathers who are survivors of childhood sexual abuse. Child Abuse Negl. 2018;80:9–31.

Ayers S, Crawley R, Webb R, Button S, Thornton A, Smith H, et al. What are Women Stressed About after Birth? Birth. 2019;46(4):678–85.

PubMed   Google Scholar  

Dennis CL, Chung-Lee L. Postpartum depression help-seeking barriers and maternal treatment preferences: a qualitative systematic review. Birth. 2006;33(4):323–31.

Lancaster CA, Gold KJ, Flynn HA, Yoo H, Marcus SM, Davis MM. Risk factors for depressive symptoms during pregnancy: a systematic review. Am J Obstet Gynecol. 2010;202(1):5–14.

Oram S, Fisher HL, Minnis H, Seedat S, Walby S, Hegarty K, et al. The Lancet Psychiatry Commission on intimate partner violence and mental health: advancing mental health services, research, and policy. Lancet Psychiatry. 2022;9(6):487–524.

Myors KA, Johnson M, Cleary M, Schmied V. Engaging women at risk for poor perinatal mental health outcomes: a mixed-methods study. Int J Ment Health Nurs. 2015;24(3):241–52.

Maura J, Weisman de Mamani A. Mental Health Disparities, Treatment Engagement, and attrition among Racial/Ethnic minorities with severe Mental illness: a review. J Clin Psychol Med Settings. 2017;24(3–4):187–210.

Smith TE, Easter A, Pollock M, Pope LG, Wisdom JP. Disengagement from care: perspectives of individuals with Serious Mental illness and of service providers. Psychiatr Serv. 2013;64(8):770–5.

Easter A, Pollock M, Pope LG, Wisdom JP, Smith TE. Perspectives of treatment providers and clients with Serious Mental Illness regarding Effective Therapeutic relationships. J Behav Health Serv Res. 2016;43(3):341–53.

Cabana MD, Jee SH. Does continuity of care improve patient outcomes? J Fam Pract. 2004;53(12):974–80.

Haggerty JL, Reid RJ, Freeman GK, Starfield BH, Adair CE, McKendry R. Continuity of care: a multidisciplinary review. BMJ. 2003;327(7425):1219–21.

Biringer E, Hartveit M, Sundfør B, Ruud T, Borg M. Continuity of care as experienced by mental health service users - a qualitative study. BMC Health Serv Res. 2017;17(1):1–15.

Dharni N, Essex H, Bryant MJ, Cronin de Chavez A, Willan K, Farrar D, et al. The key components of a successful model of midwifery-led continuity of carer, without continuity at birth: findings from a qualitative implementation evaluation. BMC Pregnancy Childbirth. 2021;21(1):1–11.

Forster DA, McLachlan HL, Davey MA, Biro MA, Farrell T, Gold L, et al. Continuity of care by a primary midwife (caseload midwifery) increases women’s satisfaction with antenatal, intrapartum and postpartum care: results from the COSMOS randomised controlled trial. BMC Pregnancy Childbirth. 2016;16(1):1–13.

Levy KN, Draijer N, Kivity Y, Yeomans FE, Rosenstein LK. Transference-focused psychotherapy (TFP). Curr Treat Options Psychiatry. 2019;6(4):312–24.

O’Brien A, Fahmy R, Singh SP. Disengagement from mental health services: a literature review. Soc Psychiatry Psychiatr Epidemiol. 2009;44(7):558–68.

Becker KD, Boustani M, Gellatly R, Chorpita BF. Forty years of Engagement Research in Children’s Mental Health Services: Multidimensional Measurement and practice elements. J Clin Child Adolesc Psychol. 2018;47(1):1–23.

Chorpita BF, Becker KD. Dimensions of Treatment Engagement among Youth and caregivers: structural validity of the REACH Framework. J Consult Clin Psychol. 2022;90(3):258–71.

Download references

Acknowledgements

We are grateful for the Community Perinatal Mental Health Teams, service users and their significant sources of support who took part in this study.

This study/project is funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research Programme (Ref: 17/49/38). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

Author information

Authors and affiliations.

Department of Primary Care and Mental Health, Institute of Population Health, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, UK

L. Fisher & H. Sharp

Mood Disorders Centre, Psychology Department, Faculty of Health and Life Sciences, University of Exeter, Washington Singer Laboratories, Perry Road, Exeter, EX4 4QG, UK

A. Davey, G. Collins, E. Makinde & Heather A O’Mahen

Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK

Department of Health and Community Sciences, University of Exeter Medical School, Exeter, UK

S. Morgan-Trimmer

Section of Women’s Mental Health, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK

L. M. Howard & J. Domoney

Department of Global Health and Social Medicine, School of Global Affairs, King’s College London, London, UK

K. H. Atmore

The Tavistock and Portman NHS Foundation Trust, London, UK

Centre for Parent and Child Support and Community Perinatal services, South London and Maudsley NHS Foundation Trust, London, UK

You can also search for this author in PubMed   Google Scholar

Contributions

H.O, L.H and H.S are co-chief investigators of the study.H.O, L.H, H.S, S.MT, G.W, J.B and C.M designed the study and developed the initial programme theory. A.D wrote and H.O, L.H, H.S, S.MT and G.W contributed to the study protocol.L.F, A.D and K.A contributed to data collection.L.F undertook analysis with S.MT and G.W contributing to and supervising data analysis. H.O, H.S, A.D, J.B and C.M also contributed to the data analysis process as the core analysis team, and K.A, G.C, E.M, and J.D were included in and contributed to discussions around analysis.L.F, H.O, J.D and G.C wrote the first draft of the manuscript.All authors contributed to and have approved the final manuscript.

Corresponding author

Correspondence to Heather A O’Mahen .

Ethics declarations

Ethics approval and consent to participate.

The study was approved by the South West - Central Bristol Research Ethics Committee (Reference: 19/SW/0218) and Health Research Authority (HRA reference 265175). All participants provided informed consent to participate in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Fisher, L., Davey, A., Wong, G. et al. Women’s engagement with community perinatal mental health services: a realist evaluation. BMC Psychiatry 24 , 492 (2024). https://doi.org/10.1186/s12888-024-05804-1

Download citation

Received : 15 December 2023

Accepted : 30 April 2024

Published : 08 July 2024

DOI : https://doi.org/10.1186/s12888-024-05804-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Service Engagement
  • Patient experience
  • Realist evaluation
  • Qualitative

BMC Psychiatry

ISSN: 1471-244X

critical evaluation of research article example

  • Open access
  • Published: 08 July 2024

Evaluation of cross-cultural adaptation and validation of the Persian version of the critical thinking disposition scale: methodological study

  • Hossein Bakhtiari-Dovvombaygi 1 , 2 ,
  • Kosar Pourhasan 1 ,
  • Zahra Rahmaty 3 ,
  • Akbar Zare-Kaseb 1 ,
  • Abbas Abbaszadeh 2 ,
  • Amirreza Rashtbarzadeh 1 &
  • Fariba Borhani 2 , 4  

BMC Nursing volume  23 , Article number:  463 ( 2024 ) Cite this article

Metrics details

Introduction

Assessing critical thinking disposition is crucial in nursing education to foster analytical skills essential for effective healthcare practice. This study aimed to evaluate the cross-cultural adaptation and validation of the Persian version of the Critical Thinking Disposition Scale among Iranian nursing students.

A total of 390 nursing students (mean age = 21.74 (2.1) years; 64% female) participated in the study. Face and content validity were established through feedback from nursing students and expert specialists, respectively. Construct validity was assessed using exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). The EFA was used to explore the number of factors and the items that were loading on them. The CFA was used to confirmed the fidnings of the EFA on the same sample. Convergent and discriminant validity were examined, along with reliability through internal consistency and test-retest reliability.

EFA revealed a two-factor structure, comprising “Critical Openness” and “Reflective Skepticism,” explaining 55% of the total variance. CFA confirmed the model’s fit (χ² = 117.37, df = 43, χ²/df = 2.73, p  < 0.001; RMSEA = 0.067; CFI = 0.95; TLI = 0.93, SRMR = 0.041). Convergent and discriminant validity were supported, with significant factor loadings ( p  < 0.001) ranging from 0.61 to 0.77. The CTDS exhibited strong internal consistency (α = 0.87) and excellent test-retest reliability (ICC = 0.96).

The validation of the CTDS in Persian language settings provides a reliable tool for assessing critical thinking disposition among Iranian nursing students. The two-factor structure aligns with previous research, reflecting students’ propensity towards critical openness and reflective skepticism. The study’s findings underscore the importance of nurturing critical thinking skills in nursing education.

Peer Review reports

Critical thinking can be seen as the ability to think logically, dynamically, comprehensively and practically when judging a situation to investigate and make appropriate decisions [ 1 , 2 ]. This ability helps to gain insight and examine an idea or concept from different perspectives [ 3 ]. Critical thinking has become an educational ideal, with most policy makers and educationists calling for the development of critical attitudes in students [ 4 ]. Critical thinking has been identified as one of the most important outcomes of higher education courses [ 5 ].

There is increasing evidence showing that critical thinking is considered an important part of preregistered nursing students and registered nurses when they are working in various clinical practice settings [ 6 , 7 ]. Critical thinking is one of the basic skills that prepares nursing students to effectively manage patient problems, make the best clinical decisions, provide safe and high-quality care, and better control critical situations. On the other hand, negative consequences such as depression, failure to solve patient problems, and incomplete clinical reasoning can be consequences of poor critical thinking [ 8 , 9 ].

Critical thinking is expected in nursing program graduates at the international level [ 10 , 11 ]. Therefore, it is important to evaluate and measure the levels of critical thinking of nursing students at different stages and education so that educators can adjust learning activities to ensure the desired results [ 2 , 12 , 13 ]. Educators are the ones who are responsible for and have the opportunity to shape this skill during the years of education and trayning new generations [ 14 ]. Despite the importance of critical thinking in the nursing profession, studies have reported a lack of critical thinking skills among undergraduate students in the field [ 15 , 16 ].

Critical thinking has two main components: critical thinking skills and critical thinking disposition (CTD). The skills component refers to the cognitive processes of thinking, while the disposition component refers to personal desire and internal motivation for critical thinking [ 9 ]. Several studies have highlighted the need for reliable assessment tools for critical thinking, specifically in nursing, rather than in a general context [ 17 , 18 , 19 ].

To our knowledge, and based on our literature review, the CTD is the only specific tool for assessing the tendency to think critically. However, this tool has not been used or validated in the Iranian educational context, population and language. Considering the lack of effective tools for evaluating CTD in undergraduate nursing programs in Iran, the purpose of this study was to translate and evaluate the psychometric properties of the Persian version of CTD among nursing students.

Study design

This was a cross-sectional study utilizing cross-cultural adaptation to translate and investigate the validity and reliability of the CTDS for use among Iranian nursing students [ 20 ]. The translated scale underwent examination for reliability and validity tests.

Study population and sampling

Convenience sampling was employed at the School of Nursing, Shahid Beheshti University in Tehran. This method involved selecting participants who were readily available and willing to take part in the study. Specifically, the study targeted all undergraduate nursing students, who were invited to participate in the research. Recruitment continued until the desired sample size was achieved. To maintain the integrity of the data, students who submitted incomplete questionnaires were excluded from the analysis. Undergradute studies in Iran normally involve 4 years of education in general nursing, as well as clinical rotations in all hospital units and public health sectors.

There are two general recommendations concerning the minimum sample size necessary for conducting factorial analysis. The first recommendation emphasizes the significance of the absolute number of cases (N), while the second recommendation highlights the importance of the subject-to-variable ratio (p). Guilford suggested that N should be no less than 200 [ 21 ]. Additionally, MacCallum et al. recommended that the subject-to-variable ratio should be at least 5 [ 22 ]. A total of 390 nursing students voluntarily participated in the study.

Measurements

The CTDS, developed by Sosu, is an instrument used to measure the dispositional dimension of critical thinking [ 23 ]. Self-report questionnaires were given to the students. The demographic questionnaire collected included participants’ age, gender, education, and grade point average (GPA). The Critical Thinking Disposition Scale (CTDS) was used to measure the dispositional aspect of critical thinking. This scale comprises 11 items, employing a five-point Likert-type response format (1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree). Total scores range from 11 to 55. The subscores include the first seven items reflecting a level of critical openness with a score ranging from 7 to 35 and the last four items indicating a level of reflective skepticism with a score ranging from 4 to 20. Higher CTDS scores indicate a greater degree of critical thinking [ 23 ].

Translation of the CTD scale

Following correspondence with the instroment developer, Dr. Sosu, and obtaining permission, the scale underwent translation using the standard Backward-Forward method. Initially, the scale was independently and simultaneously translated from English to Persian by two translators proficient in both Farsi and English. In the subsequent phase, these translations were juxtaposed and merged into a unified translation. This facilitated the comparison and identification of discrepancies, which were then rectified based on feedback from a panel of experts, including two psychometric experts and two nursing professors. In the third stage, the resulting Persian version was given to two translators fluent in Persian and English (distinct from those in the initial experts) to translate it back to English, thereby completing the translation of the scale from Persian to English. In the fourth stage, the two English translations were compared, and any disparities were resolved by the experts, culminating in a single translation. Subsequently, the prefinal version was evaluated for content and face validity.

Face validity

Face validity refers to the degree to which a measurement method seems to accurately measure the intended construct [ 24 ]. In a qualitative assessment of face validity, 10 nursing students were asked to evaluate factors such as the clarity of phrases and words, the coherence and relevance of items, the potential for ambiguity, and the necessity of removing or combining items. Additionally, two nursing professors and two psychometric specialists scrutinized the scale to determine whether it indeed appeared to measure its intended construct.

Content validity

Content validity examines the extent to which a collection of scale items aligns with the pertinent content domain of the construct it aims to assess [ 24 ]. The qualitative evaluation of the content validity involved a panel consisting of two nursing professors in the field of nursing education and two statisticians who were experts in psychometric topics. Their input on item placement, word selection, grammar adherence, and scoring accuracy of the scale and its instructions were solicited, with their feedback serving as the foundation for any required adjustments.

Construct validity

Exploratory factor analysis.

To explore the number of existing subscales and potential factors, Exploratory Factor Analysis (EFA) was performed using Principal Component Analysis (PCA) with varimax rotation. The scree plot and parallel analysis suggested the number of existing factors. The scree plot displays the eigenvalues of each factor extracted from the data in descending order. The number of factors was retained by examining the slope of the curve. A sharply decreasing slope indicates the optimal number of factors that capture the most variance in the data.

Confirmatory factor analysis

To confirm the findings of the EFA, Confirmatory Factor Analysis (CFA) was conducted using MPLUS to confirm the 2-factor structure identified with the items loaded on each factor in the EFA. Model fit indices, including Comparative Fit Index (CFI), the standardized residual root mean squared error (SRMR) of approximation and the root mean squared error of approximation (RMSEA), with cutoff points of > 0.95, < 0.08 and < 0.06, respectively, were used [ 25 ]. Factor loadings were reported using standardized beta coefficients to evaluate the strength of the relationships between items and factors, and a p value of 0.05 was considered a significant factor loading.

Convergent and discriminant validity

The mean scores for Critical Openness and Reflective Skepticism were computed. Convergent and Discriminant Validity was checked for correlations between students’ GPA as an indicator of academic achievement and the scores of the subscales of the CTDS.

  • Reliability

To assess reliability, internal consistency was tested using Cronbach’s alpha coefficient calculations for the total score and subscores. To assess the consistency of the test-retest approach over a two-week period, a group of 40 individuals from the target demographic underwent examination. Their scores from both sessions were analyzed to determine test-retest reliability, and the Intraclass Correlation Coefficient (ICC) was calculated.

Ethical considerations

The current study underwent assessment and received approval from the Research Ethics Committee at Shahid Beheshti University of Medical Sciences (Ethical code: IR.SBMU.RETECH.REC.1403.013). Permissions were duly acquired from the pertinent authorities at the research sites as well as the developer of the original scale. Nursing students were provided with comprehensive information regarding the study’s objectives, their right to withdraw from participation, and the confidentiality of their data. Informed consent was obtained from all participating students. All procedures adhered strictly to the appropriate guidelines and regulations.

Data analysis

Descriptive statistics, including the mean, standard deviation, median, range, and frequency, were used to describe the population and their critical thinking scores. Analysis of demographic characteristics, EFA, and reliability tests were performed using IBM SPSS Statistics (Version 27). CFA was performed using MPLUS (Program Copyright © 1998–2017 Muthén & Muthén Version 8) software.

Characteristics of the participants

A total of 390 participants (mean age = 21.74 (2.1) years; 64% female) completed the questionnaire.

Face validity and content validity

Face validity was established through the feedback of 10 nursing students, while content validity was assessed by four expert specialists. No alterations were made to the items in terms of their simplicity and clarity during the evaluation of both face and content validity.

The scree plot and parallel analysis suggested a two-factor solution (Fig.  1 ), which accounted for 55% of the total variance in the scores (Table  1 ). Factor loadings revealed a clear factor structure, with items loading on two factors. The rotated factor loadings are presented in Table  2 . The items clustered together on two distinct factors, with no cross-loadings observed.

figure 1

Scree Plot of the Persian version of the Critical Thinking Disposition Scale

The two factors were interpreted as follows: Factor 1, labeled “Critical Openness,” comprised items related to the level of critical openness; Factor 2, labeled “Reflective skepticism” included items reflecting the level of reflective skepticism. These factors align well with previously established dimensions.

CFA confirmed the model including the two factors with their respective indicators based on the EFA results. The CFA model demonstrated acceptable fit to the data: χ² (55) = 1500.38, p  < 0.001; RMSEA = 0.067; CFI = 0.95; TLI = 0.93, SRMR = 0.041. Although the chi-square test was significant, other fit indices indicated a reasonably good fit to the data.

The standardized factor loadings ranged from 0.61 to 0.77, Fig.  2 , all of which were statistically significant ( p  < 0.001) (Table  3 ). These loadings provided further support for the factor structure identified in the EFA.

figure 2

Factor structure of the Persian version of the Critical Thinking Disposition Scale

Convergent and discriminant validity were examined by determining factor scores based on item allocation. The analysis revealed a sample mean of 28.65 (SD = 2.7) for the Critical Openness factor and a mean of 16.8 (SD = 1.8) for Reflective Skepticism. A weak yet statistically significant correlation was observed between students’ GPA and their level of critical openness ( r  = 0.15, p  = 0.003), indicating a slight association between academic performance and this aspect of cognitive disposition.

The reliability of the CTDS-P was assessed through rigorous statistical analysis. Internal consistency was robust, as indicated by a Cronbach’s alpha coefficient of 0.87 for the overall scale, demonstrating the coherence of the items within the measure. Subscale analysis revealed strong reliability, with values of 0.83 for critical openness and 0.80 for reflective skepticism, indicating the consistency of responses across different dimensions of the construct. Additionally, the scale exhibited excellent test-retest reliability, as evidenced by an Intraclass Correlation Coefficient (ICC) of 0.96, with a 95% confidence interval ranging from 0.93 to 0.98, suggesting high stability and consistency of the scores over time.

The CTDS has undergone translation and cross-validation in different populations across the USA [ 26 ], Norway [ 27 ], Brazil [ 28 ], Spain [ 29 ], Turkey [ 30 ], and Vietnam [ 31 ]. The reliability and validity of this scale have been demonstrated in studies conducted among high school students [ 29 ] and university students [ 26 , 27 , 28 , 30 , 31 ]. The CTDS was first introduced as a tool to measure critical thinking disposition in undergraduate and postgraduate students [ 4 ].

This study used comprehensive reliability and validity tests to validate the CTDS in the Persian language and in Iranian nursing student participation. This study revealed the existence of two factors, critical openness and reflective skepticism. These factors align well with previously established studies [ 4 , 27 , 30 , 31 ]. Conversely, Spanish, Brazilian, and US versions demonstrated that the one-factor model fit better for their population [ 26 , 28 , 29 ].

In the face validity and content validity tests, neither the simplicity nor the clarity of the items were altered. The validity of the content was done qualitatively. Similar to previous quantitatively measured studies, our study has also confirmed the validity of the content [ 28 , 31 ].

The internal consistency of the CTDS demonstrated the coherence of the items within the measure. Several studies have reported similar internal consistency values for the CTDS, with Cronbach’s alpha measuring 0.88 according to Nguyen et al. 2023. Sosu et al. 2013, Akin et al. 2015, Yockey 2016, Bravo et al. 2020, and Gerdts-Andresen et al. 2022 also reported values of 0.79, 0.78, 0.79, 0.77, and 0.76, respectively. It is widely recognized that a Cronbach’s alpha coefficient of 0.70 or higher is acceptable [ 32 ]. Consequently, the CTDS has exhibited strong internal consistency across diverse linguistic and cultural contexts. In addition, the scale exhibited excellent test-retest reliability, thereby indicating a high level of stability and consistency in scores across time. Additionally, our study demonstrated an outstanding ICC [ 33 , 34 ].

There are several limitations to this study. Initially, the self-assessment survey may have been prone to social desirability bias, leading to potential overestimation of reported measures. To mitigate bias, this study utilized an anonymous survey. Moreover, the study used a cross-sectional design, which prevented the establishment of prospective predictive validity.

To conclude, our investigation establishes the CTDS as a reliable and valid tool for evaluating critical thinking disposition among Iranian nursing students. With its two-factor structure of “Critical Openness” and “Reflective Skepticism,” the scale offers valuable insights into cognitive disposition. Its robust psychometric properties underscore its potential for enhancing critical thinking skills in healthcare education and practice. Further research avenues may explore its nuanced applications in fostering analytical reasoning and problem-solving abilities.

Data availability

The datasets generated and/or analysed during the current study are not publicly available due to the necessity to ensure participant confdentiality policies and laws of the country but are available from the corresponding author on reasonable request.

Susiani TS, Salimi M, Hidayah R, editors. Research based learning (RBL): How to improve critical thinking skills? SHS Web of Conferences; 2018: EDP Sciences.

Nes AAG, Riegel F, Martini JG, Zlamal J, Bresolin P, Mohallem AGC, et al. Brazilian undergraduate nursing students’ critical thinking need to be increased: a cross-sectional study. Revista brasileira de enfermagem. 2022;76:e20220315.

Article   PubMed   PubMed Central   Google Scholar  

Wang Y, Nakamura T, Sanefuji W. The influence of parental rearing styles on university students’ critical thinking dispositions: the mediating role of self-esteem. Think Skills Creativity. 2020;37:100679.

Article   Google Scholar  

Sosu EM. The development and psychometric validation of a critical thinking Disposition Scale. Think Skills Creativity. 2013;9:107–19.

Hart C, Da Costa C, D’Souza D, Kimpton A, Ljbusic J. Exploring higher education students’ critical thinking skills through content analysis. Think Skills Creativity. 2021;41:100877.

Zhang C, Fan H, Xia J, Guo H, Jiang X, Yan Y. The effects of reflective training on the disposition of critical thinking for nursing students in China: a controlled trial. Asian Nurs Res. 2017;11(3):194–200.

Organization WH. Global standards for the initial education of professional nurses and midwives. World Health Organization; 2009.

Organization WH. State of the world’s nursing 2020: investing in education, jobs and leadership. 2020.

Dehghanzadeh S, Jafaraghaee F. Comparing the effects of traditional lecture and flipped classroom on nursing students’ critical thinking disposition: a quasi-experimental study. Nurse Educ Today. 2018;71:151–6.

Article   PubMed   Google Scholar  

Kermansaravi F, Navidian A, Kaykhaei A. Critical thinking dispositions among junior, senior and graduate nursing students in Iran. Procedia-Social Behav Sci. 2013;83:574–9.

Carvalho DPSRP, Vitor AF, Cogo ALP, Bittencourt GKGD, Santos VEP, Ferreira Júnior MA. Measurement of general critical thinking in undergraduate nursing students: experimental study. Texto Contexto-Enfermagem. 2019;29:e20180229.

Swing VK. Early identification of transformation in the proficiency level of critical thinking skills (CTS) for the first semester associate degree nursing (ADN) student. Capella University; 2014.

Dembitsky SL. Creating capable graduate nurses: a study of congruence of objectives and assessments of critical thinking in a southeastern state. Capella University; 2010.

López M, Jiménez JM, Martín-Gil B, Fernández-Castro M, Cao MJ, Frutos M, et al. The impact of an educational intervention on nursing students’ critical thinking skills: a quasi-experimental study. Nurse Educ Today. 2020;85:104305.

Wong SHV, Kowitlawakul Y. Exploring perceptions and barriers in developing critical thinking and clinical reasoning of nursing students: a qualitative study. Nurse Educ Today. 2020;95:104600.

Azizi-Fini I, Hajibagheri A, Adib-Hajbaghery M. Critical thinking skills in nursing students: a comparison between freshmen and senior students. Nurs Midwifery Stud. 2015;4(1).

Carter AG, Creedy DK, Sidebotham M. Measuring critical thinking in pre-registration midwifery students: a multi-method approach. Nurse Educ Today. 2018;61:169–74.

Scheffer BK, Rubenfeld MG. A consensus statement on critical thinking in nursing. SLACK Incorporated Thorofare, NJ; 2000. pp. 352–9.

Paul SA. Assessment of critical thinking: a Delphi study. Nurse Educ Today. 2014;34(11):1357–60.

Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine. 2000;25(24):3186–91.

Article   CAS   PubMed   Google Scholar  

Guilford JP. Psychometric methods. 1954.

Zhang S, Hong S. Sample size in factor analysis. Psychol Methods. 1999;4(1):84–99.

Sosu E. The development and psychometric validation of a critical thinking disposition scale. Think Skills Creativity. 2013;9:107–19.

Wellington J, Szczerbinski M. Research methods for the social sciences. A&C Black; 2007.

Hancock G, Mueller R. Stractural equation modeling a second cource (second edi) information age publishing. 2020.

Yockey RD. Validation study of the critical thinking dispositions scale: a brief report. North Am J Psychol. 2016;18(1).

Gerdts-Andresen T, Hansen MT, Grøndahl VA. Educational Effectiveness: validation of an instrument to measure students’ critical thinking and Disposition. Int J Instruction. 2022;15(1):685–700.

Luiz FS, Leite ICG, Carvalho PHBd, Püschel VAA, Braga LM, Dutra HS, et al. Validity evidence of the critical thinking Disposition Scale, Brazilian version. Acta Paulista De Enfermagem. 2021;34:eAPE00413.

Bravo MJ, Galiana L, Rodrigo MF, Navarro-Perez JJ, Oliver A. An adaptation of the critical thinking disposition scale in Spanish youth. Think Skills Creativity. 2020;38:100748.

Akın A, Hamedoglu MA, Arslan S, Akın Ü, Çelik E, Kaya Ç, et al. The adaptation and validation of the Turkish version of the critical thinking Disposition Scale (CTDS). Int J Educational Researchers. 2015;6(1):31–5.

Google Scholar  

Nguyen TV, Kuo CL, Wang CY, Le NT, Nguyen MTT, Chuang YH. Assessment of the psychometric properties of the Vietnamese version of the critical thinking Disposition Scale. Nurse Educ Today. 2023;127:105848.

Ursachi G, Horodnic IA, Zait A. How reliable are measurement scales? External factors with indirect influence on reliability estimators. Procedia Econ Finance. 2015;20:679–86.

Weir JP. Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. J Strength Conditioning Res. 2005;19(1):231–40.

Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–63.

Download references

Acknowledgements

The authors would like to thank the nursing student who participated in this study.

This research did not receive any specific grant from funding agencies in the public, commercial, or not for profit sectors.

Author information

Authors and affiliations.

Student Research Committee, School of Nursing and Midwifery, Shahid Beheshti University of Medical Sciences, Tehran, Iran

Hossein Bakhtiari-Dovvombaygi, Kosar Pourhasan, Akbar Zare-Kaseb & Amirreza Rashtbarzadeh

Medical Ethics and Law Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran

Hossein Bakhtiari-Dovvombaygi, Abbas Abbaszadeh & Fariba Borhani

Institute of Higher Education and Research in Healthcare, Department of Biology and Medicine, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland

Zahra Rahmaty

School of Nursing & Midwifery, Shahid Beheshti University of Medical Sciences, Tehran, Iran

Fariba Borhani

You can also search for this author in PubMed   Google Scholar

Contributions

HBD, and FB, was involved in the conception and organization of the study. KP, AZK, and AR were involved in the execution and data collection of the study; ZR and AA participated in statistical analysis design and/or execution. KP, HBD and AZK, prepared the first draft of the manuscript. All authors contributed to the preparation, critical review and all of them approved the final manuscript.

Corresponding author

Correspondence to Fariba Borhani .

Ethics declarations

Ethics approval and consent to participate, consent for publication.

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Bakhtiari-Dovvombaygi, H., Pourhasan, K., Rahmaty, Z. et al. Evaluation of cross-cultural adaptation and validation of the Persian version of the critical thinking disposition scale: methodological study. BMC Nurs 23 , 463 (2024). https://doi.org/10.1186/s12912-024-02129-y

Download citation

Received : 14 May 2024

Accepted : 27 June 2024

Published : 08 July 2024

DOI : https://doi.org/10.1186/s12912-024-02129-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Critical thinking
  • Nursing student
  • Psychometric properties

BMC Nursing

ISSN: 1472-6955

critical evaluation of research article example

  • Open access
  • Published: 09 July 2024

Evaluation of the sentinel yellow fever surveillance system in Uganda, 2017–2022: strengths and weaknesses

  • Mercy Wendy Wanyana 1 ,
  • Patrick King 1 ,
  • Richard Migisha 1 ,
  • Benon Kwesiga 1 ,
  • Paul Edward Okello 1 ,
  • Daniel Kadobera 1 ,
  • Lilian Bulage 1 ,
  • Joshua Kayiwa 2 ,
  • Annet Martha Nankya 3 ,
  • Alex Riolexus Ario 2 &
  • Julie R. Harris 4  

BMC Infectious Diseases volume  24 , Article number:  686 ( 2024 ) Cite this article

Metrics details

Uganda has a sentinel surveillance system in seven high-risk sites to monitor yellow fever (YF) patterns and detect outbreaks. We evaluated the performance of this system from 2017 to 2022.

We evaluated selected attributes, including timeliness (lags between different critical time points), external completeness (proportion of expected sentinel sites reporting ≥ 1 suspect case in the system annually), and internal completeness (proportion of reports with the minimum required data elements filled), using secondary data in the YF surveillance database from January 2017–July 2022. We conducted key informant interviews with stakeholders at health facility and national level to assess usefulness, flexibility, simplicity, and acceptability of the surveillance system.

In total, 3,073 suspected and 15 confirmed YF cases were reported. The median time lag from sample collection to laboratory shipment was 37 days (IQR:21–54). External completeness was 76%; internal completeness was 65%. Stakeholders felt that the surveillance system was simple and acceptable, but were uncertain about flexibility. Most (71%) YF cases in previous outbreaks were detected through the sentinel surveillance system; data were used to inform interventions such as intensified YF vaccination.

The YF sentinel surveillance system was useful in detecting outbreaks and informing public health action. Delays in case confirmation and incomplete data compromised its overall effectiveness and efficiency.

Peer Review reports

Yellow fever (YF) is a vaccine-preventable acute viral haemorrhagic fever, transmitted by Aedes mosquitoes from and between non-human and human primates [ 1 ]. YF outbreaks can have high case-fatality rates (up to 50%), and there is no specific treatment for infection [ 2 ]. Recently, YF has re-emerged in several African countries, including Uganda, that lie in the ‘ yellow fever belt , ’ characterised by the presence of equatorial rainforest [ 3 , 4 , 5 ]. The continued presence of YF is enabled by ongoing low vaccination coverage, inadequate disease surveillance and response, and low commitment to YF prevention initiatives [ 6 ].

Since 2016, Uganda reported four YF outbreaks with a total of 68 suspect cases and 21 confirmed cases [ 7 , 8 , 9 ]. In these outbreaks, incidence ranged from 3 to 13 cases per 100,000 population with a case-fatality rate of 33% [ 7 , 10 ]. This re-emergence presents new demands on disease surveillance, especially as Uganda joins the rest of the world to attempt to achieve the World Health Organization’s target to eliminate YF epidemics by 2026 [ 11 ].

In Uganda, YF vaccination rates remain low despite the high risk [ 10 ], underscoring the need for an effective surveillance system. Surveillance for YF is vital to inform the need for intensified, targeted YF vaccination and to contain outbreaks. Currently, Uganda has both passive and sentinel surveillance systems for YF. The passive surveillance system uses the Integrated Disease Surveillance and Response (IDSR) approach, in which suspected YF cases are routinely reported from all health facilities together with other reportable diseases [ 12 ]. In the sentinel surveillance system, seven health facilities in central and western Uganda routinely detect suspected YF cases, collect samples, and send them to the Uganda Virus Research Institute where they are tested for YF [ 13 ]. However, the limited access to effective diagnostics for passive surveillance, difficulties in the clinical recognition of the disease, and late reporting [ 3 , 14 , 15 , 16 ] may limit the overall effectiveness of the YF surveillance systems.

Currently, there is limited evidence on the effectiveness of the YF sentinel surveillance system in Uganda. We described sentinel YF surveillance in Uganda, mapped the surveillance system’s processes, evaluated them to identify strengths and weaknesses, and suggested recommendations aimed at improving YF surveillance in Uganda.

Study setting

The evaluation was conducted in Uganda, a YF-endemic country in Eastern Africa [ 17 ]. Uganda is characterised as a high-risk country for YF transmission due to its low vaccination coverage (4% in 2022) and sporadic outbreaks occurring every three to five years [ 18 ].

Study design

We based our evaluation on the US Centers for Disease Control and Prevention’s (US CDC) Updated Guidelines for Evaluating Public Health Surveillance Systems. The guidelines provide a framework with which to assess a system’s performance using key attributes of timeliness, completeness, usefulness, acceptability, flexibility and simplicity [ 19 ].

Study variables and data collection

Study variables.

For timeliness, we calculated the median time lag between the onset of symptoms and sample collection, the time lag between sample collection and dispatch of the sample to the lab, and the time lag between dispatch to the lab and receipt at the laboratory. For external completeness, we calculated the percentage of the expected YF sentinel sites reporting suspected cases to the YF surveillance system between 2017 and 2022.

Internal completeness was calculated as the proportion of suspected case records with all minimum 10 data elements completed as indicated in the WHO Recommended Surveillance Standards, Second edition [ 20 ]. Minimum data elements for case-based reporting for YF include a unique identifier, geographic area (district of residence), date of birth, date of onset of symptoms, ever received YF vaccine, date of sample receipt at the laboratory, tests done, date of result, final classification, and outcome.

We assessed usefulness using stakeholders’ perceptions about actions taken as a result of surveillance outputs. We assessed acceptability using the perceived willingness of key stakeholders to participate in the surveillance system. We assessed flexibility using stakeholders’ perceptions of the ability of the surveillance system to cope with changes. We defined simplicity as perceived ease of performing tasks in the YF surveillance system.

Data collection

We obtained data on the YF surveillance system’s purpose, operations, current implementation, and processes using a topic guide and process map. Additionally, we reviewed relevant documents, including guidelines and reports, to obtain more information on the system’s processes and current implementation using a document review guide. We then assessed the surveillance system’s timeliness, external completeness and internal completeness using data from the YF surveillance database. Furthermore, we assessed the system’s usefulness, acceptability, flexibility, and simplicity. To assess these attributes (usefulness, acceptability, flexibility, and simplicity) we purposively selected key stakeholders at the Uganda Virus Research Institute, the Ministry of Health and sentinel site health facilities and conducted key informant interviews.

Data management and analysis

We used Epi info 7 software (CDC, Atlanta, USA) to analyse quantitative data using descriptive statistics. We assessed the trend in the time lags between onset of symptoms and sample collection and the time lag between collection of samples, and dispatch to the laboratory using the Mann-Kendall test. We used the chi-square test to assess the difference in positivity rates across regions.

For qualitative data, we transcribed audio-recorded interviews and analysed data using a deductive thematic analysis in Atlas ti 7 software (Scientific Software Development GmbH, Berlin Germany). Transcripts were coded and themes were generated based on pre-conceived themes on the YF surveillance system’s usefulness, acceptability, flexibility, and simplicity.

Description and operations of the YF sentinel surveillance system, Uganda, 2017–2022

YF sentinel surveillance is part of the arbovirus surveillance system established by the Department of Arbovirology, Emerging and Re-Emerging Infectious Diseases at UVRI in 2013. The surveillance system aims to (1) prevent outbreaks of arboviral diseases through early detection, diagnosis, and identification within the region; (2) provide risk assessments of the different emerging viruses (transmission, spread, human impact); and (3) recommend and implement public health measures for control where possible.

For YF, the surveillance system has seven sentinel sites (Fig.  1 ). These sites were enrolled in a phased manner, starting with St. Francis Nkonkonjeru in 2013; Bukakata Health Centre III, Kisubi Hospital, St Ulrika Health Centre III, Entebbe Regional Referral Hospital, and Nyamirami Health Centre in 2017; and Bundibugyo Hospital in 2020. These sites were selected based on previous entomological studies that identified the presence of YF vectors ( Aedes aegypti mosquito species) carrying viruses of the Flaviviridae family in the areas where these facilities are located. The surveillance systems target at-risk YF populations based on their proximity to these “high-risk environments.”

figure 1

Location of YF sentinel surveillance sites in Uganda, 2017– 2022

Processes in the YF sentinel surveillance system, Uganda, 2017–2022

The YF sentinel surveillance system starts with a clinician in a sentinel site suspecting YF based on a case definition of acute onset of fever that is not responsive to malaria treatment. However, clinicians reported often using their clinical judgement to identify suspect YF cases instead of the case definition. A blood sample of ≥ 2 ml is collected from the patient in a sterile vacutainer and blood is centrifuged to separate serum. The serum is separated and stored in a liquid nitrogen tank until the collection date (once every month). Samples are transferred to UVRI for testing by the UVRI YF sentinel surveillance project team. Additionally, every suspected viral haemorrhagic fever sample submitted to UVRI that is negative for other viral haemorrhagic fevers is submitted for YF testing. Samples are tested using the YF testing algorithm in the WHO guidelines [ 21 ]. Interpretation of results is based on vaccination history, date of onset of symptoms, date of sample collection, and travel history. A positive PCR test from a person with no history of recent YF vaccination OR positive anti-viral immunoglobulin M (IgM) test in a person with no history of vaccination and Plaque Reduction Neutralisation Test (PRNT) titre with YF antibodies detected at ≥ 4 times greater than PRNT titres for dengue, West Nile and Zika flaviviruses is considered a positive test for YF. The laboratory testing process takes a total of 2 weeks. Results are entered into the database and shared with MoH and WHO every week.

Personnel and tasks in the YF sentinel surveillance system, Uganda, 2017–2022

There were various personnel responsible for the different tasks within the sentinel surveillance system (Table  1 ). These include clinicians, laboratory technicians, drivers, laboratory staff, project management staff, and information analysts.

Availability of surveillance guidelines, documents, and protocols for the YF surveillance system, Uganda, 2017–2022

All YF sentinel sites reported having received a written protocol and guidelines on sample collection and storage at the establishment of the sentinel sites. However, at the time of the visit, only 3 of the sites had a copy of these documents available.

Data flow, reporting, feedback mechanism, and data management for the YF surveillance system, Uganda, 2017–2022

Sentinel health facilities collect data on suspected cases using the paper-based UVRI Viral Haemorrhagic Fever case investigation form (Fig.  2 ). Additionally, a summary of the patient information (name, residence, age, and sample collection date) is recorded in a book at the health facility. All completed, hard copy case investigation forms are submitted to UVRI, together with their corresponding sample. Following testing, data are entered into an EpiInfo database and sent in a Microsoft Access data to the Ministry of Health Public Health Emergency Operations Centre (MoH PHEOC) and the World Health Organization every week. The MoH PHEOC reviews the database every week, if there are any positive cases reported, the MoH PHEOC notifies the District Surveillance Focal Persons through email, phone calls, and SMS Alerts. Data are not routinely analysed by person, place, and time.

figure 2

Data flow reporting and feedback mechanism in the YF surveillance system, Uganda, 2017–2022

Performance of the sentinel YF surveillance system based on key attributes, Uganda, 2017–2022

Over the review period, a total of 3,073 suspect cases with samples collected were reported in the YF sentinel surveillance system. Of these, 15 (0.49%) were confirmed as YF cases.

The overall median number of days between the onset of symptoms and sample collection was 3 [Interquartile Range (IQR = 1–5)]. Confirmed cases had a shorter median time between onset of symptoms and sample collection was 2 [Interquartile Range (IQR = 1–10)] (Fig.  3 C). The overall median number of days between sample collection and dispatch of the sample to the lab was 37 (IQR = 21–54). Confirmed cases had a longer median time between sample collection and dispatch of sample to the laboratory was 47 [Interquartile Range (IQR = 34–69)] (Fig.  3 D). The shortest time lag occurred between sample dispatch and receipt at the laboratory (median: 1 day, IQR = 0–1). From 2017 to 2022, the overall median time lag between onset of symptoms and sample collection reduced from 5 to 2 days ( p  = 0.001) (Fig.  3 A). Similarly, the overall time lag between collection of samples and dispatch to the laboratory reduced from 36 to 28 days ( p  = 0.002) (Fig.  3 B).

figure 3

Time lags between onset of symptoms, sample collection and dispatch of samples to the laboratory, A Overall time lag between the onset of symptoms and sample collection, 2017–2022, B Overall time lag between sample collection and collection of sample and dispatch to the laboratory, 2017–2022, C time lag between the onset of symptoms and sample collection, for confirmed cases 2017–2022, D The time lag between sample collection and collection of sample and dispatch to the laboratory for confirmed cases, 2017–2022

Completeness

External completeness.

With the exception of 2022, there was an increase in the number of samples submitted each year (Table  2 ). From 2017 to 2019, external completeness was 100%, with all the YF sentinel sites reporting at least one suspected case into the system each year. From 2020 to 2022, one site did not report any suspected cases. During this period, external completeness was 86%. The central region had higher overall positivity rates in comparison with the western region, 0.75% Vs 0.12%, p  = 0.014.

Internal completeness

Overall internal completeness was 65%, ranging from 56% in 2017 to 76% in 2022 (Fig.  4 ). Of the ten data fields analysed for internal completeness, unique identification number, age, YF IgM results, and PRNT tests were the most often completed data fields (> 95%). Vaccination status, IgM result release date, and PRNT result release dates were the least reported fields at < 1%.

figure 4

Internal completeness of data reported in the YF sentinel surveillance system, Uganda, 2017–2022

According to the respondents, the YF sentinel surveillance system met its objectives of detecting YF outbreaks and on-going YF transmission in communities and informing prevention and control interventions. Of the 21 confirmed cases detected during 2019–2022, 15 (71%) were detected through the YF surveillance system.

“…the sentinel YF surveillance system is very helpful in detecting YF cases…in previous outbreaks , most of the index YF cases were identified through the sentinel surveillance system….”

MoH official, Public Health Emergency Operations Centre

Additionally, key informants reported that the sentinel YF surveillance system provided data that helped guide decisions on the selection of interventions for YF prevention and control.

… the YF sentinel surveillance system has enabled us to identify which communities are at risk of YF transmission… as a result , these have been prioritised in the intensified YF vaccination campaigns last year ….

MoH official, Department of Integrated Epidemiology, Surveillance and Public Health Emergencies

…currently we use data obtained from the YF sentinel surveillance system to advocate for the introduction of the YF vaccine into the routine vaccination schedule…this will start soon …and if we didn’t have those data to justify our recommendations, they would not be taken up….

Acceptability

Respondents reported willingness, commitment, and interest in participating in the YF surveillance system. In particular, stakeholders at YF sentinel sites reported that their active involvement in the running of the YF surveillance system made it more acceptable to them.

… We are very committed and willing to participate in the YF surveillance system. For no extra pay , we continuously assess and report any suspected YF cases to UVRI….

Health worker at a YF sentinel site

… all of us, including community members, are actively involved in the YF surveillance system and our contributions are valued…we have regular discussions with project managers from UVRI on how we can improve YF surveillance….

Flexibility

Respondents reported uncertainty about the YF sentinel surveillance system’s capacity to adapt to the changing surveillance needs. Reasons cited for this uncertainty included not having experienced a change in YF surveillance needs, and lack of funding to support changes even if needed.

… We have not experienced any changes in the YF surveillance system…we , therefore , cannot tell how flexible the system is… .

MoH official, department of integrated epidemiology, surveillance and public health emergencies

“… Due to limited funding , the sentinel YF surveillance system cannot be as flexible as we would love it to be…. for example , we cannot have more sentinel sites even if we wanted. For example , we experienced a YF outbreak in the northern region of the country in 2016 suggesting a need for enhanced surveillance in this area but no surveillance sites have been set up in this area due to [lack of] funding…” .

MoH official, Department of integrated Epidemiology, Surveillance and Public Health Emergencies

The respondents’ tasks within the sentinel surveillance system were reported as being easy to do.

… the processes are easy and simple…all you have to do is the case investigation form which is easy to fill and wait for UVRI to pick the form ….

Prevention and control of YF outbreaks require a reliable and effective surveillance system. We described Uganda’s YF sentinel surveillance system, mapped the surveillance system’s processes, and evaluated them to identify strengths and weaknesses. During 2017–2022 in Uganda, most sentinel surveillance sites actively submitted samples, with an increase in samples submitted over time. Most sites submitted at least one positive sample during the evaluation period. We noted limited use of case definitions, with clinicians more often using individual clinical judgement and expertise to identify suspected cases, as well as delays in case confirmation and incomplete data.

In malaria-endemic areas like Uganda where acute febrile illnesses are common, non-malarial acute febrile illnesses such as YF may be missed. In these settings, laboratory-based arbovirus sentinel surveillance improves detection of YF which is key for achieving the WHO target of eliminating YF epidemics by 2026 [ 22 ]. Most sites submitted samples over the evaluation period. Continuous submission of samples by sentinel sites is critical for the functionality of YF surveillance system [ 23 ]. One site, Entebbe Regional Referral Hospital, stopped submitting samples in 2020, during a period when it was restricted to providing only COVID-19 management services. We observed an increase in the number of samples over time, with positivity rates similar to those previously documented in Uganda and the sentinel YF surveillance system in the United Republic of Tanzania [ 24 , 25 ]. Generally, sites in the central region had higher positivity rates compared to sites in the western region. This supports previous findings that have indicated a higher seroprevalence, and likely more circulation, of YF in the central region [ 25 ].

Interestingly, some sites had no cases identified throughout the period despite submission of large numbers of samples. It is possible that the limited use of case definitions and reliance on clinical judgement could have contributed to this, especially with less skilled health workers [ 26 ]. In Uganda, lower cadres sometimes perform duties of specialised health workers with minimum supervision [ 27 , 28 ]. In this situation, ‘over-testing’ as a result of using clinical judgment instead of the case definition could waste resources in an already resource-constrained system [ 29 ]. This may also be true even at the sites submitting some positive samples. However, the sites without any positives also generally submitted fewer samples than the sites with positive samples. The use of simplified validated standard case definitions provided in the Uganda National Technical Guidelines on Integrated Disease Surveillance and Response 3rd Edition could standardise diagnosis and improve identification of suspected cases within the surveillance system [ 12 ]. This will improve the performance of the surveillance system as correct knowledge and use of standard case definitions among health workers at sentinel sites is essential for an effective surveillance system. Further investigations into the reasons for the lack of positive samples at these sites could shed additional light on these findings.

Although most sites submitted samples, internal completeness of data was very low and below the targets for an effective surveillance system (100% completeness for the minimum data elements of unique identifier, district of residence, date of birth, date of onset of symptoms, ever received YF vaccine, date of sample receipt at the laboratory, tests done, date of result, final classification, and outcome) [ 20 ]. In particular, vaccination history was almost never filled out. This variable is critical to interpretation of YF testing results [ 30 ]; because receipt of the YF vaccine can cause a positive antibody result even in an uninfected person, positive test results among vaccinated persons tend to be discounted [ 21 , 31 ]. A lack of information on YF vaccination status can incur investigation costs and use resources unnecessarily. Beyond this, incomplete data can make follow-up investigations challenging if a patient cannot be located. Failure to value data quality and lack of feedback and supervision among health workers may contribute to incomplete data even when forms are reported as easy to complete [ 32 , 33 ]. Incomplete data gaps may remain unaddressed where there is limited feedback and data flow to sentinel sites and districts as identified in this assessment. We recommend regular data quality audits, mentorship and supportive supervision to improve data completeness data within the surveillance system [ 34 ].

We found sizeable time lag between sample collection and shipment to the laboratory which was higher than the expected WHO threshold [ 30 , 35 ]. Because of reported resource constraints, samples took at least 3 weeks to be shipped to the laboratory instead of the recommended < 24-hour timeframe. Similar findings have been reported in the Central African Republic, and other African YF endemic countries with challenges in transportation of laboratory samples [ 36 ]. Such delays may hinder the timely detection of YF outbreaks. Due to this long duration of confirming a suspected case, YF cases would have either healed or died by the time a case is confirmed, limiting the usefulness of the YF surveillance system. To address such delays the YF surveillance system could adopt the Uganda laboratory hub system, which was used to reduce turnaround times for HIV test results and transport costs by half improving early infant diagnosis [ 37 ]. Preliminary findings from a project improving yellow fever testing turn-around time at a Health Centre III indicated that using the public Uganda laboratory hub transport system reduced time between sample collection and shipment to the laboratory to < 7 days between May to June 2023.

While stakeholders reported that the system was useful in identifying individual cases for response, sentinel surveillance systems often are designed to monitor patterns. Despite this, data generated by the sentinel YF surveillance system were not routinely analysed. This was attributed to the lack of personnel assigned to data analysis roles. The lack of routine analysis of YF sentinel surveillance data limits its usefulness in monitoring disease trends and using these data for decision-making [ 35 ]. Specifically, this may result in missed opportunities in tracking patterns of YF transmission over time. Additionally, analysis of YF sentinel surveillance data is vital to achieving WHO’s goal to eliminate yellow fever epidemics by 2026 [ 38 ]. Enhanced efforts to carry out periodic analysis of the sentinel data could improve the usefulness of the system.

Limitations

Although the qualitative approach to evaluating system attributes elicited respondents’ in-depth perspectives, these could have been biased due to social desirability. The positive predictive value of the system could not be ascertained as only the gold standard measurement of YF diagnostics was used.

In conclusion, the yellow fever sentinel surveillance system had sentinel sites in areas with documented YF circulation in Uganda that facilitated detection of most YF outbreaks in Uganda. However, gaps in delays in case confirmation, incomplete data and inconsistent use of case definitions reduced its overall effectiveness and efficiency.

Data availability

The datasets upon which our findings are based belong to the Uganda Public Health Fellowship Program. For confidentiality reasons, the datasets are not publicly available. However, the datasets can be availed upon reasonable request from the corresponding author with permission from the Uganda Public Health Fellowship Program.

Abbreviations

Enzyme-Linked Immunosorbent Assay

Integrated Disease Surveillance and Response

Ministry of Health

Public Health Emergency Operations Centre

Plaque Reduction Neutralisation Test

Uganda Virus Research Institute

World Health Organisation

Monath TP, Vasconcelos PFC. Yellow fever. J Clin Virol [Internet]. 2015;64:160–73. Available from: https://www.sciencedirect.com/science/article/pii/S1386653214003692

WHO (World Health Organisation). Yellow fever: rapid field entomological assessment during yellow fever outbreaks in Africa: handbook: methodological field approaches for scientists with a basic background in entomology [Internet]. Geneva PP - Geneva: World Health Organization; 2014. Available from: https://apps.who.int/iris/handle/10665/112785

Nomhwange T, Jean Baptiste AE, Ezebilo O, Oteri J, Olajide L, Emelife K, et al. The resurgence of yellow fever outbreaks in Nigeria: a 2-year review 2017–2019. BMC Infect Dis [Internet]. 2021;21(1):1054. Available from: https://doi.org/10.1186/s12879-021-06727-y

Gardner CL, Ryman KD. Yellow fever: a reemerging threat. Clin Lab Med [Internet]. 2010 Mar;30(1):237–60. Available from: https://pubmed.ncbi.nlm.nih.gov/20513550

Wamala JF, Malimbo M, Okot CL, Atai-Omoruto AD, Tenywa E, Miller JR, et al. Epidemiological and laboratory characterization of a yellow fever outbreak in northern Uganda, October 2010-January 2011. Int J Infect Dis IJID Off Publ Int Soc Infect Dis. 2012 Jul;16(7):e536-42.

Chippaux J-P, Chippaux A. Yellow fever in Africa and the Americas: a historical and epidemiological perspective. J Venom Anim Toxins Incl Trop Dis [Internet]. 2018;24(1):20. Available from: https://doi.org/10.1186/s40409-018-0162-y

Nwaiwu AU, Musekiwa A, Tamuzi JL, Sambala EZ, Nyasulu PS. The incidence and mortality of yellow fever in Africa: a systematic review and meta-analysis. BMC Infect Dis [Internet]. 2021;21(1):1089. Available from: https://doi.org/10.1186/s12879-021-06728-x

WHO (World Health Organisation). Yellow Fever Uganda [Internet]. 2022 [cited 2022 Jul 25]. Available from: https://www.who.int/emergencies/disease-outbreak-news/item/2022-DON367#:~:text=Yellow fever vaccine has not,mass vaccination campaigns (PMVCs).

WHO (World Health Organisation). Yellow Fever- Uganda [Internet]. 2020 [cited 2023 Sep 20]. Available from: https://www.who.int/emergencies/disease-outbreak-news/item/2020-DON249

Kwagonza L, Masiira B, Kyobe-Bosa H, Kadobera D, Atuheire EB, Lubwama B, et al. Outbreak of yellow fever in central and southwestern Uganda, February–may 2016. BMC Infect Dis [Internet]. 2018;18(1):548. Available from: https://doi.org/10.1186/s12879-018-3440-y

Organization WH. A global strategy to eliminate yellow fever epidemics (EYE) 2017–2026. 2018;

MoH (Minstry of Health Uganda). National Technical Guidelines for Integrated Disease Surveillance and Response Third Edition [Internet]. 2021. Available from: https://www.afro.who.int/sites/default/files/2021-09/2_Uganda3rdIDSRTechGuideline_PrintVersion_10Sep2021.pdf

Lutwama JJ. Uganda Arboviruses Surveillance Activities. African J Pharmacol Ther. 2012;1(4).

Johansson MA, Vasconcelos PFC, Staples JE. The whole iceberg: estimating the incidence of yellow fever virus infection from the number of severe cases. Trans R Soc Trop Med Hyg [Internet]. 2014/06/30. 2014 Aug;108(8):482–7. Available from: https://pubmed.ncbi.nlm.nih.gov/24980556

WHO (World Health Organisation). WHO report on global surveillance of epidemic-prone infectious diseases [Internet]. World Health Organization; 2000. Available from: https://www.who.int/csr/resources/publications/surveillance/Yellow_fever.pdf

Ingelbeen B, Weregemere NA, Noel H, Tshapenda GP, Mossoko M, Nsio J, et al. Urban yellow fever outbreak-Democratic Republic of the Congo, 2016: Towards more rapid case detection. PLoS Negl Trop Dis [Internet]. 2018 Dec 7;12(12):e0007029–e0007029. Available from: https://pubmed.ncbi.nlm.nih.gov/30532188

WHO (World Health Organisation). Yellow Fever- Uganda [Internet]. 2022 [cited 2023 Feb 2]. Available from: https://www.who.int/emergencies/disease-outbreak-news/item/2022-DON367#:~:text=Uganda is endemic for yellow,Fever Epidemics (EYE) Strategy.

WHO (World Health Organisation). Yellow Fever Uganda [Internet]. 2022 [cited 2022 Aug 19]. Available from: https://www.who.int/emergencies/disease-outbreak-news/item/2022-DON367#:~:text=Uganda has not introduced the,and Koboko (2019)%2 C in

(US CDC) USC for DC and P. Updated guidelines for evaluating public health surveillance systems: recommendations from the guidelines working group. MMWR Recomm Rep. 2001;50(13):1–35.

WHO (World Health Organisation). WHO recommended surveillance standards. World Health Organization; 1999.

WHO (World Health Organisation). Yellow fever laboratory diagnostic testing in Africa: Interim guidance, 2016.

Sanou AS, Dirlikov E, Sondo KA, Kagoné TS, Yameogo I, Sow HE, et al. Building Laboratory-Based Arbovirus Sentinel Surveillance Capacity During an Ongoing Dengue Outbreak, Burkina Faso, 2017. Heal Secur [Internet]. 2018 Nov 27;16(S1):S-103-S-110. Available from: https://doi.org/10.1089/hs.2018.0048

Guillot C, Bouchard C, Aenishaenslin C, Berthiaume P, Milord F, Leighton PA. Criteria for selecting sentinel unit locations in a surveillance system for vector-borne disease: A decision tool. Front Public Heal [Internet]. 2022;10. Available from: https://www.frontiersin.org/articles/10.3389/fpubh.2022.1003949

Rugarabamu S, Mwanyika GO, Rumisha SF, Sindato C, Lim H-Y, Misinzo G, et al. Seroprevalence and associated risk factors of selected zoonotic viral hemorrhagic fevers in Tanzania. Int J Infect Dis [Internet]. 2021;109:174–81. Available from: https://www.sciencedirect.com/science/article/pii/S1201971221005610

Byaruhanga T, Kayiwa JT, Nankya AM, Ataliba IJ, McClure CP, Ball JK, et al. Arbovirus circulation, epidemiology and spatiotemporal distribution in Uganda. IJID Reg. 2023 Mar;6:171–6.

Kienle GS, Kiene H. Clinical judgement and the medical profession. J Eval Clin Pract. 2011 Aug;17(4):621–7.

Scott IA. Errors in clinical reasoning: causes and remedial strategies. BMj. 2009;338.

Baine SO, Kasangaki A. A scoping study on task shifting; the case of Uganda. BMC Health Serv Res. 2014 Apr;14:184.

Onyebujoh PC, Thirumala AK, Ndihokubwayo J-B. Integrating laboratory networks, surveillance systems and public health institutes in Africa. Afr J Lab Med. 2016;5(3):431.

WHO (World Health Organisation). Yellow fever laboratory diagnostic testing in Africa. World Health Organization; 2016.

Gibney KB, Edupuganti S, Panella AJ, Kosoy OI, Delorey MJ, Lanciotti RS, et al. Detection of anti-yellow fever virus immunoglobulin m antibodies at 3–4 years following yellow fever vaccination. Am J Trop Med Hyg. 2012 Dec;87(6):1112–5.

Tilahun B, Derseh L, Atinafu A, Mamuye A, Mariam TH, Mohammed M, et al. Level and contributing factors of health data quality and information use in two districts in Northwest Ethiopia: social-ecological perspective. BMC Med Inform Decis Mak [Internet]. 2021;21(1):373. Available from: https://doi.org/10.1186/s12911-021-01741-1

Tola K, Abebe H, Gebremariam Y, Jikamo B. Improving Completeness of Inpatient Medical Records in Menelik II Referral Hospital, Addis Ababa, Ethiopia. Prineas RJ, editor. Adv Public Heal [Internet]. 2017;2017:8389414. Available from: https://doi.org/10.1155/2017/8389414

Gimbel S, Mwanza M, Nisingizwe MP, Michel C, Hirschhorn L, Hingora A, et al. Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research (CFIR): results from the African Health Initiative. BMC Health Serv Res [Internet]. 2017;17(3):828. Available from: https://doi.org/10.1186/s12913-017-2660-y

Organization WH. Manual for the monitoring of yellow fever virus infection. World Health Organization; 2004.

Rachas A, Nakouné E, Bouscaillou J, Paireau J, Selekon B, Senekian D, et al. Timeliness of yellow fever surveillance, Central African Republic. Emerg Infect Dis. 2014 Jun;20(6):1004–8.

Kiyaga C, Sendagire H, Joseph E, McConnell I, Grosz J, Narayan V, et al. Uganda’s new national laboratory sample transport system: a successful model for improving access to diagnostic services for Early Infant HIV Diagnosis and other programs. PLoS One. 2013;8(11):e78609.

WHO WHO. Eliminate Yellow fever Epidemics (EYE): a global strategy, 2017–2026. Relev Epidemiol Hebd. 2017 Apr;92(16):193–204.

Download references

Acknowledgements

We would like to thank the UVRI Department of Arbovirology, Emerging and Re-Emerging Infectious Diseases for providing technical support and guidance provided during this evaluation. We appreciate the technical support provided by the Division of Surveillance, Information and Knowledge Management. The authors extend their appreciation to the management and staff of the sentinel and non-sentinel site health facilities who participated in this evaluation.

The project was supported by the President’s Emergency Plan for AIDS RELIEF (PEPFAR) through the United States Centres for Disease Control and Prevention Cooperative Agreement number GH001353-01 through Makerere University School of Public Health to the Uganda Public Health Fellowship Program, Ministry of Health. The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent the official views of the US Centres for Disease Control and Prevention, the Department of Health and Human Services, Makerere University School of Public Health, or the Ministry of Health.

Author information

Authors and affiliations.

Uganda Public Health Fellowship Program, Uganda National Institute of Public Health, Kampala, Uganda

Mercy Wendy Wanyana, Patrick King, Richard Migisha, Benon Kwesiga, Paul Edward Okello, Daniel Kadobera & Lilian Bulage

Ministry of Health, Uganda National Institute of Public Health, Kampala, Uganda

Joshua Kayiwa & Alex Riolexus Ario

Arbovirology Department, Uganda Virus Research Institute, Entebbe, Uganda

Annet Martha Nankya

US Centers for Disease Control and Prevention, Kampala, Uganda

Julie R. Harris

You can also search for this author in PubMed   Google Scholar

Contributions

MW and KP: Participated in the conception, design, analysis, and interpretation of the study and wrote the draft manuscript; JK and AMN participated in the acquisition, analysis and interpretation of data; RM and PEO reviewed the report, reviewed the drafts of the manuscript for intellectual content and made multiple edits to the draft manuscript; RM, DK, BK, LB, ARA, and JRH reviewed the manuscript to ensure intellectual content and scientific integrity. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mercy Wendy Wanyana .

Ethics declarations

Ethics approval and consent to participate.

This evaluation was done to inform public health practice and therefore determined as non-research. The Ministry of Health Public Health Emergency Operations Centre, the custodian of the YF surveillance data, granted administrative clearance for accessing and using the data in the YF database for this assessment. The Office of the Center for Global Health, US Center for Disease Control and Prevention determined that this activity was not human subject research and its primary intent was for public health practice or disease control. We sought administrative clearance from the sentinel site health facilities to use the data. We obtained verbal informed consent from interviewees including key stakeholders at the Uganda Virus Research Institute, Ministry of Health, sentinel site health facilities, and non-sentinel site health facilities facility staff before data collection using a consent form. Data held on computers were encrypted with a password which was made available to the evaluation team. This activity was reviewed by the CDC and was conducted consistent with applicable federal law and CDC policy.§ §See e.g., 45 C.F.R. part 46, 21 C.F.R. part 56; 42 U.S.C. § 241(d); 5 U.S.C. § 552a; 44 U.S.C. § 3501 et seq.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

The funders had no role in the study design, data collection, data analysis and decision to publish or preparation of the manuscript.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Wanyana, M.W., King, P., Migisha, R. et al. Evaluation of the sentinel yellow fever surveillance system in Uganda, 2017–2022: strengths and weaknesses. BMC Infect Dis 24 , 686 (2024). https://doi.org/10.1186/s12879-024-09580-x

Download citation

Received : 30 November 2023

Accepted : 01 July 2024

Published : 09 July 2024

DOI : https://doi.org/10.1186/s12879-024-09580-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Yellow fever
  • Surveillance system

BMC Infectious Diseases

ISSN: 1471-2334

critical evaluation of research article example

  • Open access
  • Published: 09 July 2024

Improving the non-structural preparedness of the selected hospital based on the FOCUS-PDCA 1 model: action research

  • Alireza Basiri 1 ,
  • Mohsen Abbasi Farajzadeh 1 , 2 ,
  • Mohammad Belal 1 , 2 &
  • Esmail Heidaranlu 3  

BMC Emergency Medicine volume  24 , Article number:  109 ( 2024 ) Cite this article

Metrics details

Introduction

With the intensification of the country’s development process, the expansion of cities and population, and the inclusion of Iran in the accident-prone category, reducing the vulnerability of non-structures has received more attention from the organizations involved. In addition to damage to communities and infrastructure, accidents can affect hospitals and their non-organizational components. Hospitals, as the front line of providing medical services after accidents, must maintain their stability, ensure the safety of their patients and employees, and continue to operate without interruption as in normal conditions. Therefore, it is necessary to evaluate the non-structural safety and their preparedness to ensure they can perform acceptable in critical conditions.

This applied research was conducted in 2023 (September to December) using the participatory action research method in all selected hospital departments. The level of non-structural preparedness of the hospital was checked using the valid “Hospital Safety Index” questionnaire and the non-structural weaknesses of the hospital were identified. Then, in action research using the FOCUS-PDCA model, a program was implemented to improve the non-structural preparedness of different departments of hospitals in the face of accidents and disasters. The non-structural readiness level of the hospital was compared before and after the implementation of the change.

Based on the evaluation conducted in the present study, the lowest level of safety was observed in the water supply system, office furniture and appliances, and fuel storage. The waste management systems, the fire protection system, and the long-distance communication systems were at a desirable performance level. Although in the evaluation before the change, the overall score of the hospital was 71.01%, and it had a desirable performance level in non-structural factors, in all the involved parts of the hospital, the sensitive, critical, and practical parts in the operation of the hospital had an average and sometimes low safety level. According to the obtained safety score, the safety level of the selected hospital before the change was 7 out of 10 (level seven of safety evaluation = medium). After the change and corrective measures, the non-structural safety assessment score was 76.93, and the hospital’s safety level was raised by one step to 8 out of 10 (8th level of safety assessment = relatively favorable).

The present study showed that the application of Total Quality Management (TQM), primarily its application tool FOCUS-PDCA, is efficient and helpful in improving the non-structural preparedness of hospitals. Using action research in the health field in accidents and disasters can open blind knots in different dimensions of preparedness (structural, non-structural, and functional).

Peer Review reports

In the last decade, the incidence of unexpected disasters has increased worldwide, with a death rate of about 106,000 people per year [ 1 ]. Almost all countries are exposed to unexpected disasters, including hurricanes, floods, earthquakes, fires, droughts, terrorist attacks, volcanic eruptions, chemical accidents, and diseases. Natural disasters can start quickly or slowly and have serious adverse effects on health and social levels and economic consequences [ 2 , 3 ]. After India, Bangladesh, and China, Iran ranks fourth in unexpected natural disasters [ 4 ] So, out of 40 natural disasters reported worldwide, 31 have occurred in Iran, and there is a possibility of their recurrence in the future [ 5 ].

If you pay attention to the disaster management cycle and the appropriate prevention and preparedness program, it can be a suitable response when disasters occur and thus reduce the death, injury, disability and burden caused by these disasters [ 3 , 6 , 7 ]. In case of unexpected accidents, hospitals are considered as the most important reception centers for accident victims and because they are among the first organizations involved in the consequences of casualties and injuries of these accidents [ 8 , 9 ] Hospitals’ preparation as an institution providing healthcare services is essential and critical in reducing deaths and physical injuries caused by accidents, crises, and emergencies [ 10 ]. Therefore, it is necessary to determine the level of safety and performance of the hospital against disasters and emergencies [ 11 ].

The hospital’s evaluation provides the basis for identifying the components of improving safety and prioritizing interventions according to their type and location to reduce mortality, morbidity, disability, and other social and economic costs [ 12 ]. Vulnerability intervention studies usually include in-depth analysis of structural and non-structural risks, health system, and hospital vulnerability. Each of these aspects requires experts with experience in disaster risk reduction. Such investigations typically take several months and may be costly to the hospital [ 13 ]. In this regard, the hospital’s safety index is an essential tool to move toward the hospital’s goal, which is to improve the provision of health services to people. These goals include less vulnerability, better preparedness, and safer against disasters and emergencies. In hospitals, whose residents include patients, nurses, doctors, administrative and service personnel, and clients, they should also be responsible for the role of relief when accidents and disasters occur, for this reason, ensuring the stability and efficiency of non-structural components of architecture and facilities. It is necessary [ 12 ]. The hospital is made up of many components to provide services to patients, and the relationships between these components must have the necessary coherence so that this institution can perform its duties well Because a defect in any of the components leads to a problem in the process of providing services [ 14 ]. One of these components that should be considered is non-structural components. Non-structural components are generally non-structural factors, including architectural, mechanical, telecommunication, electrical, and medical equipment. Non-load-bearing walls, panels, load-bearing walls, suspended ceilings, windows, heating-cooling and ventilation equipment, steam boilers, elevators, emergency power generators, liquid storage tanks, telecommunication devices, and medical equipment and tools are all non-structural elements of the hospital [ 15 , 16 ].

Architectural components and electrical and mechanical installations in each building include the major part of the construction costs of the entire building. In hospitals, the share of electrical and mechanical facilities is more than in conventional residential, administrative, and commercial buildings. Suppose the cost of providing medical equipment and devices is also considered. In that case, it can be seen that the share of non-structural components of the building in the total investment of a hospital is significant [ 17 , 18 ].

Past studies show that only 15% of hospital construction costs are allocated to structural elements [ 19 ].

85% of all costs are related to architectural, mechanical, electrical, and storage components. Also, hospital equipment and special treatment devices can be added to the previous costs [ 20 ].

Comprehensive quality management is among the effective and widely used tools in the field of health that have been used in recent years at the national and even international levels to improve organizational processes. Its important tool is FOCUS-PDCA.

The FOCUS-PDCA tool consists of nine steps, symbolized by the following: (F) Find a problem to improve. Organization (O); turn on (C); Understanding (U); selection (S); plan (P); do (D); Review (C); And, rule (A) - condensed in the acronym, FOCUS-PCDA. The FOCUS-PCDA quality improvement model has been widely used in many fields [ 21 ]. is a scientific, coherent, and practical method to improve processes and has a complete and relevant management toolbox to solve organizational challenges with the help of process owners [ 22 ].

This study aims to measure the quantitative and qualitative performance of equipment and the vulnerability of non-structural components of selected hospitals in the city of Tehran as a survey, with a special focus on non-governmental hospitals and improving performance with the help of process owners.

This applied research was conducted in 2023 (September to December) using the participatory action research method in all selected hospital departments with the participation of the owners of the hospital’s exposure to accidents and disasters in order to improve the hospital’s vulnerability from the perspective of personnel preparation. A statistical sample is a hospital. Field investigations based on the hospital safety assessment tool against disasters, the Persian version of FHSI, include 145 items in 3 general groups: structural safety, non-structural safety, and functional capacity. A statistical sample is a hospital. Field investigations based on the hospital safety assessment tool against disasters, the Persian version of FHSI, include 145 items in 3 general groups: structural safety, non-structural safety, and functional capacity [ 23 ]. The data is ranked according to the low, medium and high safety level indicated by the numbers 0, 1 and 2, and the total obtained between 0.0 and 1.0 is classified into 3 levels A, B and C. Be that shows [ 24 ].

In this study, non-structural safety, which includes 93 items in the selected hospital, has been investigated. The hospital safety index is a tool designed to evaluate the safety of hospitals and their critical role in responding to disasters and emergencies, and it represents a high quality of care.

This research is of semi-experimental and interventional (before and after). In this study, non-structural safety, which includes 93 items in the selected hospital, has been investigated. The hospital safety index is a tool designed to evaluate the safety of hospitals and their critical role in responding to disasters and emergencies, and it represents a high quality of care Evaluation team with expert evaluators with experience in hospital construction, providing health services, management, or hospital support activities All specialists involved in the process had the necessary training regarding the objectives and methodology of hospital safety assessment, how to complete the checklist, interpret the results, and prepare the final assessment report Evaluation team with expert evaluators with experience in hospital construction, providing health services, management, or hospital support activities All specialists involved in the process had the necessary training regarding the objectives and methodology of hospital safety assessment, how to complete the checklist, interpret the results, and prepare the final assessment report Organization of the evaluation team Once the desired hospital is selected, the evaluation team includes a hospital manager, treatment manager, medical assistant services manager, health manager, technical and engineering manager, information and communication technology manager, support manager, safety manager, quality improvement manager, manager Nursing, medical equipment manager, structural safety expert, accident and disaster risk management committee secretary, structural safety expert and occupational health expert were formed The characteristics of the hospital and its conditions were investigated [ 25 ]. After reviewing the evaluators and the obtained results, continuous meetings were held with the process owners based on the FOCUS-PDCA model, and corrective measures were formulated and implemented considering the three critical indicators of less cost, less time, and high execution capability After the change was made, re-evaluation was done by evaluators regarding the non-structural safety of the hospital (Fig.  1 ).

figure 1

Flowchart of the work execution method

Determining the validity and reliability of the tool

The hospital safety assessment instrument questionnaire by Ardalan et al. has been localized and the Farsi version is called Hospital Safety Assessment Instrument (FHSI). This questionnaire contains 145 items in 3 dimensions of structural, non-structural and functional safety, which have been confirmed in previous studies [ 23 ]. This tool is used to evaluate the hospital in three safety levels: weak, moderate and favorable. High safety score or category A (66 − 100%) Medium safety score or category B (36 − 65%) Low safety score or category C (0 − 35%). And based on the three-choice Likert scale according to the Farsi Hospital Safety Index, it was done [ 26 ].

Executive method of the FOCUS-PDCA model

According to the points and standards related to non-structural safety, separately in 12 dimensions of the tool in the items that were at the average, weak, and very weak level (preparation percentage less than 70), FOCUS-PDCA model training in the form of holding a conference, Justification classes and use of visual and audio resources were taught for process owners in 8 two-hour sessions in the conference hall of the hospital. Then, for the continuity and durability of the educational materials, a training booklet designed by the researcher, which has been approved by expert professors in the field of health and disasters and includes general information on non-structural safety and validation items and points that can be modified in the desired items, was distributed among the managers and responsible for the parts involved in the distribution process. Also, three virtual groups were formed on WhatsApp according to the level of the participants to answer their questions and doubts. The reports were followed up through cyberspace. Again, during four weeks, the researcher, together with the hospital risk committee team, visited all the parts of the hospital that were involved in non-structural safety and reminded them of the contents according to their expertise, then with the cooperation of the personnel and officials of each part with Paying attention to their expertise in the desired areas by implementing the FOCUS-PDCA model, a practical solution for correcting the existing weaknesses was implemented by the risk management committee with the help of the process owners of each part, focusing on the non-structural preparedness standard guidelines of the hospital. According to the action research, the leading solutions were focused on the three principles of low cost, low time, and high execution capability. The re-evaluation of the hospital’s non-structural preparedness was done by the researcher and with the help of the managers of the hospital’s risk committee after one month of completing the training. Moreover, finally, the results before and after the change were compared and reported.

“Hospital Disaster Risk Assessment” tool, non-structural vulnerability in 12 distinct sectors including architectural safety (16 indicators), infrastructure protection, access and physical security (4 indicators), critical systems/electrical systems (10 indicators), long - communication systems range (8 indicators), water supply system (6 indicators), fire protection system (5 indicators), waste management systems (5 indicators), fuel storage systems (such as gas, diesel and diesel) (5 indicators), Heating, ventilation and air conditioning system (HVAC) (8 indicators), office and storage furniture (fixed and mobile) (2 indicators) and medical supplies and equipment used for diagnosis and treatment (19 indicators) (Tables  1 and 2 ).

Non-structural safety results in three levels of safety equipment: Low safety: no or zero safety. Moderate Safety: Safety precautions are followed to some extent. And safety, health and safety are very valuable.

This tool evaluates the hospital in three safety levels: weak, moderate and favorable. After analyzing the results in parts with unacceptable safety (percentage less than 70%), some prioritized actions for the lack of safety based on the FOCUS-PDCA (action research) model were reviewed and implemented with the participation of process owners.

Based on the assessment, the lowest level of safety was observed in the water supply system, office furniture and appliances, and fuel storage. In waste management systems, fire protection systems, and long-distance communication systems, they had a desirable performance level (Fig.  2 ). Although in the pre-intervention evaluation, the overall score of the studied hospital was 71.01% (Table  3 ), They had a desirable performance level in non-structural factors; In all hospitals, the sensitive, critical and practical departments in the operation of the hospital had an average and sometimes low safety level. According to the obtained safety score, the safety level of the selected hospital before the intervention was 7 out of 10 (level seven of safety evaluation = medium) (Table  4 ). After the intervention was carried out. Measures were taken based on the evaluation; the lowest level of safety was observed in the heating and cooling system and medical equipment. In waste management systems and long-distance communication systems, they had a more desirable performance level (Fig.  3 ). Comparison of improvement of safety level of non-structural components before and after change in twelve dimensions. The results showed that the research action and the formation of improvement teams of process owners have played an effective role in achieving non-structural preparation (Fig.  4 ).

In the evaluation after the intervention, the non-structural safety evaluation score was improved to 76.93 (Table  5 ) and the hospital’s safety level was increased by one step to 8 out of 10 (8th level of safety evaluation = relatively favorable) (Table  6 ). After the implementation of the prioritized corrective measures based on the components, lower cost, less time and more implementation capability (Table  7 ), the highest level of improvement in medical gas system, office equipment and warehouses 25%, water supply system 17% respectively heating and cooling system 6.75%, architectural safety 6.67%, medical equipment 6.12% and electrical systems 5% (Table  8 ).

figure 2

The spider web diagram of non-structural component scores before the intervention

figure 3

The spider web diagram of non-structural component scores after the intervention

figure 4

Comparison of improving the safety level of non-structural components before and after the change

In the field of national crisis management, the United States Department of National Security considers disaster preparedness as one of the stages of crisis management, which includes data collection, research, planning, organizational development, provision of resources, training, obtaining documents and certificates, equipment, and records management [ 27 ]. Also, WHO has identified the lack of training for preparedness against unexpected events and disasters at the state level and ordinary people as one of the main reasons for the high losses caused by natural disasters [ 28 ]. In the studied hospital, the condition of non-structural factors was average. The findings of this study showed that the overall average non-structural safety of the studied hospital after the implementation of the FOCUS-PDCA model and the participation of process owners in each area in multiple group discussion sessions was reported to be 76.93%, which is evaluated as (good) according to WHO standards. The study by Dargahi A. et al. (2017) showed that the average non-structural vulnerability to disasters is 41.5%, and the average preparedness of healthcare centers in Coping with disasters was 18.3% [ 13 ]. In addition, Arab M et al. (2009), in a study titled A study concluded from their study in Tehran hospitals, that The operational preparedness of hospitals in response to disasters is 51%. Both studies recorded a lower percentage of preparedness than the present study [ 29 ]. However, Ojaghi S et al. (2009), in a study found that the preparation of teaching hospitals to deal with disasters could be more robust [ 30 ]. Based on the study by Al Khalaileh MA et al. (2012), 65% of nurses stated that hospitals’ preparedness level to deal with disasters could be much higher. The study showed that Jordanian hospitals were poorly prepared against disasters [ 31 ]. In the study of Hatami H and colleagues (2017) entitled Functional, structural, and Non-structural Preparedness of Ahvaz Health Centers against Disasters, they obtained a safety index of 4 out of 10 and a non-structural safety of 54.82% [ 15 ]. In the study of Sabzghabaie A. et al. (2013) titled Hospital Safety in Hospitals Affiliated with Shahid Beheshti University, 49.44% of disaster preparedness was evaluated for Shahid Beheshti Hospital, which did not match the results of our study [ 32 ]. In a study conducted by Arab M and colleagues (2009), their hospital’s preparedness against earthquake events and their relationships at public hospitals showed that in 47% of the studied hospitals in the area, The safety of dangerous equipment and materials against accidents was at the medium level of preparedness and 14% were at the low level of preparedness [ 29 ]. In addition, in the study of Jagnarine S, which was conducted on 45 hospitals in the Caribbean, under the title Safe hospitals in the Caribbean, only 2% of hospitals had relatively good safety, 80% had moderate safety, and 18% had low safety, which The results of our study were different [ 33 ]. The difference in the preparedness of the non-structural components of hospitals is due to several factors, including the difference in the research environment, data collection method, oral interview, written questionnaire or observation, when collecting data, training, and expertise of the people who collected the data, the type of checklists used. Moreover, relatively long-term studies are concerned.

In the Abbasabadi-Arab et al. (2023) study done in 604 hospitals, the average non-structural score was 65.24, and functional safety was 63.36. The provinces of Hamedan (76.81 and Kerman) had the highest score, and the provinces of Yazd (53.74) and Lorestan (57.31) had the lowest score [ 34 ]. The study by Asadi G. and colleagues (2022) titled Assessment of Safety Status and Functional, structural and Non-structural Preparedness of Health Centers in Hamadan against Disasters was done. The results of preparation level and non-structural safety level are 63.96%. The level of non-structural safety is at an acceptable level that can be improved [ 3 ]. Based on Takzadeh H and colleagues’ (2015) study on the safety Assessment of urban and rural health service centers. Isfahan health homes and health centers in barriers and emergencies, the average scores of non-structural factors in urban and rural centers were 74.2 and 82.5, respectively [ 35 ].

In the study of Tabrizi et al. in East Azarbaijan, the safety rate of non-comedogenic drugs was reported to be 61.75% [ 36 ]. The study conducted by Amerion A and colleagues (2010) showed that the preparedness of hospitals to deal with disasters is more than 70% [ 37 ]. Mahboobi et al.‘s study (2008) reported that the average preparedness of three hospitals and healthcare centers in Kermanshah regarding equipment and facilities was 74.6%, compared to The present study, which is almost equal [ 38 ]. In a study conducted by Asefzadeh S et al. (2016). The preparedness of the studied hospitals in dealing with disasters with an average non-structural safety of 67.71% (safety average) in two hospitals [ 39 ]. In a study by Cliff BJ et al. in a rural hospital in the United States, the preparedness of these hospitals was evaluated at 78% [ 40 , 41 ]. In the study by Zhong S et al. (2014), the preparedness of hospitals was 81% [ 42 ]. The results of the present study are consistent with other studies that examine the level of preparedness of hospitals in Tehran against disasters, and it confirms that the non-structural performance of hospitals in accidents and disasters is at an average to a high level.

However, it should be noted that according to the weak chain theory, the non-structural elements of a hospital are like the links of a chain, and a weakness in any of them may cause the whole complex to break. Therefore, the average values of the safety level of the set of non-structural factors of a hospital cannot be a suitable indicator for the level of vulnerability of a hospital. Therefore, investigating each non-structural factor and predicting solutions to improve the functional level and implement their corrective measures will be closer to reality. The results of our study showed that the non-structural safety in the hospital under our study was 70.01% before and after the intervention. They were increased to 76.93%, which was a good-high level.

Architectural safety

The hospital’s non-structural safety percentage in architecture before the change and implementation of the FOCUS-PDCA model by the process owners was 63.33%. These components do not have a direct effect on the operation of the hospital. However, their destruction during accidents can seriously disrupt the efficiency of different hospital departments. False ceilings, facades of internal walls and columns, plasterwork, and non-load-bearing separating blades are highly vulnerable if they are not well connected to the main structure of the building. After implementing the FOCUS-PDCA model and applying corrective measures in the areas of the safety of the surrounding walls and the safety of movement inside the building (such as corridors and staircases), such as installing guards and fences on the walls, roofs, installing stair railings, installing stair railings, etc., the level score Architectural safety was increased to 70%, in a study by Luke J. et al. (2023) the preparedness and evaluation of 6 tertiary and rural hospitals in Australia found that the architecture’s safety components are very resistant to disasters, and each recorded high HSI scores. Differences in structure, architectural safety, and continuity of supply of critical services have been identified. The safety of the architectural components of the studied hospitals has a score of 79% [ 43 ]. In a study conducted by Lestri F et al. (2022), the safety score of architectural components in the studied hospitals located in four provinces of Indonesia, namely in Jakarta, Yogyakarta, North Sumatra, and West Java, the percentage is between 36% and 65% [ 44 ]. In a study conducted by Asefzadeh S et al. (2016), the safety score of architectural components in Velayat Hospital was 88.88%, and Rajaee Hospital was 69.44% [ 39 ]. In the study by Sadeghi et al. (2015), suitable Solutions and Missions in These Situations in the Second Semester of 2015 were conducted regarding the safety level of the architectural components of Baath Hospital. 100%, family hospital 58% and Ahvaz 578 hospital 41% had safety [ 45 ], which is different from the results obtained in the present study, which shows that the level of activity in order to obtain the necessary safety according to the standard in these centers is different. have been.

Infrastructure protection, access and of physical security

88% of the studied hospitals had good Infrastructure protection, access, and physical security resources in the emergency exit system. The emergency exit routes, roads, and stairs needed to be marked on most hospital floors. In this regard, it is recommended to take some measures to correctly mark emergency exits, train staff to guide people and move supplies and equipment that make the use of emergency exits difficult. In the study by Yang et al. (2023) regarding assessing the resilience of critical infrastructures in France, which has been done in two ways, the safety level of infrastructure protection and the availability of physical resources at an acceptable level existed [ 46 , 47 ]. In the study of Suparni, the current safety levels of the hospital are such that patients, hospital staff, and the hospital’s ability to function during and after an incident are potentially at risk [ 48 ]. In the study of Amerion A et al. (2010) regarding the safety level of infrastructure protection and access to physical resources in the emergency exit system, a score was obtained (72-100%), which is close to the score obtained in the present study [ 37 ]. In Hatami H et al.‘s study (2017), evacuation or emergency exit has the lowest score, with 33.3% [ 15 ]. preparedness for confronting crises such as floods, earthquakes, fires, and storms in some selected hospitals in Iran. Iranian Journal of Military Medicine preparedness rate (48.8%) has been obtained [ 49 ], and the study of Hojat M et al. (2012) reported the preparedness rate (39.63%) And the study of Kavari SH and colleagues (2006) [ 50 ]. to Shiraz University of Medical Sciences. Health Information Management analyzed the preparation level of this department, which is at 50%, which is low [ 51 ]. In the study of Zaboli R and colleagues (201), the level of preparedness (31.6%) was reported [ 52 ]. In the study of Hosseini Shokouh S and colleagues (2008), the level of preparedness in infrastructure is reported as 33.3% [ 53 ]. In the study of Mohammadi Yegane S et al., the level of preparedness regarding emergency exits is reported as 14% [ 54 ] and the review of the conducted studies is in line with the present study.

Critical systems/electrical systems

In the investigation of the hospital under study, the safety of the hospital in critical systems/electrical systems was reported as 80%. In order to improve the safety level of the electrical sector, it is recommended that the main cables located in the channels are installed in such a way that their standard and safety distance from each other is observed so that in case of an accident in one of the cables, the other cables will not suffer an accident. It was also recommended to consider an emergency lighting system using rechargeable generators (batteries) for power outages so that emergency evacuation is not associated with casualties. In addition, an uninterrupted power supply (UPS) should be provided for critical parts of the hospital during an earthquake. Considering that in this area, the promotion team was not formed due to a score above 70%, but the ongoing and routine actions of the hospital’s engineering department in the areas of critical systems/systems for maintaining the standard distance of cables from each other, as well as the placement of cables in safe channels and ducts, practically caused in the following evaluation, the safety level was increased to 85%. In the Tal MA (2021) et al. study, acceptable accuracy was obtained in assessing the current situation and predicting the future situation [ 55 ]. In the study of Zegarra RO (2023), prehospital and disaster medicine was conducted in 18 government hospitals. The average safety score of the critical systems of the studied hospitals is 63% [ 56 ]. In the study of Kjølle et al. (2012). Reliability Engineering and System Safety have been done. Regarding the risk analysis of critical infrastructures emphasizing electricity supply and its interdependencies in the hospitals of Oslo, Norway, a severe power cut can have cascading effects and consequences for other infrastructures [ 57 ]. In a study by Asefzadeh et al. (2016), the safety score of electrical systems in Velayat Hospital is 81.25%, and Rajaei Hospital is 43.75% [ 39 ]. In the study by Mohammadi Yegane et al. on the safety of electrical systems, Golestan Hospital, with 100%, and Family Hospital, with 43%, had the lowest percentage of safety in electrical systems [ 54 ]. The scores obtained in the studies conducted in Tehran’s Velayat and Shariati hospitals are almost equal.

Telecommunication systems

The hospital obtained 93.75% non-structural safety in telecommunication systems in the current study. It must be accepted that admitting the wounded is highly dependent on establishing external communications and coordinating within the hospital; there is an urgent need to establish internal communications. In this regard, strengthening and restraining equipment and cables in the hospital’s communication system, creating multiple layers of secure communication, and using wireless technology can guarantee communication during incidents. The study by Bisanda MP et al. (2023) showed that the resilience of mobile and sustainable telecommunications communication systems in disaster conditions is critical and important [ 58 ]. It was done in the study of Lamine H et al. (2022). The evaluation of the level of preparedness and response, including nine hospitals of the University of Tunis in disasters, showed that 7 of the nine university hospitals had safety category “B” in communication systems with overall safety indicators between 37% and 62% [ 59 ]. The study by Zaboli R et al. (2014) conducted a study on communication and information systems related to emergency management, which stated that the personal information of personnel is the most important and critical element for any system, especially crisis management. In a disaster, key personnel information must be available for quick response. In this study, the preparedness of communication and early warning was reported as 46.2%, which was lower than the average [ 52 ].

On the other hand, in the study of Amerion A et al. (2010), the percentage of preparedness in this area was reported as 66% [ 29 ]. In a study conducted by Asefzadeh S and colleagues (2016), the safety score of communication systems in Velayat Hospital and Rajaee Hospital was 92.85% and 42.85% [ 37 ]. In the study of Sadeghi M et al. (2015), hospital 501, at 81%, and Hospital 578 in Ahvaz, at 26%, had the highest and lowest percentage of safety in communication systems, respectively [ 45 ].

Water supply systems

The studied hospital obtained a 50% safety level in water supply systems. Failure to supply drinking water required in emergencies for at least 72 h, considering the number of approved beds in the hospital and its occupancy rate, can cause the hospital to face a severe challenge at the time of response and water cutoff. Although most hospitals are usually equipped with an emergency water supply system, these systems are primarily designed for normal conditions and cannot respond in emergencies. After the implementation of the FOCUS-PDCA model by the process owners and the implementation of corrective measures in the field of the water supply system, such as locking the entrance doors of tankers and no access to the place of storage of tankers and the use of supplementary and backup pumps, the safety level score was increased to 66.66%. In the study of J EL-Matury H et al., the safety level of this system has been evaluated as “B,” which shows their ability to function during and after disasters.

Moreover, disasters are potentially at risk [ 60 ]. In a study by Luke J et al., twenty-two articles were reviewed using hospital resilience checklists and assessing non-structural components and water supply systems against natural disasters in ten countries. It shows that the average Amagi score of water systems in the reviewed studies is around 46%, which requires immediate intervention and the removal of relevant deficiencies [ 61 ]. In the study of Asadi G. et al. (2022), the lowest level of preparedness in the non-structural field in Hamedan health centers is reported in water supply systems at 38.78% [ 4 ]. In the study of Salari H and colleagues (2009), the most preparedness related to the water supply system was stated with 85% of the preparedness rate [ 28 ]. In a study conducted by Asefzadeh S. et al. (2016), the safety score of water supply systems in Velayat Hospital is 100%, and Rajaei Hospital is 70% [ 39 ]. In the study of Sadeghi M et al. (2015), especially in the water supply systems of Golestan Hospital with 84% and Family Hospital with 24%, they had the lowest percentage of safety in water supply systems [ 45 ].

Fire protection system

The studied hospital achieved a 90% safety level in fire protection systems. The safety status of this area was satisfactory in all parts of the hospital under study. Most of the hospitals had fire alarm systems and manual fire extinguishers. However, after an earthquake, it may not be possible for the fire department to be present at the hospital in time, in which case the fire may spread and spread. One of the measures that should be considered is using automatic fire extinguishing systems in areas such as warehouses, parking lots, and areas where automatic fire extinguishing does not harm the existing equipment and employees. In the study by Irwanto et al. (2021) conducted in a hospital in Andozi, the safety level of the fire protection system was reported to be 83% [ 62 ]. In the study of F Lestari et al. (2022), hospital safety index guidelines include four parameters: types of risks, structural safety, non-structural safety, disasters, and emergency management. The overall safety index is 67% and at level A, which means the hospital will likely maintain its performance in emergencies and disasters. Based on the evaluation, the fire protection system’s average safety score in DKI Jakarta’s provinces is 76%, Yogyakarta is 70%, and West Java is 66% [ 63 ]. In the study of Asadi G et al. (2022fire extinguishing systems are reported with 74.37% [ 4 ]. The study by Hatami H and colleagues titled Functional, structural, and Non-structural Preparedness of Ahvaz Health Centers against Disasters was done. Although most of the studied health centers had a sufficient fire safety level of about 72.45%, they did not have any information about fire detection and control systems. Despite the presence of fire extinguishers in most centers, they lacked a fire alarm system [ 19 ]. In the study of Mohammadi et al. (2011), fire control preparedness was reported to be 57% [ 54 ].

Waste management systems

In the field of waste management systems, the studied hospital obtained a 100% safety level. In the systematic waste management of healthcare centers, it is practically considered to implement a fixed and continuous program to remove the storage and collect production waste inside and transport it outside the sources of production of such waste. Therefore, hospital waste collection and transportation management are placed in two separate departments. Collection and transportation inside the hospital and collection and transportation outside the hospital. All hospitals have places for temporary storage of hospital waste, and some are equipped with a waste incinerator system.

However, it should be noted that in emergencies, in addition to the volume of patients to medical centers and, as a result, the production of hospital waste will increase, the planned structures for separation, burning, or transfer of waste outside the hospital will also be disrupted. In Blanchette RA et al.‘s (2023) study, in the field of waste management systems, the safety score is reported between 33% and 97% [ 64 ]. In Awodele O et al.‘s (2016) study, all facilities have identical waste management processes, including segregation, on-site collection/transportation, on-site storage, and off-site transportation. Medical waste collection staff mainly use gloves as personal protective equipment. Intervention programs helped ensure compliance and safety of processes. Only Hospital B provided on-site treatment of its waste (sharps only) with an incinerator [ 65 , 66 ].

Fuel storage system

In the fuel storage systems (such as gas, diesel, and diesel), the studied hospital achieved a 70% safety level. Considering that the hospital’s score in this area was good, corrective measures were suggested, and the process improvement was not accepted. Suggested corrective measures in this area include a change in the place of storage of stored fuel, as well as the way of storing fuel and installing flexible connections for pipes in the place of connection to the tanks, which can help in improving the safety of the fuel storage system. In the study of Muhammad-Idris et al. (2022), epidemics, riots, fires, and gas explosions, among other things. Serious effects have been reported in 65 to 85% [ 67 ]. In the study of Vichova K et al. (2019), an assessment was made to provide emergency fuel sources to hospitals. The evaluated hospitals are divided into types of faculty hospitals, regional hospitals, city hospitals, and private hospitals. Zlín Hospital is a type of private hospital that was evaluated in the first stage, and the level of fuel storage system preparedness in this hospital is 57.14%; the second stage is the evaluated hospital from the Czech region, which is a regional hospital and the level of preparedness is 33.34% in the field of energy supply. An emergency has been reported. In the third stage, the hospital in the South Moravia region, which is a teaching hospital, was investigated, and the level of preparedness of this hospital was 94.45% in the field of fuel systems in the region. In the fourth stage, a hospital from the Hradec Králové region, which is a type of region, is evaluated. It has 71.43% preparedness in the field of emergency fuel supply.

Finally, the hospital from the Ústí nad Labem region, which is a type of urban hospital, was rejected, and this hospital received 64.29% of the preparedness score [ 68 ]. In a study conducted by Asefzadeh S et al. (2016), the safety score of fuel storage systems in Velayat Hospital is 87.50% and Rajaei Hospital is 50% [ 39 ]. In the study of Sadeghi M et al. (2015), especially in fuel storage systems, Golestan and Family Hospital had the highest safety percentage with 90%, and Ahvaz 578 Hospital had the lowest safety percentage with 49% [ 45 ]. Regarding the preparation of non-structural components in the safety of heating, cooling, and air conditioning systems, 84% of the reviewed studies had the highest and 45% the lowest percentage of safety in the medical gas system [ 69 ]. In a study conducted by Asefzadeh S (2016) et al., the safety score of cooling, heating, and air conditioning systems in Velayat Hospital was 85.71% and 42.85% in Rajaei Hospital [ 39 ]. In Sadeghi M et al. (2015) study on the safety of heating, cooling, and air conditioning systems, 501 hospitals had the highest safety percentage at 77%, and family hospitals had the lowest safety percentage at 20% [ 45 ].

Medical gas system

In the field of medical gas systems, the studied hospital obtained a 50% safety level. The fall of the equipment and facilities of the adjacent building on the medical gas building can cause irreparable critical and material damages. In some hospitals, a central oxygen system is likely to lose its efficiency during an earthquake. Therefore, portable medical gas systems should be considered in all departments for emergencies. After the implementation of the FOCUS-PDCA model by the process owners and the implementation of corrective measures in the field of safety of the medical gas system and the relocation of the place of production and storage of medical gases and oxygen generators and the use of alternative sources for medical gases as well as the use of valves equipped with seismic cut-off for the oxygen system The center that will prevent gas wastage and the consequences of its release during the crisis, the safety level score in this area was raised to 75%. In Goniewicz et al.‘s (2023) study, the findings have been interpreted cautiously, and the most preparedness and performance related to the index of medical gases has been reported [ 70 ]. A study by Raju R et al. (2019) in India, this study concluded that determining an additional area for storing cylinders, adding another oxygen storage tank, and accuracy A lot in planning, design, and operation can increase the level of preparedness for medical gases in disasters [ 71 ]. The study by Gómez-Chaparro et al. (2018), consisting of 12 Spanish hospitals with land area and number of beds between 2314 and 23,300 square meters and 20 and 194 square meters, reports that the average annual consumption rates of medical gases by area, number of beds, number of inpatients, outpatients; The number of endoscopies, the number of emergency interventions, the number of hospital surgeries on another note, and the number of hospital discharges should be considered as suitable variables for quantification, measurement and evaluation. In evaluating the safety level of medical gas systems in the studied hospital, an average preparedness score of 68.35 [ 72 ]. was reported. In a study by Asefzadeh et al., the safety score of the medical gas system in Velayat Hospital was 57.14%, and Rajaee Hospital was 50% [ 39 ]. In Sadeghi et al. (2015) study, Golestan Hospital had the highest safety percentage at 89%, and Ahvaz 578 Hospital, with 20%, had the lowest safety percentage of medical gas system [ 45 ].

Heating, ventilation and air conditioning (HVAC) systems

In the field of Heating, ventilation, and air conditioning (HVAC) systems in particular departments, the studied hospital obtained a 56.25% safety level. These systems are made up of many different components, which are different in each hospital, and there is a possibility of damage and disturbance in each of them, even in normal conditions. One of the sectors that can be expected to be damaged during natural disasters is the heating-cooling and air-conditioning systems in particular sectors. All the equipment of these systems must be braced, and in the place of connecting the pipes and cables to the equipment, flexible joints that can withstand seismic conditions should be used. After the implementation of the FOCUS-PDCA model by the process owners and the implementation of corrective measures in this area, including measures such as bracing of pipe connections, bracing of cables connected to equipment, and the use of flexible connections, the safety level score was increased to 63%. Eight articles were examined in the study by Chair SY et al. (2023) in China under the title of heating, cooling, and air conditioning systems in hospitals. The findings of this review show the range, similarities, and differences in the performance of HVAC systems in different countries. In the early stages of the outbreak of COVID-19, various researchers reported that the main route of transmission of COVID-19 is through respiratory droplets. Recently, scientific evidence suggests that it is transmitted through the air. COVID-19 is highly transmissible in poorly ventilated and closed environments, and heating, cooling, and air conditioning systems have not performed well in the studied studies [ 73 ]. In the study of Moradi SM et al. (2021), 24 studies have been examined. The essential findings of this study were categorized into five main categories: risk analysis method, type of disaster, hospital safety methods, hospital components, and key outcomes of risk analysis and hospital safety assessment.

Regarding the preparation of non-structural components in the safety of heating, cooling, and air conditioning systems, 84% of the reviewed studies had the highest and 45% the lowest percentage of safety in the medical gas system [ 69 ]. In a study conducted by Asefzadeh S (2016) et al., the safety score of cooling, heating, and air conditioning systems in Velayat Hospital was 85.71% and 42.85% in Rajaei Hospital [ 39 ]. In Sadeghi M et al. (2015) study on the safety of heating, cooling, and air conditioning systems, 501 hospitals had the highest safety percentage at 77%, and family hospitals had the lowest safety percentage at 20% [ 45 ].

Ofce and storeroom furnishings and equipment (fxed and movable)

In Office and storeroom furnishings and equipment (fixed and movable), the studied hospital obtained a 50% safety level. It is necessary to use various fixing methods in the equipment and furniture department, including installing braces, screw packages, locking closets, and strapping cabinets and computer equipment. After the implementation of the FOCUS-PDCA model by the owners of the process and the implementation of corrective measures in this area, including measures such as: installing braces, using screw fasteners, locking closets, strapping cages, and computer equipment, the safety level score was increased to 75%. In terms of equipment, all the studied office centers, such as shelves, computers, and office furniture, needed to be in better condition. In the event of an accident, they could be turned off completely. The shipping investigation process is done in the study of Ratwani RM et al. (2023) in the state of Pennsylvania, USA. Out of 450 reviewed reports, the most frequent was in office supplies and furniture and storage (fixed and movable). The safety problem related to the types of fixing methods, including the installation of tie-downs, screw packages 33.34%, strapping of cages and computer equipment 28.2%, and locking 28.2% have been reported [ 74 ]. The study by Yenni RA et al. (2020) examines the non-structural Preparedness of the hospital in the face of disasters based on the hospital safety index. This research is a mixed study with a sequential explanatory design. The non-structural Preparedness of office equipment, furniture, and storage is 0.84% based on the hospital safety index. The results show that the obstacles hospitals face in implementing disaster preparedness are budget credits and the hospital’s focus on promoting non-structural Preparedness [ 75 ]. In the study of Parchami M and colleagues (2020) in the investigation of the safety situation and Preparedness of the hospitals of Ilam City against disasters, they admitted that in terms of office equipment in all the study hospitals, all the office equipment, including shelves, computers, and office furniture, are in a terrible condition and case of an accident, they will be destroyed. They failed, and no strengthening was done in any part of the hospital [ 13 ]. It was done in the study of MohammadiYegane et al. (2011) titled Assessment of Qualitative and Quantitative Performance of Equipment and the Non-structural Vulnerability of Tehran’s Elected Public Hospitals during an Earthquake. The safety level of hospital furniture was reported to be only 29% [ 54 ]. In a study conducted by Asefzadeh S and colleagues (2016), the safety score of office equipment and furniture and storage (fixed and movable) in Velayat Hospital and 50% in Rajaei Hospital has been obtained [ 39 ]. The study by Salari H and colleagues (2009) titled Preparedness of Governmental and Private Hospitals of Shiraz to Deal with Disasters was done—the lowest safety score related to office equipment and furniture (fixed and movable) [ 28 ]. In the study of Salari H et al. (2009), the lowest safety score was related to office equipment and furniture (fixed and movable) [ 28 ]. In the study of Sadeghi et al. (2015), in particular, Baath and 501 army hospitals had the highest safety percentage at 73%, and other army hospitals had the lowest safety percentage at 60% [ 45 ].

Medical and laboratory equipment and supplies used for diagnosis and treatment

The studied hospital obtained a 58.82% safety level in medical supplies and equipment. Falling of nearby equipment and facilities on medical gases can cause irreparable critical and material damage. After the implementation of the FOCUS-PDCA model by the process owners and the implementation of corrective measures in this area, including measures such as installing a sterile device (CSSD) in the operating room and increasing the oxygen production capacity required in the areas of safety of medical supplies and equipment, the safety level score It was upgraded to 65%. In the study of Abd Rahman NH et al. (2023), 36 medical supplies and equipment articles have been selected and reviewed. It concludes that the reliability of medical devices is categorized into three main areas: risk management, performance prediction, and maintenance maintenance. Most studies emphasized prioritization, failure and risk analysis, and performance prediction management systems using artificial intelligence should be enhanced to reduce the probability of failure. Age of equipment, type of equipment, performance, risk and safety, failures, rate of use, use of standard parts, maintenance schedule, and preventive repairs, availability of support, equipment installation location, weather conditions, calibration, error codes, failure frequency, the state of services, the measures taken, the failure and the history of the incident can have an impact on the level of preparedness of medical supplies and equipment in disasters. Scientific data on actions taken after failure, maintenance and repairs, cost, and a new predictive analysis model using artificial intelligence are expected to improve the current situation [ 76 ]. Mandić-Rajčević et al. (2023) have investigated various organizations that have attempted to develop assessment methods to identify and manage hospital disaster preparedness weaknesses. This article aimed to evaluate safety using the Hospital Safety Index (HSI), which has an overall safety index of 0.82%, with structural, non-structural, and disaster management safety indices of 0.95, 0.74, and 0.75, respectively, which indicates the possibility of its performance in disasters. HSI’s detailed case-by-case analysis highlights the necessary improvements in emergency water and power supply, telecommunications, and emergency medical equipment that rendered PHC inoperable during the 2014 floods. Most cases related to primary health care facilities such as hospitals were considered, except some cases in medical equipment, patient care, and support services. Fine-tuning the HSI to primary healthcare settings, its formal translation into different languages, and facilitating scoring and analysis can lead to a valid safety assessment tool in primary healthcare centers [ 73 ]. In a study conducted by Asefzadeh S (2016) et al., the safety score of medical supplies and equipment in Velayat Hospital is 55.55%, and Rajaei Hospital is 95% [ 39 ]. In the study of Sadeghi M et al. (2015), Baath Hospital and 501 had the highest safety percentage at 64%, and Ahoz Hospital, 578, with 49%, had the lowest safety percentage in medical supplies and equipment [ 45 ].

In general, in the non-structural evaluation of the studied hospital, it did not have optimal safety in facing accidents and disasters in some areas. The present study showed that the application of Total Quality Management (TQM) and, primarily, its application tool, FOCUS-PDCA, is efficient and helpful in improving the non-structural safety of hospitals. Using action research in the health field in accidents and disasters can open blind spots in different aspects of preparedness and safety (structural, non-structural, and functional). As in the present study, with the implementation of this model, the selected hospital’s non-structural safety was improved from a moderate to a reasonable level. This result can be considered a practical model in other medical centers and hospitals.

Data availability

Data is provided within the manuscript.

Ravan M, Revez MJ, Pinto IV, Brum P, Birkmann J. A vulnerability assessment framework for cultural heritage sites: the case of the roman ruins of tróia. Int J Disaster Risk Sci. 2023;14(1):26–40.

Article   Google Scholar  

Sutriningsih A, Wahyuni CU, Haksama S, Notobroto HB, Haryuni S. Validation of public health center resilience questionnaire in flood-prone areas of East Java province, Indonesia. J Public Health Afr. 2023;14(2):8.

Google Scholar  

Tahernejad S, Ghaffari S, Ariza-Montes A, Wesemann U, Farahmandnia H, Sahebi A. Post-traumatic stress disorder in medical workers involved in earthquake response: a systematic review and meta-analysis. Heliyon. 2023;9(1).

Asadi G, Karampourian A, Kalatpour O, Khazaei S. Assessment of safety status and functional, structural and non-structural preparedness of health centers in Hamadan against disasters. J Occup Hygiene Eng. 2022;8(4):69–77.

Yari A, Zarezadeh Y, Fatemi F, Ardalan A, Vahedi S, Yousefi-Khoshsabeghe H, et al. Disaster safety assessment of primary healthcare facilities: a cross-sectional study in Kurdistan province of Iran. BMC Emerg Med. 2021;21:1–9.

Nejadshafiee M, Nekoei-Moghadam M, Bahaadinbeigy K, Khankeh H, Sheikhbardsiri H. Providing telenursing care for victims: a simulated study for introducing of possibility nursing interventions in disasters. BMC Med Inf Decis Mak. 2022;22(1):54.

Ebadi A, Heidaranlu E. Virtual learning: a new experience in the shadow of coronavirus disease. Shiraz E Med J. 2020;21(12):1–2.

Nobakht M, Nasiri A, Belal M. Abbasi farajzade m. Assessment of preparedness levels in selected navy hospitals for natural disasters and accidents. J Mar Med. 2024;5(4):206–13.

Beyrami Jam M, Aminizadeh M, Akbari-Shahrestanaki Y, Khankeh HR. Evaluating the disaster preparedness of emergency medical services (EMS) facilities: a cross-sectional investigation in Iran. BMC Emerg Med. 2024;24(1):48.

Article   PubMed   PubMed Central   Google Scholar  

Jahangiri K, Rostami N, Sahebi A. Challenges of utilizing the primary health safety index tool for assessing the vulnerability of healthcare centers to disasters. Front Emerg Med. 2020;4(1):e1–e.

Ardalan A, Yusefi H, Rouhi N, Banar A, Sohrabizadeh S. Household disaster preparedness in the Islamic Republic of Iran: 2015 estimation. East Mediterr Health J. 2020;26(4):382–7.

Article   PubMed   Google Scholar  

Esmail Heidaranlu B, Ebadi A, Ardalan A, Khankeh H. A scrutiny of tools used for assessment of hospital disaster preparedness in Iran. Am J Disaster Med. 2015;10(4):325–38.

Parchami M, Mirzaei M, Kalatpour O. Identification and analysis of critical activities of fire department for residential fire scenarios using task and training requirements analysis methodology. J Occup Hyg Eng. 2020;7(1):41–50.

Aghalari Z, Khodadadi A, Ashrafian Amiri H, Jafarian S. Evaluation of the level of safety and disaster preparedness of comprehensive healthcare centers in Babol, Iran. Q Sci J Rescue Relief. 2019;11(4):287–93.

Jafari AJ, Baba M, Dowlati M. Disaster risk assessment in health centers of Iran University of Medical Sciences in functional, non structural & structural components during 2015–2016. Iran Occup Health. 2018;15(1):76–85.

Mojtabi Ferdowsi YS, Behzadnia MJ, Heidaranlu E, Salari MM. Evaluation of the effects of targeted oxygen therapy on the recovery outcomes of patients with COVID-19 using the experiences and capacities of a military hospital: a randomized clinical trial. J Military Med. 2023;25(1):1719–29.

Dargahi A, Farrokhi M, Poursadeghiyan M, Ahagh MMH, Karami A. Evaluation of functional preparedness and non structural safety of different health units of Kermanshah University of Medical Sciences in coping with natural disasters. Health Emergencies Disasters Q. 2017;2(4):201–6.

Heidaranlu E, Bagheri M, Moayed MS. Assessing the preparedness of military clinical nurses in the face of biological threats: with a focus on the COVID-19 disease. J Military Med. 2022;24(6):1419–26.

Hatami H, Neisi A, Kooti M. Functional, structural and non-structural preparedness of Ahvaz health centers against disasters in 2014–2015. Jundishapur J Health Sci. 2017;9(1).

Jahangiri K, Izadkhah YO, Lari A. Hospital safety index (HSI) analysis in confronting disasters: a case study from Iran. Int J Health Syst Disaster Manage. 2014;2(1):44.

Huang Y, Huang Y, Yi L, Pan W, Chen Y. A FOCUS-PDCA quality improvement model for reducing the distribution defect rate of sterile packages. Sci Rep. 2023;13(1):15016.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Heidaranlu E, Amiri M, Salaree MM, Sarhangi F, Saeed Y, Tavan A. Audit of the functional preparedness of the selected military hospital in response to incidents and disasters: participatory action research. BMC Emerg Med. 2022;22(1):168.

Heydari F, Meibody AA, Ebrahimi M, Zakeri R, Marzaleh MA, Bagheri R. Hospital safety index: evaluation of the readiness and safety of hospitals in Isfahan province, Iran. Front Emerg Med. 2024.

Salevaty J, Khankeh HR, Dalvandi A, Delshad V. The impact of nurses training and applying functional and nonstructural hospital safety in preparedness of Razi and day hospitals in disasters based on hospital safety index. 2015.

Heidaranlu E, Goyaghaj NS, Moradi A, Ebadi A. The efficacy of an intervention program for pain intensity reduction in patients undergoing arterial sheath removal after coronary artery angioplasty. J Tehran Univ Heart Cent. 2021;16(3):102.

Tepe RCC, Jahangiri K, Dolatabadi ZA. Survey of Farsi hospital safety index in teaching hospitals of Tehran University of Medical Sciences in a four-year period from 2018 to 2021. Irtiqa Imini Pishgiri Masdumiyat. 2023;11.

Rabeian M, Hosseini SH, Radabadi M, Mirghaed MT, Bakhtiari M. Evaluation of effective factors on the rate of preparedness of Tehran University of Medical Sciences’ selected hospitals in dealing with earthquake. Payavard Salamat. 2013;7(3).

Salari H, Esfandiari A, Heidari A, Julaee H, Rahimi SH. Survey of natural disasters preparedness in public and private hospitals of islamic republic of Iran (case study of shiraz, 2011). Int J Health Syst Disaster Manag. 2013;1(1):26–31.

ARAB M, ZERAATI H, AKBARI HFE, Ravangard R, A study on the executive managers’knowledge, and performance, and their hospitals preparedness against earthquake events and their relationships at public hospitals. (Affiliated by Tehran University of Medical Sciences (TUMS); 2005–2006). 2009.

Ojaghi S, Nourizad S, Mahboobi M, Khazaei M, Najafi G. Disaster crisis handling preparedness level of hospitals in Kermanshah. J Kermanshah Univ Med Sci. 2009;13(3).

Al Khalaileh MA, Bond E, Alasad JA. Jordanian nurses’ perceptions of their preparedness for disaster management. Int Emerg Nurs. 2012;20(1):14–23.

Sabzghabaie A, Kondori A, Shojaee M, Hatamabadi H, Amini A. Hospital safety in hospitals affiliated with Shahid Beheshti University of Medical Sciences in 2011-13. Pajoohandeh J. 2013;18(2):83–7.

Jagnarine S. Safe hospitals in Caribbean. Pan American Health Organization; 2014.

Abbasabadi-Arab M, Miadfar J, Yousefian S, Mobini A, Mehran Amin S. Assessing hospital safety index in the Iranian hospitals. Health Emergencies Disasters Q. 2023;8(4):313–22.

Takzadeh H, Afshari S, Sokhanvari F, Jafari M. Safety assessment of urban and rural health services centers. Isfahan Health Homes Health Centers Barriers Emergencies. 2015;2015.

Tabrizi J, Gharibi F, Mohammadi Z, editors. Survey level of physical safety of health care centers in Tabriz in 2015. The 1st Annual Conference of Health Management in Disasters and Emergencies, Tabriz, Iran; 2016.

Amerion A, Delaavari A, Teymourzadeh E. Rate of preparedness in confronting crisis in three selected border hospitals. J Military Med. 2022;12(1):19–22.

Mahboobi M. Reading of content of border hospitals’ readiness in Kermanshah state for confronting with board’s crisis: MA thesis]. Tehran: Islamic Azad University; 2008.

Asefzadeh S, Varyani AS, Gholami S. Disaster risk assessment in educational hospitals of Qazvin based on WHO pattern in 2015. Electron Physician. 2016;8(1):1770.

Cliff BJ, Morlock L, Curtis AB. Is there an association between risk perception and disaster preparedness in rural US hospitals? Prehospital and disaster medicine. 2009;24(6):512–7.

Heidaranlu E, Habibi F, Moradi A, Lotfian L. Determining functional preparedness of selected military hospitals in response to disasters. Trauma Monthly. 2020;25(6):249–53.

Zhong S, Clark M, Hou X-Y, Zang Y, FitzGerald G. Validation of a framework for measuring hospital disaster resilience using factor analysis. Int J Environ Res Public Health. 2014;11(6):6335–53.

Luke J, Franklin RC, Dyson J, Aitken P. Building toward a disaster resilient health system: a study of hospital resilience. Disaster Med Pub Health Prep. 2023;17:e219.

Lestari F, Paramitasari D, Kadir A, Firdausi NA, Fatmah, Yani Hamid A, et al. The application of hospital safety index for analyzing primary healthcare center (PHC) disaster and emergency preparedness. Sustainability. 2022;14(3):1488.

Sadeghi M, Gholamhoseini L, Esmaeel Zali M, Mehrabi N, Nejati B. The survey on the vulnerability of military hospitals in crisis situation and war and offer suitable solutions and mission in these situations in the second semester of 1391. Paramedical Sci Military Health. 2015;10(1):24–30.

Yang Z, Barroca B, Weppe A, Bony-Dandrieux A, Laffréchine K, Daclin N, et al. Indicator-based resilience assessment for critical infrastructures–A review. Saf Sci. 2023;160:106049.

Heidaranlu E, Khankeh H, Ebadi A, Ardalan A. An evaluation of non-structural vulnerabilities of hospitals involved in the 2012 east Azerbaijan earthquake. Trauma Monthly. 2017;22(2):6.

Lestari F, editor. Hospital disaster preparedness in center government hospital Bandung, West Java, Indonesia. International Conference Disaster Management & Infection Control; 2017.

Daneshmandi M, Amiri H, Vahedi M, Farshi M, saghafi AE, Zigheymat F. Assessing the level of preparedness for confronting crisis such as flood, earthquake, fire and storm in some selected hospitals of Iran. 2010.

Hojat M. Disaster disaster preparedness in hospitals of Jahrom University of Medical Sciences (2010). Journal of Inflammatory Diseases. 2012;16(3):72–7.

Kavari S, MOSLEHI S, Panahi A. Crisis management at hospitals affiliated to Shiraz University of Medical Sciences. 2007.

Zaboli R, Sajadi HS. Assessing hospital disaster preparedness in Tehran: lessons learned on disaster and mass casualty management system. Int J Health Syst Disaster Manage. 2014;2(4):220.

Hosseini Shokouh S, Arab M, Rahimi A, Rashidian A, Sadr Momtaz N. Preparedness of the Iran University of Medical Sciences hospitals against earthquake. J School Public Health Inst Public Health Res. 2008;6(3–4):61–77.

Mohammadi Yeganeh S, Asadi Lari M, Seyedin SH, Maher A. Qualitative and quantitative performance of equipment and non-structural vulnerability in selected public hospitals in Tehran City during an earthquake. Q Sci J Rescue Relief. 2011;3(1):0.

Tal MA, Aomar RA, Abel J. A predictive model for an effective maintenance of hospital critical systems. 2021.

Zegarra RO, Arévalo-Ipanaqué JM, Sánchez RAA, Martínez JAC, Echevarría JCD, González PA. Disaster preparedness and hospital safety in state hospitals in Lima (Peru). Prehosp Disaster Med. 2023;38(5):601–5.

Kjølle GH, Utne IB, Gjerde O. Risk analysis of critical infrastructures emphasizing electricity supply and interdependencies. Reliab Eng Syst Saf. 2012;105:80–9.

Bisanda MP, Amadi-Echendu J, editors. A pre and post Covid-19 trend analyses of mobile telecommunications systems. 2023 Portland International Conference on Management of Engineering and Technology (PICMET); 2023: IEEE.

Lamine H, Chebili N, Zedini C. Evaluating the level of disaster preparedness of Tunisian University hospitals using the hospital safety index: a nationwide cross-sectional study. Afr Health Sci. 2022;22(3):666–73.

EL-Matury HJ, Lestari F, Sunindijo RY. Survey data set on the readiness of hospitals for disaster and emergency in North Sumatra Indonesia. Data Brief. 2024:110358.

Luke J, Franklin R, Aitken P, Dyson J. Safer hospital infrastructure assessments for socio-natural disaster–a scoping review. Prehosp Disaster Med. 2021;36(5):627–35.

Irwanto BSP, Ernawati M, Paskarini I, Amalia AF. Evaluation of fire prevention and control system in Dr. R. Koesma Regional General Hospital of Tuban Regency in 2021. Indonesian J Occup Saf Health. 2023;12(1):143–55.

Lestari F, Paramitasari D, Fatmah, Yani Hamid A, Suparni, EL-Matury HJ, et al. Analysis of hospital’s emergency and disaster preparedness using hospital safety index in Indonesia. Sustainability. 2022;14(10):5879.

Article   CAS   Google Scholar  

Blanchette RA, van Bree EM, Bierens JJ. Hospital disaster preparedness in the Netherlands. Int J Disaster risk Reduct. 2023;93:103778.

Awodele O, Adewoye AA, Oparah AC. Assessment of medical waste management in seven hospitals in Lagos, Nigeria. BMC Public Health. 2016;16:1–11.

Heidaranlu E, Tavan A, Aminizadeh M. Investigating the level of functional preparedness of selected Tehran hospitals in the face of biological events: a focus on COVID-19. Int J Disaster Resil Built Environ. 2022;13(2):150–62.

Muhammad-Idris Z, Joshua I, Stanley A, Igboanusi C, Oguntunde R, Audu O. Disasters and hospital safety in Nigeria. GET J Biosecur One Health. 2022;1:1–6.

Vichova K, Hromada M. The evaluation system to ensure the transport of emergency supplies of fuel to the hospitals. Transp Res Procedia. 2019;40:1618–24.

Moradi SM, Nekoei-Moghadam M, Abbasnejad A, Hasheminejad N. Risk analysis and safety assessment of hospitals against disasters: a systematic review. J Educ Health Promotion. 2021;10(1).

Goniewicz M, Khorram-Manesh A, Timler D, Al-Wathinani AM, Goniewicz K. Hospital disaster preparedness: a comprehensive evaluation using the hospital safety index. Sustainability. 2023;15(17):13197.

Raju R. An evaluation of centralized medical gas and vacuum supply system in a multispeciality hospital. Rajiv Gandhi University of Health Sciences (India); 2019.

Gómez-Chaparro M, García-Sanz-Calcedo J, Armenta Marquez L. Analytical determination of medical gases consumption and their impact on hospital sustainability. Sustainability. 2018;10(8):2948.

Chair SY, Ng ST, Chao CYH, Xu JF. Heating, ventilation, and air-conditioning systems in healthcare: a scoping review. J Hosp Infect. 2023;141:33–40.

Article   CAS   PubMed   Google Scholar  

Ratwani RM, Adams KT, Kim TC, Busog D-NC, Howe JL, Jones R, et al. Assessing equipment, supplies, and devices for patient safety issues. Patient Saf. 2023;5(1):15–25.

Yenni RA, Novrikasari N, Windusari Y. Non-structural preparedness analysis at RSUP Dr. Mohammad Hoesin in disaster based on the hospital safety index. Jurnal Aisyah: Jurnal Ilmu Kesehatan. 2020;5(2):123-8.

Abd Rahman NH, Ibrahim AK, Hasikin K, Abd Razak NA. Critical device reliability assessment in healthcare services. Journal of healthcare engineering. 2023;2023.

Download references

Acknowledgements

The author would like to thank the “Clinical Research Development Unit” of Baqiyatallah Hospital for their kind cooperation.

The author(s) received no fnancial support for the research, authorship, and/or publication of this article.

Author information

Authors and affiliations.

Health Research Center, Baqiyatullah University of Medical Sciences, Tehran, Iran

Alireza Basiri, Mohsen Abbasi Farajzadeh & Mohammad Belal

Student Research Committee, Baqiyatallah University of Medical Sciences, Tehran, Iran

Mohsen Abbasi Farajzadeh & Mohammad Belal

Nursing Care Research Center, Clinical Science Institute and Nursing Faculty of Baqiyatallah University of Medical Sciences, Tehran, Iran, Baqiyatallah University of Medical Sciences, Tehran, Iran

Esmail Heidaranlu

You can also search for this author in PubMed   Google Scholar

Contributions

Writing and preparation “Abstract”: Heidaranlu & Basiri. Writing and preparation “Introduction”: Heidaranlu & Basiri. Writing and preparation “Methods”: Heidaranlu & Abbasi. Writing and preparation “Results”: Heidaranlu & Abbasi. Writing and preparation “tables and figures and Diagrams”: Heidaranlu & Basiri. Writing and preparation “Discussion”: Abbasi & Basiri. Writing and preparation “Conclusion”: Heidaranlu. All authors (Heidaranlu & Basiri & Abbasi & Belal) reviewed the manuscript.

Corresponding author

Correspondence to Esmail Heidaranlu .

Ethics declarations

Ethics approval and consent to participate.

The present article is taken from the approved research project 402000185 at Baqiyatullah University of Medical Sciences with ethics ID IR.BMSU.REC.1402.040. This study was conducted in 2023 this article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all participants participating in the study. The confidentiality and anonymity of the participants were determined by coding the questionnaires. Study participants were informed of their freedom to withdraw from the study at any time without giving a reason.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

1 - (F) Find a problem to improve. Organization (O); turn on (C); Understanding (U); selection (S); plan (P); do (D); Review (C); And, rule (A)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Basiri, A., Farajzadeh, M.A., Belal, M. et al. Improving the non-structural preparedness of the selected hospital based on the FOCUS-PDCA 1 model: action research. BMC Emerg Med 24 , 109 (2024). https://doi.org/10.1186/s12873-024-01006-w

Download citation

Received : 28 January 2024

Accepted : 13 May 2024

Published : 09 July 2024

DOI : https://doi.org/10.1186/s12873-024-01006-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

  • Non-structural vulnerability
  • Safety level assessment
  • Disasters and emergencies

BMC Emergency Medicine

ISSN: 1471-227X

critical evaluation of research article example

ACM Digital Library home

  • Advanced Search

Network reliability evaluation of a supply chain under supplier sustainability

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, recommendations, sustainable supply chain design: an application example in local business retail.

The paper demonstrates the need to consider different sustainability aspects in supply chain redesign and optimization problems. To this end a simulation model of local pharmaceutical business retail is presented as application example. Three different ...

Improving Supply Chain Sustainability and Resilience via anyLogistix: Research Trends and Future Challenges

Supply chain management faces difficult challenges today, given the rapidly evolving global business landscape. The need for resilient supply chains capable of anticipating, adapting to, and recovering from disruptions has become paramount. At the ...

Sustainability Implications of Supply Chain Responsiveness

Problem definition : A critical decision made by firms is whether to adopt a responsive supply chain (prioritizing speed) or an efficient supply chain (prioritizing cost). We consider the environmental implications of this choice, distinguishing between ...

Information

Published in.

Pergamon Press, Inc.

United States

Publication History

Author tags.

  • Multistate supply chain network (MSCN)
  • Sustainability
  • Network reliability
  • Method Based on the Removal Effects of Criteria (MEREC)
  • Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)
  • Research-article

Contributors

Other metrics, bibliometrics, article metrics.

  • 0 Total Citations
  • 0 Total Downloads
  • Downloads (Last 12 months) 0
  • Downloads (Last 6 weeks) 0

View options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

  • Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
  • Download citation
  • Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

  • All health topics »
  • Coronavirus disease (COVID-19)
  • Reproductive health
  • Air pollution
  • Mental health
  • Publications

For the future - publication thumbnail

  • Initiatives
  • Health security and AMR
  • NCDs and ageing
  • Climate change
  • Reaching the unreached

critical evaluation of research article example

  • News releases
  • Feature stories
  • Photo stories
  • Commentaries
  • Photo library
  • Press contact
  • People of the Western Pacific

6 people-02

  • Dzud in Mongolia
  • Mpox outbreak
  • Seasonal influenza
  • Avian influenza
  • Pacific islands
  • Emergency medical teams
  • The Global Outbreak Alert and Response Network (GOARN)
  • Preparedness
  • Health Data Platform >>
  • How we work
  • Where we work
  • Our programmes
  • Country support
  • Pacific technical support
  • Regional Director
  • Regional Committee
  • Collaborating Centers
  • Regional health initiatives

RCM meeting room

WHO Technical Advisory Group on Reaching the Unreached (TAG-RTU) for the Western Pacific Region

Issued on: 15 july 2024 deadline: 15 august 2024.

The World Health Organization (WHO) Regional Office for the Western Pacific is seeking additional experts to serve as members of the Technical Advisory Group on Reaching the Unreached (TAG-RTU) for the Western Pacific Region.  This “Call for experts” provides information about the advisory group, the expert profiles being sought, the process to express interest, and the selection process.

Reaching the Unreached is critical to achieving universal health coverage and to realizing the vision of the Sustainable Development Goals to leave no one behind.  At its 73 rd session, the Regional Committee for the Western Pacific endorsed the Regional Framework for Reaching the Unreached in the Western Pacific 2022-2030 .  The vision of this RTU Framework is for transformed health health-care delivery and public health systems across the Western Pacific Region that routinely reach everyone everywhere and ensure equitable health outcomes.  The RTU Framework highlights that action is required in five key domains: (1) political commitment coupled with governance, financing, and legal structures; (2) multi-stakeholder engagement; (3) data and evidence; (4) health service transformation built on high-quality primary health care; and (5) special approaches to allow quick adaptation to specific challenges or crises beyond the immediate capacity of existing health services.  Active engagement of communities is critical across all of these action domains. 

The WHO Regional Office for the Western Pacific supports the Member States in implementing the RTU Framework.  The TAG-RTU provides independent external review and strategic advice to WHO/WPRO on reaching the unreached across the WHO Western Pacific Region.

Functions of the TAG-RTU

In its capacity as an advisory body to the WHO Regional Office for the Western Pacific, the TAG-RTU shall have the following functions:

  • To review, from a scientific and technical standpoint, the content, scope and dimensions of activities related to reaching the unreached, including any research areas covered and approaches to be adopted;
  • To provide an independent evaluation of the scientific and technical aspects of the ‘reaching the unreached’ theme;
  • To recommend priorities within the ‘reaching the unreached’ theme; and
  • To review and make recommendations on progress in implementing the Regional Framework for Reaching the Unreached in the Western Pacific.

Operations of the TAG-RTU

The TAG-RTU will typically meet at least once each year.  However, WHO may convene additional meetings.  TAG-RTU meetings may be held in person (at WHO Regional Office for the Western Pacific in Manila or another location, as determined by WHO) or virtually via video or teleconference.  The working language of the meeting and documentation is English.  No stipend or honorarium is paid to members.  However, should meetings take place in person, travel costs will be covered, and an expense allowance will be provided.

Members of TAG-RTU will participate in their individual capacities and not as representatives of their organization, institution, or country.  The complete functions, composition, nomination process, and duration of appointments are detailed in the Terms of Reference.

Who can express interest?

The TAG-RTU will be multidisciplinary, with members with technical knowledge, skills, and experience relevant to reaching unreached populations across countries and areas in the Western Pacific Region.  Up to 5 members may be selected from the expressions of interest to this call.

Nominees should be recognized experts and leading thinkers in developing and implementing strategies to reach groups unreached by systems and services and bring expertise relevant to achieving the vision of the Regional Framework.  Individuals with expertise in reaching the unreached in low-and middle-income countries, including experts from unreached groups, particularly in the Asia Pacific Region, are especially encouraged to express interest. 

This call provides an excellent opportunity for leading regional and global experts to advise WHO in its mission to transform health systems and services to reach everyone and improve equity of health outcomes across the Western Pacific Region.

WHO particularly welcomes expressions of interest from individuals with one or more types of expertise listed below:

  • Ethics, equity, human rights, and gender in public health
  • Health systems transformation to reach unreached groups.  This may include expertise relevant to governance, law, financing, and/or design and implementation of policies, strategies, and initiatives that enable health systems to reach the unreached
  • Universal health coverage and primary health care
  • Community engagement – especially in engaging unreached communities in processes to identify issues and co-design solutions
  • Community expertise – including experts with lived experience of belonging to groups who are unreached by health systems
  • Multisectoral action
  • Data and evidence – particularly the development and use of processes for collection, analysis, and use of data and evidence that make unreached groups visible and can be used to support effective planning, monitoring, and evaluation
  • Addressing factors that contribute to populations being unreached, such as stigma and discrimination, social determinants of health, and the settings in which people live and work.

In assessing applications, the following criteria will be applied: 

  • Track record and recognized achievements in reaching the unreached 
  • Achievement of a balance of expertise across multiple disciplines and sectors, gender diversity, and  geographical origin

Submitting your expression of interest

To register your interest in being considered for the WHO Technical Advisory Group for Reaching the Unreached in the Western Pacific Region, please submit the following documents by 15 August 2024 24:00h  Manila time to Dr Rajendra Yadav ( [email protected] ) copied to Leonora Esguerra ( [email protected] ) and Roxanne Divina ( [email protected] )

(Please note that, if selected, membership will be in a personal capacity.  Therefore do not use the letterhead or other identification of your employer).  Also, please use the subject line “Expression of interest for the TAG on Reaching the Unreached”.)

  • Fill out the survey form at this link: https://forms.office.com/e/wQeGM0nEa2
  • Your curriculum vitae; and

https://www.who.int/about/ethics/declarations-of-interest .

After submission, your expression of interest will be reviewed by WHO.  Due to an expected high volume of interest, only selected individuals will be informed. 

Important information about the selection processes and conditions of appointment

Members of WHO advisory groups (AGs) must be free of any real, potential or apparent conflicts of interest.  To this end, applicants are required to complete the WHO Declaration of Interests for WHO Experts.  The selection as a member of an AG is, amongst other things, dependent on WHO determining that there is no conflict of interest or that any identified conflicts could be appropriately managed (in addition to WHO’s evaluation of an applicant’s experience, expertise and motivation and other criteria).

  • All AG members will serve in their individual expert capacity and shall not represent any governments, any commercial industries or entities, any research, academic, or civil society organizations, or any other bodies, entities, institutions or organizations.  They are expected to fully comply with the Code of Conduct for WHO Experts ( https://www.who.int/publications/m/item/declaration-of-interests-for-who-experts ).  AG members will be expected to sign and return a completed confidentiality undertaking prior to the beginning of the first meeting.

At any point during the selection process, telephone interviews may be scheduled between an applicant and the WHO Secretariat to enable WHO to ask questions relating to the applicant’s experience and expertise and/or to assess whether the applicant meets the criteria for membership in the relevant AG.

The selection of members of the AGs will be made by WHO in its sole discretion, taking into account the following (non-exclusive) criteria: relevant technical expertise; experience in international and country policy work; communication skills; and ability to work constructively with people from different cultural backgrounds and orientations.  The selection of AG members will also take into account of the need for diverse perspectives from different regions, especially from low and middle-income countries, and for gender balance.

If selected by WHO, proposed members will be sent an invitation letter and a Memorandum of Agreement.  Appointment as a member of an AG will be subject to the proposed member returning to WHO the countersigned copy of these two documents.

WHO reserves the right to accept or reject any expression of interest, to annul the open call process, and reject all expressions of interest at any time without incurring any liability to the affected applicant or applicants and without any obligation to inform the affected applicant or applicants of the grounds for WHO’s action.  WHO may also decide, at any time, not to proceed with the establishment of the AG, disband an existing TAG, or modify the work of the AG.

WHO shall not in any way be obliged to reveal, or discuss with any applicant, how an expression of interest was assessed, provide any other information relating to the evaluation/selection process, or state the reasons for not choosing a member.

WHO may publish the names and a short biography of the selected individuals on the WHO internet.

AG members will not be remunerated for their services in relation to the AG or otherwise.  However, should meetings take place in person, travel and accommodation expenses of AG members to participate in AG meetings will be covered by WHO in accordance with its applicable policies, rules, and procedures.

The appointment will be limited in time as indicated in the letter of appointment.

If you have any questions about this “Call for experts”, please write to Dr Rajendra Yadav ( [email protected] ), ICD Coordinator,  well before the applicable deadline. 

Series No:  2024/June/169 

IMAGES

  1. 🌷 How to write a critical review of multiple articles. How to Write

    critical evaluation of research article example

  2. FREE 7+ Critical Analysis Templates in MS Word

    critical evaluation of research article example

  3. 🌷 Critical article review format. Read This Critical Review Of Journal

    critical evaluation of research article example

  4. Example Critical Evaluation

    critical evaluation of research article example

  5. (DOC) CRITICAL REVIEW OF A JOURNAL ARTICLE 1. Source

    critical evaluation of research article example

  6. How to Critically Review a Scientific Manuscript

    critical evaluation of research article example

VIDEO

  1. Critical Evaluation of the Medical and Behavioral Research Literature Part II

  2. Critical evaluation of research papers, thesis & projects

  3. Epidemiology: Screening & Diagnostic Test, Interesting Video Lecture in Amharic Speech, Part 8

  4. Academic Writing: the Critical Evaluation Essay

  5. Epidemiology: Evaluation of Evidence & Judgment of Causality, Helpful Video in Amharic, Part 4

  6. Module 1: Critically Evaluating Research Literature

COMMENTS

  1. Write a Critical Review of a Scientific Journal Article

    What methods were used to accomplish this purpose (e.g., systematic recording of observations, analysis and evaluation of published research, assessment of theory, etc.)? What techniques were used and how was each technique performed? ... For example, stating that the sample size is insufficient is not a critical assessment. Describing why the ...

  2. (PDF) How to critically appraise an article

    SuMMarY. Critical appraisal is a systematic process used to identify the strengths. and weaknesse s of a res earch article in order t o assess the usefulness and. validity of r esearch findings ...

  3. PDF Planning and writing a critical review

    appraisal, critical analysis) is a detailed commentary on and critical evaluation of a text. You might carry out a critical review as a stand-alone exercise, or as part of your research and preparation for writing a literature review. The following guidelines are designed to help you critically evaluate a research article. What is meant by ...

  4. Writing Critical Reviews: A Step-by-Step Guide

    the article, taking the main points of each paragraph. The point of the diagram is to. show the relationships between the main points in the article. Ev en better you might. consider doing an ...

  5. How to critically appraise an article

    Key Points. Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article. Critical appraisal provides a basis for decisions on whether to use the ...

  6. Critical Analysis of Clinical Research Articles: A Guide for Evaluation

    Abstract. Critical evaluation is used to identify the strengths and weaknesses of an article, in order to evaluate the usefulness and validity of research results. The components of the critical ...

  7. Writing a Critical Review

    The critical review is a writing task that asks you to summarise and evaluate a text. The critical review can be of a book, a chapter, or a journal article. Writing the critical review usually requires you to read the selected text in detail and to read other related texts so you can present a fair and reasonable evaluation of the selected text.

  8. Critical Reviews

    What is a Critical Review of a Journal Article? A critical review of a journal article evaluates the strengths and weaknesses of an article's ideas and content. It provides description, analysis and interpretation that allow readers to assess the article's value. Before You Read the Article. What does the title lead you to expect about the article?

  9. Write a Critical Review of a Scientific Journal Article

    A critical review is an assessment of an original research article. Writing a critical review of a journal article can help you improve your research skills. By assessing the work of others, you develop skills as a critical reader and become familiar with the types of evaluation criteria that will be applied to research in your field.

  10. Critical Appraisal of Scientific Articles

    Selection, reading, and critical appraisal of publications is, however, necessary to stay up to date in one's field. This is also demanded by the precepts of evidence-based medicine (1, 2). Besides the medical content of a publication, its interpretation and evaluation also require understanding of the statistical methodology.

  11. Critical evaluation of publications

    Critical evaluation is the process of examining the research for the strength or weakness of the findings, validity, relevance, and usefulness of the research findings. [ 1] The availability of extensive information and the difficulty in differentiating the relevant information obligate the primary need of critical appraisal.

  12. The fundamentals of critically appraising an article

    Here are some of the tools and basic considerations you might find useful when critically appraising an article. In a nutshell when appraising an article, you are assessing: 1. Its relevance ...

  13. Critically reviewing literature: A tutorial for new researchers

    Critically reviewing the literature is an indispensible skill which is used throughout a research career. This article demystifies the processes involved in systematically and critically reviewing the literature to demonstrate knowledge, identify research ideas, position research and develop theory. Although aimed primarily at research students ...

  14. Writing Tips: Critically Evaluating Research

    Below is a guide with examples of how to critically evaluate research and how to communicate your ideas in writing. To develop the skill of being able to critically evaluate, when reading research articles in psychology read with an open mind and be active when reading. Ask questions as you go and see if the answers are provided. Initially skim ...

  15. Critical Analysis: The Often-Missing Step in Conducting Literature

    Literature reviews are essential in moving our evidence-base forward. "A literature review makes a significant contribution when the authors add to the body of knowledge through providing new insights" (Bearman, 2016, p. 383).Although there are many methods for conducting a literature review (e.g., systematic review, scoping review, qualitative synthesis), some commonalities in ...

  16. Full article: Critical appraisal

    For example, GRADE (Grading Recommendations, Assessment, Development, and Evaluation, Guyatt et al., Citation 2011) is designed for reviews of quantitative research and is undertaken in two broad phases. First, reviewers conduct a systematic review (a) to generate a set of findings and (b) to assess the quality of the research.

  17. A guide to critical appraisal of evidence : Nursing2020 Critical Care

    Critical appraisal is the assessment of research studies' worth to clinical practice. Critical appraisal—the heart of evidence-based practice—involves four phases: rapid critical appraisal, evaluation, synthesis, and recommendation. This article reviews each phase and provides examples, tips, and caveats to help evidence appraisers ...

  18. How to Write an Article Review (with Sample Reviews)

    Spread the loveAn article review is a critical evaluation of a scholarly or scientific piece, which aims to summarize its main ideas, assess its contributions, and provide constructive feedback. A well-written review not only benefits the author of the article under scrutiny but also serves as a valuable resource for fellow researchers and scholars. Follow these steps to create an effective ...

  19. PDF Step'by-step guide to critiquing research. Part 1: quantitative research

    critiquing the literature, critical analysis, reviewing the literature, evaluation and appraisal of the literature which are in essence the same thing (Bassett and Bassett, 2003). Terminology in research can be confusing for the novice research reader where a term like 'random' refers to an organized manner of selecting items or participants ...

  20. Structure of a Critical Review

    Summarising and paraphrasing are essential skills for academic writing and in particular, the critical review. To summarise means to reduce a text to its main points and its most important ideas. The length of your summary for a critical review should only be about one quarter to one third of the whole critical review. The best way to summarise.

  21. PDF Writing a Critique or Review of a Research Article

    Agreeing with, defending or confirming a particular point of view. Proposing a new point of view. Conceding to an existing point of view, but qualifying certain points. Reformulating an existing idea for a better explanation. Dismissing a point of view through an evaluation of its criteria. Reconciling two seemingly different points of view.

  22. Evaluating Sources

    Lateral reading is the act of evaluating the credibility of a source by comparing it to other sources. This allows you to: Verify evidence. Contextualize information. Find potential weaknesses. If a source is using methods or drawing conclusions that are incompatible with other research in its field, it may not be reliable. Example: Lateral ...

  23. Example 5

    Example 5 - Evaluation of Critiques of Scientific Articles. Characteristics to note in the rubric: Language is descriptive, not evaluative. Labels for degrees of success are descriptive ("Expert" "Proficient", etc.); by avoiding the use of letters representing grades or numbers representing points, there is no implied contract that qualities of ...

  24. A systematic review of experimentally tested implementation strategies

    Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies ...

  25. Women's engagement with community perinatal mental health services: a

    These questions are critical to understanding which factors ensure CPMHTs successfully deliver care. Rationale for using realist evaluation and research question. In this paper, we report a realist evaluation that examined how elements of CPMHTs provider care affected women's engagement with services.

  26. Evaluation of cross-cultural adaptation and validation of the Persian

    Introduction Assessing critical thinking disposition is crucial in nursing education to foster analytical skills essential for effective healthcare practice. This study aimed to evaluate the cross-cultural adaptation and validation of the Persian version of the Critical Thinking Disposition Scale among Iranian nursing students. Method A total of 390 nursing students (mean age = 21.74 (2.1 ...

  27. Evaluation of the sentinel yellow fever surveillance system in Uganda

    Background Uganda has a sentinel surveillance system in seven high-risk sites to monitor yellow fever (YF) patterns and detect outbreaks. We evaluated the performance of this system from 2017 to 2022. Methods We evaluated selected attributes, including timeliness (lags between different critical time points), external completeness (proportion of expected sentinel sites reporting ≥ 1 suspect ...

  28. Improving the non-structural preparedness of the selected hospital

    Although in the evaluation before the change, the overall score of the hospital was 71.01%, and it had a desirable performance level in non-structural factors, in all the involved parts of the hospital, the sensitive, critical, and practical parts in the operation of the hospital had an average and sometimes low safety level.

  29. Network reliability evaluation of a supply chain under supplier

    In contrast to the previous research, this study involves sustainability in network reliability evaluation. A new algorithm is proposed for computing network reliability, the probability that the current supply chain not only meets demand, budget, and suppliers' production capacity but also considers the level of supplier sustainability.

  30. WHO Technical Advisory Group on Reaching the Unreached (TAG-RTU) for

    The World Health Organization (WHO) Regional Office for the Western Pacific is seeking additional experts to serve as members of the Technical Advisory Group on Reaching the Unreached (TAG-RTU) for the Western Pacific Region. This "Call for experts" provides information about the advisory group, the expert profiles being sought, the process to express interest, and the selection process ...