Advertisement

Challenges to Engaging Women Veterans in Quality Improvement From Patient Care to Policy: Women's Health Managers' Perspectives

  • Tanya T. Olmos-Ochoa
    Correspondence
    Correspondence to: Tanya T. Olmos-Ochoa, PhD, HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, 16111 Plummer Street (152), North Hills, CA 91343. Phone: (818) 891-7711x36041.
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California
    Search for articles by this author
  • Tana M. Luger
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California

    Covenant Health Network, Phoenix, Arizona
    Search for articles by this author
  • Anneka Oishi
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California
    Search for articles by this author
  • Karen E. Dyer
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California
    Search for articles by this author
  • Annie Sumberg
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California
    Search for articles by this author
  • Ismelda Canelo
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California
    Search for articles by this author
  • Theodore K. Gideonse
    Affiliations
    Department of Health, Science, and Behavior, Program in Public Health, University of California Irvine, Irvine, California
    Search for articles by this author
  • Ann Cheney
    Affiliations
    Department of Social Medicine Population and Public Health, University of California Riverside School of Medicine, Riverside, California
    Search for articles by this author
  • Elizabeth M. Yano
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California

    Department of Health Policy & Management, Fielding School of Public Health, University of California, Los Angeles, Los Angeles, California

    Department of Medicine, Geffen School of Medicine, University of California, Los Angeles, Los Angeles, California
    Search for articles by this author
  • Alison B. Hamilton
    Affiliations
    HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP), VA Greater Los Angeles Healthcare System, Sepulveda, California

    Department of Psychiatry & Biobehavioral Sciences, Geffen School of Medicine, University of California Los Angeles, Los Angeles, California
    Search for articles by this author
Open AccessPublished:September 21, 2022DOI:https://doi.org/10.1016/j.whi.2022.08.004

      Abstract

      Introduction

      Patients are uniquely positioned to identify issues and to provide innovative solutions to problems impacting their care. Yet, patient engagement in quality improvement (QI) and health care governance remains limited and underexplored. In the Veterans Health Administration, the work of women's health managers (WHMs) includes engaging women veterans, a numerical minority with unique health care needs, in QI. We aimed to understand the extent to which WHMs engage women veterans along a continuum, highlight challenges to engagement, and identify potential strategies to facilitate multilevel patient engagement.

      Methods

      Data were generated from a multisite evaluation to improve delivery of comprehensive women's health care in Veterans Health Administration primary care sites. We conducted 39 semistructured interviews with WHMs across 21 sites. Guided by Carman et al.’s patient engagement framework, we analyzed the interviews using rapid-qualitative and content analysis methods.

      Results

      When effectively engaged, women veterans were important champions and partners in QI activities to improve the structure and delivery of care. However, most WHMs engaged women veterans in mainly informal or passive ways—that is, solicited feedback through comment cards, surveys, focus groups, and townhall meetings—and did not report pursuing more in-depth or long-term forms of engagement. WHMs also identified a variety of facilitators and challenges to engaging women veterans in QI.

      Conclusions

      There may be unanticipated benefits to health care policy from engaging patients in QI, especially for patients with unique health care needs who represent a minority within the health care system. However, managers require training and workflow integration of patient engagement tasks to increase their efficiency and allow for meaningful patient engagement.
      Patient engagement has been defined as “patients, families, their representatives, and health professionals working in active partnership at various levels across the health care system…to improve health and health care” (
      • Carman K.L.
      • Dardess P.
      • Maurer M.
      • Sofaer S.
      • Adams K.
      • Bechtel C.
      • Sweeney J.
      Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
      ). However, there remains a lack of consensus in the literature about the definitions of patient engagement, depending on the context in which the engagement occurs (
      • Barello S.
      • Graffigna G.
      • Vegni E.
      Patient Engagement as an emerging challenge for healthcare services: Mapping the literature.
      ;
      • Harrington R.L.
      • Hanna M.L.
      • Oehrlein E.M.
      • Camp R.
      • Wheeler R.
      • Cooblall C.
      • Perfetto E.M.
      Defining patient engagement in research: Results of a systematic review and analysis: Report of the ISPOR Patient-Centered Special Interest Group.
      ;
      • Higgins T.
      • Larson E.
      • Schnall R.
      Unraveling the meaning of patient engagement: A concept analysis.
      ). Although much work has been done to understand and promote patients' engagement in direct care (
      • Grande S.W.
      • Faber M.J.
      • Durand M.A.
      • Thompson R.
      • Elwyn G.
      A classification model of patient engagement methods and assessment of their feasibility in real-world settings.
      ;
      • Hibbard J.H.
      • Greene J.
      What the evidence shows about patient activation: Better health outcomes and care experiences; Fewer data on costs.
      ;
      • Higgins T.
      • Larson E.
      • Schnall R.
      Unraveling the meaning of patient engagement: A concept analysis.
      ) and as partners in health research (
      • Brett J.
      • Staniszewska S.
      • Mockford C.
      • Herron-Marx S.
      • Hughes J.
      • Tysall C.
      • Suleman R.
      A systematic review of the impact of patient and public involvement on service users, researchers and communities.
      ;
      • Domecq J.P.
      • Prutsky G.
      • Elraiyah T.
      • Wang Z.
      • Nabhan M.
      • Shippee N.
      • Murad M.H.
      Patient engagement in research: a systematic review.
      ;
      • Esmail L.
      • Moore E.
      • Rein A.
      Evaluating patient and stakeholder engagement in research: Moving from theory to practice.
      ;
      • Luger T.M.
      • Hamilton A.B.
      • True G.
      Measuring community-engaged research contexts, processes, and outcomes: A mapping review.
      ), there has been growing interest in understanding the potential benefits of engaging patients in quality improvement (QI) (
      • Baker G.R.
      • Fancott C.
      • Judd M.
      • O'Connor P.
      Expanding patient engagement in quality improvement and health system redesign: Three Canadian case studies.
      ;
      • Bergerum C.
      • Engström A.K.
      • Thor J.
      • Wolmesjö M.
      Patient involvement in quality improvement—A 'tug of war' or a dialogue in a learning process to improve healthcare?.
      ;
      • Bergerum C.
      • Thor J.
      • Josefsson K.
      • Wolmesjö M.
      How might patient involvement in healthcare quality improvement efforts work—A realist literature review.
      ;
      • Montreuil M.
      • Martineau J.T.
      • Racine E.
      Exploring ethical issues related to patient engagement in healthcare: Patient, clinician and researcher's perspectives.
      ;
      • Morassaei S.
      • Campbell M.
      • Di Prospero L.
      Measuring the impact of patient engagement from the perspective of health professionals leading quality improvement projects.
      ;
      • Pomey M.H.H.
      • Khalifa M.
      • Lebel P.
      • Neron A.
      • Dumez V.
      Patient partnership in quality improvement of healthcare services: Patients’' inputs and challenges faced.
      ). QI is “the combined and unceasing efforts of everyone—health care professionals, patients and their families, researchers, payers, planners and educators—to make the changes that will lead to better patient outcomes (health), better system performace (care) and better professional development (learning)” (
      • Batalden P.B.
      • Davidoff F.
      What is “quality improvement” and how can it transform healthcare?.
      ). QI activities can range from the individual behaviors providers incorporate into their daily work to improve their performance and the quality of services, to clinic-level projects to improve specific care processes, to organization- or system-wide strategic plans to promote a culture of continuous improvement (e.g., learning health care system) (
      • Duffy G.
      • McCoy K.
      • Moran J.
      • Riley W.
      The continuum of quality improvement in public health.
      ;
      • Kilbourne A.M.
      • Goodrich D.E.
      • Miake-Lye I.
      • Braganza M.Z.
      • Bowersox N.W.
      Quality enhancement research initiative implementation roadmap: Toward sustainability of evidence-based practices in a learning health system.
      ). In this context, patients may be uniquely positioned to identify issues that health care staff and administrators may miss, and to provide innovative, patient-centered solutions to problems impacting their health care.
      The organizational benefits of engaging patients in QI efforts have been identified in a growing literature base. Patient engagement in QI has shown promise in providing “real-time” information that frontline teams can use to develop improvement goals and projects (
      • Baker G.R.
      • Fancott C.
      • Judd M.
      • O'Connor P.
      Expanding patient engagement in quality improvement and health system redesign: Three Canadian case studies.
      ), positively impacting staff attitudes and motivation to implement change (
      • Boaz A.
      • Robert G.
      • Locock L.
      • Sturmey G.
      • Gager M.
      • Vougioukalou S.
      • Ziebland S.
      • Fielden J.
      What patients do and their impact on implementation.
      ;
      • Johnson K.E.
      • Mroz T.M.
      • Abraham M.
      • Figueroa Gray M.
      • Minniti M.
      • Nickel W.
      • Hsu C.
      Promoting patient and family partnerships in ambulatory care improvement: A narrative review and focus group findings.
      ), generating provider buy-in for patient-identified solutions (
      • Boaz A.
      • Robert G.
      • Locock L.
      • Sturmey G.
      • Gager M.
      • Vougioukalou S.
      • Ziebland S.
      • Fielden J.
      What patients do and their impact on implementation.
      ;
      • Liang L.
      • Cako A.
      • Urquhart R.
      • Straus S.E.
      • Wodchis W.P.
      • Baker G.R.
      • Gagliardi A.R.
      Patient engagement in hospital health service planning and improvement: a scoping review.
      ;
      • Pomey M.H.H.
      • Khalifa M.
      • Lebel P.
      • Neron A.
      • Dumez V.
      Patient partnership in quality improvement of healthcare services: Patients’' inputs and challenges faced.
      ;
      • Van C.
      • McInerney P.
      • Cooke R.
      Patients' involvement in improvement initiatives: A qualitative systematic review.
      ), and producing improvement efforts that meet patient, provider, and organizational needs and priorities (
      • Bergerum C.
      • Thor J.
      • Josefsson K.
      • Wolmesjö M.
      How might patient involvement in healthcare quality improvement efforts work—A realist literature review.
      ;
      • Morassaei S.
      • Campbell M.
      • Di Prospero L.
      Measuring the impact of patient engagement from the perspective of health professionals leading quality improvement projects.
      ). Common patient engagement strategies include soliciting patient feedback on care processes through surveys and focus groups, supporting patients’ self-initiated efforts to advocate for improvements in care, and including patients as representatives on organizational committees or as QI team members to codesign improvements (
      • Pomey M.H.H.
      • Khalifa M.
      • Lebel P.
      • Neron A.
      • Dumez V.
      Patient partnership in quality improvement of healthcare services: Patients’' inputs and challenges faced.
      ).
      Despite the growing interest in engaging patients in QI, substantial barriers to effective patient engagement persist. For example, providers may lack knowledge about how to engage with patients as equal contributors, preferring to solicit information from patients and applying the gained knowledge to QI efforts on their own (
      • Bergerum C.
      • Engström A.K.
      • Thor J.
      • Wolmesjö M.
      Patient involvement in quality improvement—A 'tug of war' or a dialogue in a learning process to improve healthcare?.
      ;
      • Liang L.
      • Cako A.
      • Urquhart R.
      • Straus S.E.
      • Wodchis W.P.
      • Baker G.R.
      • Gagliardi A.R.
      Patient engagement in hospital health service planning and improvement: a scoping review.
      ). A lack of clarity about patients’ expectations and roles in the QI efforts can similarly hinder communication and interpersonal dynamics (
      • Luger T.M.
      • Hamilton A.B.
      • True G.
      Measuring community-engaged research contexts, processes, and outcomes: A mapping review.
      ). Health care system leadership may also seem to encourage patient engagement in QI, but provide little organizational assistance (e.g., integration with workflow, protected time) to support the efforts (
      • Liang L.
      • Cako A.
      • Urquhart R.
      • Straus S.E.
      • Wodchis W.P.
      • Baker G.R.
      • Gagliardi A.R.
      Patient engagement in hospital health service planning and improvement: a scoping review.
      ). Thus, health care systems are challenged by a lack of capacity and organizational resources to engage patients as equals in QI. Patient engagement in decision-making for QI remains underexplored (
      • Sharma A.E.
      • Grumbach K.
      Engaging patients in primary care practice transformation: Theory, evidence and practice.
      ), and there is a lack of consensus on the methods of patient engagement that are most beneficial for improvement, in which settings, and by whom in the organization (
      • Bergerum C.
      • Thor J.
      • Josefsson K.
      • Wolmesjö M.
      How might patient involvement in healthcare quality improvement efforts work—A realist literature review.
      ).
      Incorporating patient preferences and experiences into QI through patient engagement may be especially impactful for patient populations requiring tailored care to meet their needs. Engaging patients who represent a minority of users within the health care system, such as those with unique health care needs or who account for disproportionate health care costs (
      • Zulman D.M.
      • O'Brien C.W.
      • Slightam C.
      • Breland J.Y.
      • Krauth D.
      • Nevedal A.L.
      Engaging high-need patients in intensive outpatient programs: A qualitative synthesis of engagement strategies.
      ), can assist health care systems’ delivery of equitable care for all patients (
      • Green A.R.
      • Tan-McGrory A.
      • Cervantes M.C.
      • Betancourt J.R.
      Leveraging quality improvement to achieve equity in health care.
      ) through the production of tailored programs or policies. In the Veterans Health Administration (VA), for example, although women veterans represent the fastest growing segment of the veteran VA users, they remain a numerical minority compared with men veterans. As an integrated health care system that has historically provided care primarily to veteran men, the VA recognized the need to invest substantial resources to make care comprehensive, high-quality, gender-sensitive, and responsive to the unique needs of women.
      As part of an ongoing system-wide improvement effort, VA policy recommends that women veterans receive both primary care (e.g., care for acute and chronic conditions) and gender-specific care (e.g., Pap smears, breast examinations, contraceptive counseling) from a women's health-trained primary care provider within a single visit (
      • Bergman A.A.
      • Frankel R.M.
      • Hamilton A.B.
      • Yano E.M.
      Challenges with delivering gender-specific and comprehensive primary care to women veterans.
      ;
      • Yano E.M.
      • Bair M.J.
      • Carrasquillo O.
      • Krein S.L.
      • Rubenstein L.V.
      Patient aligned care teams (PACT): VA's journey to implement patient-centered medical homes.
      ). VA policy also encourages the colocation of mental health care, given the higher rates of certain mental health conditions and military sexual trauma among women veterans (
      • Bergman A.A.
      • Frankel R.M.
      • Hamilton A.B.
      • Yano E.M.
      Challenges with delivering gender-specific and comprehensive primary care to women veterans.
      ), and mandates the development of a multidisciplinary Women Veterans Health Committee at each facility, with women veteran participation recommended in a consultant role (i.e., not as voting committee members), to support strategic planning and improvements for women's health (
      Veterans Health Administration (VHA)
      VHA directive 1330.01 - Health care services for women veterans. (1330.01).
      ). However, the extent to which women veteran patients are involved in QI varies across VA medical centers. A better understanding of how women veterans are engaged in QI activities could provide important insights for how VA and other integrated health care systems can engage patient populations with unique needs to better design and support improvements to their care.
      Furthermore, although many studies of patient engagement naturally focus on the perspectives of patients and their frontline clinical providers, other employees of the health care system, like managers, can play a similarly vital role in supporting QI efforts (
      • Giannitrapani K.F.
      • Rodriguez H.
      • Huynh A.K.
      • Hamilton A.B.
      • Kim L.
      • Stockdale S.E.
      • Rubenstein L.V.
      How 8middle managers facilitate interdisciplinary primary care team functioning.
      ;
      • Pannick S.
      • Sevdalis N.
      • Athanasiou T.
      Beyond clinical engagement: A pragmatic model for quality improvement interventions, aligning clinical and managerial priorities.
      ). Middle managers are those employees with whom frontline staff interact directly and frequently, but who are supervised by senior management, thus representing a link between different levels of the organization (
      • Pannick S.
      • Sevdalis N.
      • Athanasiou T.
      Beyond clinical engagement: A pragmatic model for quality improvement interventions, aligning clinical and managerial priorities.
      ). Middle managers can play a critical role in the success of QI efforts by brokering the relationship between system-level QI priorities and frontline clinical interests and helping to translate senior managers’ commitment to QI into frontline action (
      • Birken S.A.
      • Lee S.-Y.D.
      • Weiner B.J.
      Uncovering middle managers' role in healthcare innovation implementation.
      ;
      • Pannick S.
      • Sevdalis N.
      • Athanasiou T.
      Beyond clinical engagement: A pragmatic model for quality improvement interventions, aligning clinical and managerial priorities.
      ).
      In VA, women's health managers (WHMs)—including Women's Health Medical Directors, Women Veteran Program Managers, and Women's Health Care Coordinators—play this middle manager role by translating the VA's national commitment to delivering comprehensive women's health care into the frontline organization and management of VA women's health services, including QI efforts. WHMs are also often responsible for managing activities to engage women veterans within and outside the VA health care system. Because providing direct patient care is not typically the primary function of WHMs, their perspectives on patient engagement are novel and may be instrumental to informing and encouraging patient engagement in QI. Our article contributes to what is known about patient engagement in QI by exploring VA WHMs' ability to engage women veteran patients in QI efforts to improve the delivery of comprehensive women's health care.

      Methods

      Study Location

      The VA is a congressionally mandated and funded integrated health care system with a long-standing history of practice-based health services research that supports its mission to care for eligible veterans and their beneficiaries with integrity, commitment, advocacy, respect, and excellence (
      • Atkins D.
      • Kilbourne A.M.
      • Shulkin D.
      Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration.
      ;
      Department of Veterans Affairs
      Mission, vision, core values & goals.
      ). To this end, the VA's modernization efforts to become a learning health system include empowering employees to “understand and embrace change” (
      Department of Veterans Affairs
      VA modernization.
      ). Change efforts to improve care delivery have been particularly important in VA women's health. Women veterans have unique health needs and experience care differently than men veterans (
      • deKleijn M.
      • Lagro-Janssen A.L.
      • Canelo I.
      • Yano E.M.
      Creating a roadmap for delivering gender-sensitive comprehensive care for women Veterans: Results of a national expert panel.
      ;
      Women Veterans Health Strategic Health Care Group
      Gender differences in performance measures: VHA 2008-2011.
      ). To address these differences, the VA's Office of Women's Health initiated a 5-year redesign plan for women's health care in 2008 (
      • Whitehead A.M.
      • Czarnogorski M.
      • Wright S.M.
      • Hayes P.M.
      • Haskell S.G.
      Improving trends in gender disparities in the Department of Veterans Affairs: 2008-2013.
      ). Handbook 1330.01 requires that all VA medical centers and community-based outpatient clinics provide gender-sensitive primary care services, within a patient-centered medical home model, provided by trained women's health primary care providers, with care coordination for reproductive and gynecology services (
      Veterans Health Administration (VHA)
      VHA directive 1330.01 - Health care services for women veterans. (1330.01).
      ). However, the policies defining comprehensive women's health care have been implemented with varying success across VA sites (
      • deKleijn M.
      • Lagro-Janssen A.L.
      • Canelo I.
      • Yano E.M.
      Creating a roadmap for delivering gender-sensitive comprehensive care for women Veterans: Results of a national expert panel.
      ;
      • Yano E.M.
      • Haskell S.
      • Hayes P.
      Delivery of gender-sensitive comprehensive primary care to women veterans: Implications for VA Patient Aligned Care Teams.
      ). The need to address this variation and meet women veterans' needs spurred a range of ongoing and planned QI efforts.

      Data Source

      Data for this study were generated from a multisite QI effort to improve delivery of comprehensive women's health care at VA sites of care. An external contractor supported site teams, composed of WHMs and other women's health staff, to implement site-identified QI projects to meet local needs and priorities. Detailed information about the study design and how sites were identified is provided elsewhere (
      • Hamilton A.B.
      • Olmos-Ochoa T.T.
      • Canelo I.
      • Rose D.
      • Hoggatt K.J.
      • Than C.
      • Yano E.M.
      Dynamic waitlisted design for evaluating a randomized trial of evidence-based quality improvement of comprehensive women's health care implementation in low-performing VA facilities.
      ). Trained interviewers conducted 39 baseline interviews with WHMs across 21 sites (Table 1) before starting site-level QI projects. The goal of the interviews was to assess each participating site's implementation of comprehensive women's health care for the contractor to tailor additional support to each site's needs. In addition to questions about potential barriers and facilitators to implementing comprehensive women's health care, WHMs were also asked to describe their existing efforts to engage women veterans in QI. Interviews were 30–60 minutes in length, audio recorded, and professionally transcribed. Appropriate confidentiality assurances were provided to participants, and verbal consent was obtained at the start of each interview. VA designated this multisite evaluation as non-research and did not require institutional review board approval.
      Table 1Women's Health Managers (WHMs) Demographics
      WHMs (n = 39)No. (%)
      Gender
       Women35 (89.7)
       Men4 (10.3)
      Years in VA (n = 37)
       ≤25 (13.5)
       3–57 (18.9)
       6–98 (21.6)
       ≥1017 (45.9)
      Years in role (n = 38)
       <16 (15.8)
       1–211 (28.9)
       3–59 (23.7)
       6–98 (21.1)
       ≥104 (10.5)
      WHM role
       Women veteran program manager21 (53.8)
       Women's health medical director14 (35.9)
       Women's health care coordinator4 (10.3)
      Abbreviation: VA, Department of Veterans Affairs.

      Data Analysis

      Although several stakeholder engagement frameworks are available (
      • Bammer G.
      Key issues in co-creation with stakeholders when research problems are complex.
      ;
      • Boyer A.P.
      • Fair A.M.
      • Joosten Y.A.
      • Dolor R.J.
      • Williams N.A.
      • Sherden L.
      • Wilkins C.H.
      A multilevel approach to stakeholder engagement in the formulation of a clinical data research network.
      ;
      • Davis S.
      • Berkson S.
      • Gaines M.E.
      • Prajapati P.
      • Schwab W.
      • Pandhi N.
      • Edgman-Levitan S.
      Implementation science workshop: Engaging patients in team-based practice redesign — Critical reflections on program design.
      ;
      • Edwards H.
      • Huang J.
      • Jansky L.
      • Mullins C.D.
      What works when: mapping patient and stakeholder engagement methods along the ten-step continuum framework.
      ;
      International Association for Public Participation
      IAP2 spectrum review: Summary of engagement process.
      ;
      International Association for Public Participation
      IAP2 resources.
      ), we used
      • Carman K.L.
      • Dardess P.
      • Maurer M.
      • Sofaer S.
      • Adams K.
      • Bechtel C.
      • Sweeney J.
      Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
      framework to guide our analysis because of its specific focus on engagement at multiple levels of the health care system, which is an important component in QI. The framework describes patient engagement as existing along a continuum: 1) Consultation, where patient input is considered but limited; 2) Involvement, where patients may have greater access to information, but still limited decision-making power; and 3) Partnership and Shared Leadership, where patients actively share power and decision-making authority with other stakeholders. The continuum of engagement is considered across multiple levels of a health care organization, including in direct patient care, organizational design and governance, and policymaking. Varying degrees of patient engagement along the continuum may be needed within and across QI efforts based on organizational context and priorities.
      Rapid qualitative analysis methods guided our initial review of the data (
      • Bernard H.
      Research Methods in Anthropology.
      ;
      • Hamilton A.B.
      VA HSR&D Cyeberseminar spotlighting on women's health.
      ). Interview guide domains were used to create a structured site summary template that study team members used to summarize all interview data from each of the 21 sites. These summaries provided a synopsis of the characteristics and organization of women's health care at each site ahead of formal coding. A four-person coding team reviewed these structured summaries and
      • Carman K.L.
      • Dardess P.
      • Maurer M.
      • Sofaer S.
      • Adams K.
      • Bechtel C.
      • Sweeney J.
      Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
      framework domains to generate an initial list of codes with which all transcripts were coded. We used ATLAS.ti to organize our coding, which was guided by content analysis principles (
      • Cole F.L.
      Content analysis: Process and application.
      ;
      • Watkins D.C.
      Rapid and rigorous qualitative data analysis: The “RADaR” technique for applied research.
      ). The code list was iteratively refined and discrepancies in coding were resolved through consensus. After coding, the team identified and refined general themes related to WHMs' engagement of women veterans in QI.

      Results

      WHMs engaged women veterans in a variety of QI efforts ranging from simple efforts to inform better recruitment of women veterans into VA, to conducting patient satisfaction surveys to expand care delivery in their clinics, to more complex efforts aimed at improving the system-level organization of care and VA policy. Women veteran engagement in these QI efforts varied along Carman's continuum of engagement (
      • Carman K.L.
      • Dardess P.
      • Maurer M.
      • Sofaer S.
      • Adams K.
      • Bechtel C.
      • Sweeney J.
      Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
      ), in which Consultation represents the low end of engagement, Involvement the midpoint, and Partnership and Shared Leadership the high end. Figure 1 provides examples of how women veterans were engaged in QI efforts along the engagement continuum at three different levels of the organization (i.e., direct care, organizational design and governance, and policy).
      Figure thumbnail gr1
      Figure 1Women's health managers engaging women veterans in quality improvement along a continuum of patient engagement. This framework is adapted from Carman et al.’s Multidimensional Framework for Patient and Family Engagement in Health and Health Care (
      • Carman K.L.
      • Dardess P.
      • Maurer M.
      • Sofaer S.
      • Adams K.
      • Bechtel C.
      • Sweeney J.
      Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
      ). Abbreviations: VA, Veterans Health Administration; WHM, Women's Health Managers (WHMs); WVs, women veterans.

      Consultation

      At the Consultation end of the engagement continuum, WHMs primarily solicited women veterans' input in informal or passive QI efforts to improve WHM processes (e.g., outreach) and to discuss how women veterans’ individual experiences of care could be improved. WHMs described having an “open door policy” for women veterans to discuss their health, providing their contact information to women veterans, asking women veterans about their satisfaction with care during or immediately after a clinical encounter with a WH provider, and conducting automated population outreach and management through letter, phone, and email campaigns (e.g., information about upcoming events, preventive care reminders) that could result in informal conversations with women veterans about potential improvements to care. The main goal of this type of engagement was to inform improvements in direct care.
      At the organizational design and governance levels, WHMs described engagement efforts that were more structured, although still consultive in nature, focused on population- and system-level decision-making. Feedback from women veterans was elicited in several different ways, including comment and suggestion cards left in the waiting rooms and care satisfaction surveys. One site used an SMS-based system to survey pregnant women veterans about their needs; this strategy revealed that homelessness or near homelessness was a significant issue that required additional efforts from WHMs to retain these women veterans in care. A few WHMs described receiving reports from their Patient Advocate office—a VA resource where patients can file concerns related to their VA care—that were used to help triage issues to improve the quality and general experience of care for women veterans. Women veterans also had opportunities to provide feedback and suggestions directly through focus groups, panels, committees, and town halls. One WHM shared the following about her facility's town hall for women veterans:We had… a women veteran town hall a couple months ago and… [of] a couple things that were mentioned, one of them that stood out to me… was chronic pain… We have a lot of women vets that have chronic pain issues. We have a good program here I think for chronic pain but there’s nothing that’s specifically just for women.
      Another WHM described inviting more than 3,000 women veterans to her site's upcoming inaugural tele-town hall, which would allow women to conveniently participate via videoconference.
      To inform policy improvements, WHMs elicited focused information from women veterans through in-person or virtual means. For example, some WHMs involved women veterans in both women and mixed-gender patient focus groups, including groups focused on identifying strategies to improve access to care challenges and barriers to receiving VA services. At one site, a woman veteran was asked to participate on a veteran panel as part of a strategic planning meeting with facility leadership to voice her unique perspective as a woman veteran. WHMs also organized women veterans’ participation in national initiatives to impact policy, such as the national VA End Harassment Campaign, an anonymous survey-based campaign on the environment of care in VA.

      Involvement

      Moving further along the engagement continuum, WHMs shared numerous examples of engaging women veterans through Involvement. In contrast with Consultation, where WHMs engaged women veterans to elicit their perspectives, in Involvement, women veterans were engaged as more active participants in the improvement efforts. To improve their outreach efforts, for example, WHMs engaged women veterans in a peer-to-peer role by having women veterans share information and resources with other women veterans with the goal of enrolling them in VA care. Word-of-mouth communication between enrolled and nonenrolled women veterans was described as a particularly effective recruitment strategy:
      [T]he word of mouth of the veterans going out there …We have female veterans enrolled in the clinic that are more than willing to volunteer and to help their comrades.
      Similarly, WHMs sought out women veterans who were involved in community organizations (e.g., veteran service organizations) to help identify best practices for reaching nonenrolled women veterans.
      At both the organizational design and policy levels, WHMs involved women veterans as patient representatives on advisory councils and other committees to integrate their values, experiences, and perspectives. These included Women Veterans' Health Committees, Women's Health Advisory Teams, Gender Disparity Committees, and strategic planning and executive boards. Engagement of women veterans in committees and councils was intended to be ongoing and, thus, further along the continuum of engagement (Involvement) compared with other forms of engagement, such as one-time participation in focus groups (Consultation). WHMs supported women veteran participants by informing them of opportunities to participate on these committees, checking in with them about their participation, and often sitting on the same committees together. Most frequently, women were engaged in councils and committees as patient representatives to provide a patient and a woman veteran perspective. However, some women also served on committees as formal consultants (e.g., on the Executive Management Board described as the top board) to inform policy and strategic planning.

      Partnership and Shared Leadership

      WHMs described engaging women veterans at the Partnership and Shared Leadership end of the engagement continuum the least. Characterized by equal patient representation and shared decision-making authority with nonpatients, this type of engagement was limited to efforts in direct care. WHMs described engaging women veterans to colead (with WHMs) the development of women veteran programming (e.g., planning baby showers for pregnant women veteran patients and peer-to-peer health education) with the goal of engaging other patients and improving satisfaction with care. In addition, WHMs also engaged women veterans through existing VA partnerships with community organizations, as well as local, state, and national government committees and events. Women veteran members of committees and other groups external to VA often partnered with WHMs to inform improvements in WHMs' outreach to women veterans not enrolled in VA. One WHM described a community women veterans' group she partnered with as “a conduit for communication with the women veterans’ population” outside VA. At the organizational design and policy levels, it is possible that some of the women veterans who participated in committees were engaged in shared leadership, but this was not confirmed.

      WHMs’ Perceived Benefits of and Challenges to Engaging Patients in QI

      WHMs identified multiple benefits to involving women veterans in QI, including being able to incorporate the patient perspective (through Consultation) into improvement efforts ranging from simple to complex at multiple levels of the VA system. WHMs described some women veterans as facilitating engagement efforts by approaching WHMs in the facility or in the community to provide feedback about their care experiences.
      At the Involvement level, some WHMs engaged women veterans on committees where they participated as consultants and advocates. One WHM shared that engaging women veterans in the Veterans Advisory Council, which was attended by the facility director, brought about needed changes and motivated the women veterans involved to continue their participation:The veterans that are on [the Council] have made it clear to other veterans that this is to come and make suggestions, not complaints. And the director at that time… she would attend as well once a month and any recommendations that she could implement she would implement them, which has made the veterans really excited about that Council…
      Another WHM shared that when she had difficulty communicating to VA leadership or gaining their support for a needed women's health improvement, she regularly consulted with and relied on a group of women veterans who were willing to attend leadership meetings to “champion some of those ideas,” with some success.
      Despite numerous examples of fruitful engagement activities across a spectrum of QI efforts, WHMs also experienced significant challenges to patient engagement, including unsystematic collection of feedback from women veterans (e.g., through comment cards and open door policies), difficulty identifying women veterans to recruit for planned women's health events, and inconsistent participation from women veterans on key committees. WHMs shared that the women veteran patients they engaged often cited work, school, and childcare needs as barriers to engagement and participation, particularly on committees. WHMs often relied on VA employees who were women veteran patients to fill the role of patient representatives on organizational committees when nonemployee women veterans were unavailable. Women veteran employees were more consistently available during committee hours and often worked in the same location that the meetings were held, facilitating their involvement.

      Discussion

      WHMs at primary care VA clinics described their efforts to engage women veterans at various levels of the health care organization to address organizational priorities and support QI efforts. Although not their primary task, WHMs described women veteran engagement as both important and feasible. Using
      • Carman K.L.
      • Dardess P.
      • Maurer M.
      • Sofaer S.
      • Adams K.
      • Bechtel C.
      • Sweeney J.
      Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
      engagement continuum to interpret our results, we found that WHMs were consistently able to engage patients in Consultation (i.e., solicitation of patient input) by asking women veterans to provide feedback about their satisfaction with care. Some WHMs were able to foster greater patient Involvement (i.e., greater access to information, some decision-making power), such as by engaging women veterans in recruitment efforts for women's health events. WHMs provided few examples of patient engagement in Partnership and Shared Leadership (i.e., shared decision-making power), apart from WHMs and women veterans at times coleading patient-facing programming such as baby showers. Although patient engagement is feasible in VA clinics, greater participation of women veterans beyond Consultation may require additional supports for WHMs and women veterans. Patients bring unique insights that may catalyze innovative strategies and models to deliver high quality and responsive care, which can be especially important for patient populations with unique needs. Additionally, there may be unanticipated benefits to health care policy from engaging patients beyond Consultation and outside of their direct care experiences (
      • Conklin A.
      • Morris Z.
      • Nolte E.
      What is the evidence base for public involvement in health-care policy? Results of a systematic scoping review.
      ;
      • Luger T.M.
      • Hamilton A.B.
      • True G.
      Measuring community-engaged research contexts, processes, and outcomes: A mapping review.
      ). As a learning health care system with a growing population of women veteran patients, the VA is likely to benefit from greater engagement of women veterans across the organization to better design and tailor care their care.
      In VA, WHMs experienced significant challenges to patient engagement, including the unsystematic collection of feedback from women veterans, difficulty identifying women veterans to recruit for planned women's health events, and inconsistent participation from women veterans in key committees. Of note, as a workaround, some sites included VA women employees who were veterans themselves and received their care in VA as patient representatives on committees. Although veteran employee participation certainly constitutes patient engagement (because many VA employees also receive their care from the system), as insiders, VA employees are likely to have greater familiarity with and access to information about the health care system than nonemployee veteran patients. Although this insider status may facilitate staff participation in committees (e.g., available during business hours, protected time to participate), it may also inform their motivation for participating, and they may express different needs than nonemployee patients. These constraints surrounding nonemployee patient participation in committees may partially explain why WHMs provided few examples of engagement at the Partnership and Shared Leadership level. In addition, the hierarchical organization of the VA, which can concentrate decision-making power at the top, can also limit opportunities for collaboration at the highest stages of engagement (
      • Hamilton A.B.
      • Yano E.M.
      The importance of symbolic and engaged participation in evidence-based quality improvement in a complex integrated healthcare system: Response to "The science of stakeholder engagement in research".
      ). Nonetheless, several WHMs shared examples of how women veterans were engaged at higher levels, such as consultants on strategic planning committees. However, not all QI efforts warrant greater patient engagement, or even direct patient engagement, for the engagement to be appropriate and meaningful. For some patients (e.g., patients with dementia), it may be more appropriate to engage their caregivers or other patient advocates, who may be well-positioned to translate what they learn from patients into recommendations for improvement.
      The benefits of effective patient engagement have been previously explored (
      • Carman K.L.D.
      • Maurer M.E.
      • Workman T.
      • Pathak-Sen E.
      A roadmap for patient and family engagement in healthcare practice and research.
      ;
      • Ponte P.R.
      • Conlin G.
      • Conway J.B.
      • Grant S.
      • Medeiros C.
      • Nies J.
      • Conley K.
      Making patient-centered care come alive: Achieving full integration of the patient's perspective.
      ;
      • Simmons L.A.
      • Wolever R.Q.
      • Bechard E.M.
      • Snyderman R.
      Patient engagement as a risk factor in personalized health care: A systematic review of the literature on chronic disease.
      ). Our results support previous findings that describe patient involvement on key health care system committees as beneficial for both patients and the health care organization (
      • Ponte P.R.
      • Conlin G.
      • Conway J.B.
      • Grant S.
      • Medeiros C.
      • Nies J.
      • Conley K.
      Making patient-centered care come alive: Achieving full integration of the patient's perspective.
      ). One WHM described women veterans as advocates for women's health and relied on them to stress the importance of certain women's health priorities when attending meetings with leadership. Several WHMs also described local town hall meetings that facility leaders attended to hear directly from veterans about their needs. Thus, women veterans' direct communication with facility leadership about issues affecting their care can enable shared advocacy with WHMs. Involvement in QI and organizational decision-making may also give women veterans a greater opportunity to amplify their voices to leadership. Because women veterans are a numerical minority in a health care system that predominantly serves men, engaging them meaningfully in QI could advance more equitable care transformation that meets all veterans' needs. Greater transparency about how the information shared at town halls will be used and how patients might further participate in decision-making may be needed to achieve the shared power and responsibility that represents exemplary patient engagement (
      • Hamilton A.B.
      • Yano E.M.
      The importance of symbolic and engaged participation in evidence-based quality improvement in a complex integrated healthcare system: Response to "The science of stakeholder engagement in research".
      ) and is foundational to learning health care systems (
      • Kass N.E.
      • Faden R.R.
      Ethics and learning health care: The essential roles of engagement, transparency, and accountability.
      ).

      Limitations

      Several study limitations should be considered when interpreting results. Data were gathered at baseline before the start of the VA project to improve delivery of comprehensive women's health care. Thus, interview questions focused on the ways in which participants engaged women veterans rather than the content and valence of feedback gathered, how feedback was used, or the specific nature of patient participation in committees and other activities. As a result, WHMs' responses remained largely descriptive. Without a more detailed understanding of how women veterans' feedback was used, it is difficult to gauge how the range of patient engagement reported by WHMs resulted in or informed improvements. Further inquiry into how the facilities used feedback from women veterans is needed to identify best practices for patient engagement in QI and to inform patient-endorsed improvements in the implementation of comprehensive care. Nonetheless, participants were from facilities of different sizes and diverse geographic locations, suggesting that the themes identified were robust across sites.

      Implications for Practice and/or Policy

      To better understand patients’ engagement beyond Consultation, the perspectives of health care personnel responsible for the management of care are imperative. Managers are uniquely positioned to address patient engagement at multiple levels of the organization given their role of translating leadership priorities and policies into the processes that organize the delivery of clinical care on the ground. This finding suggests that managers are poised to play a critical role in organizing and supporting implementation of changes that translate what is learned from patients through various forms of engagement into improvements to care delivery and organizational governance.
      Given the volume of resources needed to fully engage patients as well as the challenges patients face in finding time to participate in engagement activities, patient engagement may be understandably and particularly challenging beyond Consultation at higher levels along the engagement continuum. Our findings suggest that patient engagement in QI, although restricted primarily to Consultation and Involvement with limited decision-making, is feasible and viewed as beneficial by managers. Possible constraints for patient participation like transportation and compensated time should be addressed to promote participation of patients that represent a minority within the health care system. Additionally, given the context of the coronavirus disease 2019 pandemic and the increased challenges of holding in-person gatherings, opportunities for virtual patient engagement may be better supported by the organization and potentially more familiar to patients.
      Additionally, providing managers with context-specific patient engagement training may also improve their ability to engage patients beyond Consultation and along the engagement continuum as appropriate. There is existing evidence that providing health professionals with a structured curriculum focused on patient engagement concepts and approaches is effective in supporting patient engagement activities and promoting their importance (
      • Morassaei S.
      • Campbell M.
      • Di Prospero L.
      Measuring the impact of patient engagement from the perspective of health professionals leading quality improvement projects.
      ). Training managers in methods to best identify when and how to engage patients may be beneficial and providing guidelines for engagement, such as an adaptation of the International Association for Public Participation spectrum (
      • Bammer G.
      Key issues in co-creation with stakeholders when research problems are complex.
      ;
      International Association for Public Participation
      IAP2 spectrum review: Summary of engagement process.
      ;
      International Association for Public Participation
      IAP2 resources.
      ), could help managers to make informed decisions regarding the most appropriate level and form of patient engagement. Furthermore, standardizing effective processes so that they are a clear component of the manager's role and integrated into existing workflows may help increase managers' efficiency and impact surrounding patient engagement efforts, thereby helping to decrease managers' work burden and, ultimately, allowing for richer patient engagement that truly transforms the health care system.

      Conclusions

      Effectively integrating patient feedback into organizational design and governance is pertinent for any health care system interested in improving the health and health care experiences of its patient population. Our findings suggest that managers, although uniquely positioned to engage patients at multiple levels within health care organizations, may require additional supports and training to effectively engage patients across the engagement continuum. To better realize the goals of patient engagement in QI, further research is needed to gain a better understanding of contextual factors influencing engagement, the ways in which patient engagement results in or informs QI, and possible best practices. Such research could inform guidelines to help facilities achieve effective patient engagement within their resource- and regulation-constrained environments.

      Acknowledgments

      The views expressed in this article are those of the authors and do not reflect the position or policy of the Department of Veterans Affairs or the United States government. We wish to acknowledge the support of VA Women's Health Services, Atlas Research Inc., and the participating VA women's health clinic site leads for their significant contributions to this work. We would also like to thank Chloe E. Bird, PhD, Senior Sociologist, RAND Corporation, for reviewing an early version of this manuscript (funded through CIN 13–417).

      References

        • Atkins D.
        • Kilbourne A.M.
        • Shulkin D.
        Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration.
        Annual Review of Public Health. 2017; 38: 467-487
        • Baker G.R.
        • Fancott C.
        • Judd M.
        • O'Connor P.
        Expanding patient engagement in quality improvement and health system redesign: Three Canadian case studies.
        Healthc Manage Forum. 2016; 29: 176-182
        • Bammer G.
        Key issues in co-creation with stakeholders when research problems are complex.
        Evidence & Policy: A Journal of Research, Debate and Practice. 2019; 15: 423-435
        • Barello S.
        • Graffigna G.
        • Vegni E.
        Patient Engagement as an emerging challenge for healthcare services: Mapping the literature.
        Nursing Research and Practice. 2012; 2012: 905934
        • Batalden P.B.
        • Davidoff F.
        What is “quality improvement” and how can it transform healthcare?.
        Quality and Safety in Health Care. 2007; 16: 2-3
        • Bergerum C.
        • Engström A.K.
        • Thor J.
        • Wolmesjö M.
        Patient involvement in quality improvement—A 'tug of war' or a dialogue in a learning process to improve healthcare?.
        BMC Health Services Research. 2020; 20: 1115
        • Bergerum C.
        • Thor J.
        • Josefsson K.
        • Wolmesjö M.
        How might patient involvement in healthcare quality improvement efforts work—A realist literature review.
        Health Expect. 2019; 22: 952-964
        • Bergman A.A.
        • Frankel R.M.
        • Hamilton A.B.
        • Yano E.M.
        Challenges with delivering gender-specific and comprehensive primary care to women veterans.
        Womens Health Issues. 2015; 25: 28-34
        • Bernard H.
        Research Methods in Anthropology.
        5th ed. AltaMira Press, Lanham, MD2011
        • Birken S.A.
        • Lee S.-Y.D.
        • Weiner B.J.
        Uncovering middle managers' role in healthcare innovation implementation.
        Implementation Science. 2012; 7: 28
        • Boaz A.
        • Robert G.
        • Locock L.
        • Sturmey G.
        • Gager M.
        • Vougioukalou S.
        • Ziebland S.
        • Fielden J.
        What patients do and their impact on implementation.
        Journal of Health Organization and Management. 2016; 30 (AltaMira Press 278): 258
        • Boyer A.P.
        • Fair A.M.
        • Joosten Y.A.
        • Dolor R.J.
        • Williams N.A.
        • Sherden L.
        • Wilkins C.H.
        A multilevel approach to stakeholder engagement in the formulation of a clinical data research network.
        Medical Care. 2018; 56: S22-S26
        • Brett J.
        • Staniszewska S.
        • Mockford C.
        • Herron-Marx S.
        • Hughes J.
        • Tysall C.
        • Suleman R.
        A systematic review of the impact of patient and public involvement on service users, researchers and communities.
        Patient. 2014; 7: 387-395
        • Carman K.L.
        • Dardess P.
        • Maurer M.
        • Sofaer S.
        • Adams K.
        • Bechtel C.
        • Sweeney J.
        Patient and family engagement: A framework for understanding the elements and developing interventions and policies.
        Health Affairs (Millwood). 2013; 32: 223-231
        • Carman K.L.D.
        • Maurer M.E.
        • Workman T.
        • Pathak-Sen E.
        A roadmap for patient and family engagement in healthcare practice and research.
        (Available:)
        • Cole F.L.
        Content analysis: Process and application.
        Clinical Nurse Specialist CNS. 1988; 2: 53-57
        • Conklin A.
        • Morris Z.
        • Nolte E.
        What is the evidence base for public involvement in health-care policy? Results of a systematic scoping review.
        Health Expectations. 2015; 18: 153-165
        • Davis S.
        • Berkson S.
        • Gaines M.E.
        • Prajapati P.
        • Schwab W.
        • Pandhi N.
        • Edgman-Levitan S.
        Implementation science workshop: Engaging patients in team-based practice redesign — Critical reflections on program design.
        Journal of General Internal Medicine. 2016; 31: 688-695
        • deKleijn M.
        • Lagro-Janssen A.L.
        • Canelo I.
        • Yano E.M.
        Creating a roadmap for delivering gender-sensitive comprehensive care for women Veterans: Results of a national expert panel.
        Medical Care. 2015; 53: S156-S164
        • Department of Veterans Affairs
        Mission, vision, core values & goals.
        (Available:)
        https://www.va.gov/about_va/mission.asp
        Date: 2015
        Date accessed: March 10, 2022
        • Department of Veterans Affairs
        VA modernization.
        (Available:)
        www.va.gov/modernization/
        Date: 2019
        Date accessed: March 10, 2022
        • Domecq J.P.
        • Prutsky G.
        • Elraiyah T.
        • Wang Z.
        • Nabhan M.
        • Shippee N.
        • Murad M.H.
        Patient engagement in research: a systematic review.
        BMC Health Services Research. 2014; 14: 89
        • Duffy G.
        • McCoy K.
        • Moran J.
        • Riley W.
        The continuum of quality improvement in public health.
        Quality Management Forum. 2010; 35: 1
        • Edwards H.
        • Huang J.
        • Jansky L.
        • Mullins C.D.
        What works when: mapping patient and stakeholder engagement methods along the ten-step continuum framework.
        Journal of Comparative Effectiveness Research. 2021; 10: 999-1017
        • Esmail L.
        • Moore E.
        • Rein A.
        Evaluating patient and stakeholder engagement in research: Moving from theory to practice.
        Journal of Comparative Effectiveness Research. 2015; 4: 133-145
        • Giannitrapani K.F.
        • Rodriguez H.
        • Huynh A.K.
        • Hamilton A.B.
        • Kim L.
        • Stockdale S.E.
        • Rubenstein L.V.
        How 8middle managers facilitate interdisciplinary primary care team functioning.
        Healthcare. 2019; 7: 10-15
        • Grande S.W.
        • Faber M.J.
        • Durand M.A.
        • Thompson R.
        • Elwyn G.
        A classification model of patient engagement methods and assessment of their feasibility in real-world settings.
        Patient Education and Counseling. 2014; 95: 281-287
        • Green A.R.
        • Tan-McGrory A.
        • Cervantes M.C.
        • Betancourt J.R.
        Leveraging quality improvement to achieve equity in health care.
        Joint Commission Journal on Quality and Patient Safety. 2010; 36: 435-442
        • Hamilton A.B.
        VA HSR&D Cyeberseminar spotlighting on women's health.
        in: Cyberseminar: Qualitative methods in rapid turn-around health services research. 2013 (Available:)
        • Hamilton A.B.
        • Olmos-Ochoa T.T.
        • Canelo I.
        • Rose D.
        • Hoggatt K.J.
        • Than C.
        • Yano E.M.
        Dynamic waitlisted design for evaluating a randomized trial of evidence-based quality improvement of comprehensive women's health care implementation in low-performing VA facilities.
        Implementation Science Communications. 2020; 1: 59
        • Hamilton A.B.
        • Yano E.M.
        The importance of symbolic and engaged participation in evidence-based quality improvement in a complex integrated healthcare system: Response to "The science of stakeholder engagement in research".
        Translational Behavioral Medicine. 2017; 7: 492-494
        • Harrington R.L.
        • Hanna M.L.
        • Oehrlein E.M.
        • Camp R.
        • Wheeler R.
        • Cooblall C.
        • Perfetto E.M.
        Defining patient engagement in research: Results of a systematic review and analysis: Report of the ISPOR Patient-Centered Special Interest Group.
        Value in Health. 2020; 23: 677-688
        • Hibbard J.H.
        • Greene J.
        What the evidence shows about patient activation: Better health outcomes and care experiences; Fewer data on costs.
        Health Affairs (Millwood). 2013; 32: 207-214
        • Higgins T.
        • Larson E.
        • Schnall R.
        Unraveling the meaning of patient engagement: A concept analysis.
        Patient Education and Counseling. 2017; 100: 30-36
        • International Association for Public Participation
        IAP2 spectrum review: Summary of engagement process.
        (Available:)
        • International Association for Public Participation
        IAP2 resources.
        (Available:)
        https://www.iap2.org/page/resources
        Date: 2022
        Date accessed: March 10, 2022
        • Johnson K.E.
        • Mroz T.M.
        • Abraham M.
        • Figueroa Gray M.
        • Minniti M.
        • Nickel W.
        • Hsu C.
        Promoting patient and family partnerships in ambulatory care improvement: A narrative review and focus group findings.
        Advances in Therapy. 2016; 33: 1417-1439
        • Kass N.E.
        • Faden R.R.
        Ethics and learning health care: The essential roles of engagement, transparency, and accountability.
        Learning Health Systems. 2018; 2: e10066
        • Kilbourne A.M.
        • Goodrich D.E.
        • Miake-Lye I.
        • Braganza M.Z.
        • Bowersox N.W.
        Quality enhancement research initiative implementation roadmap: Toward sustainability of evidence-based practices in a learning health system.
        Medical Care. 2019; 57: S286-S293
        • Liang L.
        • Cako A.
        • Urquhart R.
        • Straus S.E.
        • Wodchis W.P.
        • Baker G.R.
        • Gagliardi A.R.
        Patient engagement in hospital health service planning and improvement: a scoping review.
        BMJ Open. 2018; 8: e018263
        • Luger T.M.
        • Hamilton A.B.
        • True G.
        Measuring community-engaged research contexts, processes, and outcomes: A mapping review.
        Milbank Quarterly. 2020; 98: 493-553
        • Montreuil M.
        • Martineau J.T.
        • Racine E.
        Exploring ethical issues related to patient engagement in healthcare: Patient, clinician and researcher's perspectives.
        Journal of Bioethical Inquiry. 2019; 16: 237-248
        • Morassaei S.
        • Campbell M.
        • Di Prospero L.
        Measuring the impact of patient engagement from the perspective of health professionals leading quality improvement projects.
        Journal of Continuing Education in the Health Professions. 2021; 41: 247-252
        • Pannick S.
        • Sevdalis N.
        • Athanasiou T.
        Beyond clinical engagement: A pragmatic model for quality improvement interventions, aligning clinical and managerial priorities.
        BMJ Quality & Safety. 2016; 25: 716-725
        • Pomey M.H.H.
        • Khalifa M.
        • Lebel P.
        • Neron A.
        • Dumez V.
        Patient partnership in quality improvement of healthcare services: Patients’' inputs and challenges faced.
        Patient Experience. 2015; 2: 6
        • Ponte P.R.
        • Conlin G.
        • Conway J.B.
        • Grant S.
        • Medeiros C.
        • Nies J.
        • Conley K.
        Making patient-centered care come alive: Achieving full integration of the patient's perspective.
        Journal of Nursing Administration. 2003; 33: 82-90
        • Sharma A.E.
        • Grumbach K.
        Engaging patients in primary care practice transformation: Theory, evidence and practice.
        Family Practice. 2017; 34: 262-267
        • Simmons L.A.
        • Wolever R.Q.
        • Bechard E.M.
        • Snyderman R.
        Patient engagement as a risk factor in personalized health care: A systematic review of the literature on chronic disease.
        Genome Medicine. 2014; 6: 16
        • Van C.
        • McInerney P.
        • Cooke R.
        Patients' involvement in improvement initiatives: A qualitative systematic review.
        JBI Database of Systematic Reviews and Implementation Reports. 2015; 13: 232-290
        • Veterans Health Administration (VHA)
        VHA directive 1330.01 - Health care services for women veterans. (1330.01).
        Author, Washington, DC2017
        • Watkins D.C.
        Rapid and rigorous qualitative data analysis: The “RADaR” technique for applied research.
        International Journal of Qualitative Methods. 2017; 16 (1609406917712131)
        • Whitehead A.M.
        • Czarnogorski M.
        • Wright S.M.
        • Hayes P.M.
        • Haskell S.G.
        Improving trends in gender disparities in the Department of Veterans Affairs: 2008-2013.
        American Journal of Public Health. 2014; 104: S529-S531
        • Women Veterans Health Strategic Health Care Group
        Gender differences in performance measures: VHA 2008-2011.
        (Available:)
        www.womenshealth.va.gov
        Date: 2012
        Date accessed: March 10, 2022
        • Yano E.M.
        • Bair M.J.
        • Carrasquillo O.
        • Krein S.L.
        • Rubenstein L.V.
        Patient aligned care teams (PACT): VA's journey to implement patient-centered medical homes.
        Journal of General Internal Medicine. 2014; 29: S547-S549
        • Yano E.M.
        • Haskell S.
        • Hayes P.
        Delivery of gender-sensitive comprehensive primary care to women veterans: Implications for VA Patient Aligned Care Teams.
        Journal of General Internal Medicine. 2014; 29: S703-S707
        • Zulman D.M.
        • O'Brien C.W.
        • Slightam C.
        • Breland J.Y.
        • Krauth D.
        • Nevedal A.L.
        Engaging high-need patients in intensive outpatient programs: A qualitative synthesis of engagement strategies.
        Journal of General Internal Medicine. 2018; 33: 1937-1944

      Biography

      Tanya T. Olmos-Ochoa, PhD, MPH, is an Investigator at the Center for the Study of Healthcare Innovation, Implementation, and Policy at VA Greater Los Angeles. Her work focuses on improving the organization and delivery of care to address healthcare disparities and better support the healthcare workforce.
      Tana M. Luger, PhD, is the Director of Research and Analytics for Covenant Health Network, a non-profit group of post-acute health care providers. Her work focuses on the application of patient-facing and clinical informatics to optimize health care quality and population health.
      Anneka Oishi, BA, is a Research Assistant at the VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy.
      Karen E. Dyer, PhD, is a Research Health Scientist at the VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy.
      Annie Sumberg, MPH, is a Research Health Scientist at the VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy.
      Ismelda Canelo, MPA, is a Health Science Specialist, VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy. She has 20 years of experience managing VA primary care and women's health organizational and implementation research.
      Theodore K. Gideonse, PhD, is an Assistant Professor in the Department of Health, Society, and Behavior and Director of the Undergraduate Program for University of California, Irvine's Program in Public Health.
      Ann Cheney, PhD, is an Associate Professor in the Department of Social Science Population and Public Health. Her work focuses on structural inequality in vulnerable populations, including immigrant, veteran, and rural populations.
      Elizabeth M. Yano, PhD, MSPH, is Director of the VA HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP) and Director of the Women’s Health Research Network Consortium, and Adjunct Professor in Public Health and Medicine at UCLA.
      Alison B. Hamilton, PhD, MPH, is Chief Officer of Implementation & Policy, VA HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy (CSHIIP), and Professor-in-Residence in the Department of Psychiatry and Biobehavioral Sciences, UCLA David Geffen School of Medicine.