Advertisement

Evidence Map: Reporting of Results by Sex or Gender in Randomized, Controlled Trials with Women Veteran Participants (2008 to 2018)

  • Elisheva R. Danan
    Correspondence
    Correspondence to: Elisheva R. Danan, MD, MPH, Center for Care Delivery and Outcomes Research, Minneapolis VA Health Care System, 1 Veterans Drive (152), Minneapolis, MN 55417. Phone: (612) 467-1979; fax: (612) 727-5699.
    Affiliations
    VA HSR&D Center for Care Delivery and Outcomes Research, Minneapolis VA Health Care System, Minneapolis, Minnesota

    Department of Medicine, University of Minnesota Medical School, Minneapolis, Minnesota
    Search for articles by this author
  • Kristen Ullman
    Affiliations
    VA HSR&D Center for Care Delivery and Outcomes Research, Minneapolis VA Health Care System, Minneapolis, Minnesota
    Search for articles by this author
  • Ruth S. Klap
    Affiliations
    VA HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, California
    Search for articles by this author
  • Elizabeth M. Yano
    Affiliations
    VA HSR&D Center for the Study of Healthcare Innovation, Implementation & Policy, VA Greater Los Angeles Healthcare System, Los Angeles, California

    Department of Health Policy and Management, UCLA Fielding School of Public Health, Los Angeles, California
    Search for articles by this author
  • Erin E. Krebs
    Affiliations
    VA HSR&D Center for Care Delivery and Outcomes Research, Minneapolis VA Health Care System, Minneapolis, Minnesota

    Department of Medicine, University of Minnesota Medical School, Minneapolis, Minnesota
    Search for articles by this author

      Abstract

      Background

      Higher participation of women in randomized, controlled trials (RCTs) has not led to significantly improved reporting of sex-stratified results. A recent evidence map of research on women veterans revealed that many studies did not report results by sex or gender. This study's objectives were to compare characteristics of RCTs with women veteran participants that did or did not report results by sex or gender and to assess how sex and gender are addressed in research with women veterans.

      Methods

      We extended the prior evidence map with a systematic search for RCTs with women veterans, published between 2008 and 2018. We compared the characteristics of RCTs that reported results by sex or gender with those of RCTs that did not, and reviewed methodology and reporting of sex/gender analyses.

      Results

      In addition to 11 studies from the prior evidence map, we assessed 1,820 abstracts for relevance and ultimately included 45 unique RCTs. Five trials included only women and 40 included both men and women (median, 14.3% women). Ten studies reported results by sex or gender. These trials were larger (median study size of n = 343.5 vs. n = 125.5) and included a higher median proportion of women participants (16.8% vs. 11.2%) than studies without sex/gender results. Ten of 11 trials that tested pharmacologic or device interventions did not report results by sex or gender.

      Conclusions

      Reporting of results by sex or gender remains low in veteran research, but may improve with larger studies and increased recruitment of women veterans into trials. Trials of pharmacologic or device interventions may be targets for future reporting requirements. Standardization could improve attention to sex and gender in methodology and reporting.

       Rationale

      Increased attention to the inclusion of women in medical research by the National Institutes of Health ([
      National Institutes of Health (NIH)
      NIH policy and guidelines on the inclusion of women and minorities as subjects in clinical research (amendment to the 1994 Federal Register).
      ,
      U.S. Department of Health and Human Services
      NIH guidelines on the inclusion of women and minorities as subjects in clinical research.
      ) and the National Academy of Medicine (
      Institute of Medicine (US) Committee on Understanding the Biology of Sex and Gender Differences.
      ) has led to higher participation by women in clinical trials (
      Office of Research on Women’s Health
      Report of the advisory committee on research on women's health, fiscal years 2015-2016: Office of Research on Women s Health and NIH support for research on women’s health.
      ). Despite these advances, most published research results are not disaggregated by sex or gender (
      • Avery E.
      • Clark J.
      Sex-related reporting in randomised controlled trials in medical journals.
      ,
      • Phillips S.P.
      • Hamberg K.
      Doubly blind: A systematic review of gender in randomised controlled trials.
      ). Grouping male and female study subjects together without attention to potential sex and gender differences can lead to overgeneralization of study results (
      • Clayton J.A.
      • Tannenbaum C.
      Reporting sex, gender, or both in clinical research?.
      ). This practice can overestimate the benefits and underestimate the harms for male or female patients due to biologic (sex) or sociocultural (gender) differences in response to the intervention.
      The challenge of conducting research that is meaningfully applicable to women is intensified for studies related to veterans’ health. Although women are the fastest growing segment of the Veterans Health Administration (VA) population, they represent an extreme minority of living veterans (10%) and VA users (7%;
      • Frayne S.M.
      • Phibbs C.S.
      • Saechao F.
      • Maisel N.C.
      • Friedman S.A.
      • Finlay A.
      • Berg E.
      • Balasubramanian V.
      • Dally S.K.
      • Ananth L.
      • Romodan Y.
      • Lee J.
      • Iqbal S.
      • Hayes P.M.
      • Zephyrin L.
      • Whitehead A.
      • Torgal A.
      • Katon J.G.
      • Haskell S.
      Sourcebook: Women Veterans in the Veterans Health Administration. Volume 3. Sociodemographics, Utilization, Costs of Care, and Health Profile.
      ). Women veterans are projected to comprise 15.9% of the veteran population by 2040 (
      Department of Veterans Affairs, Office of Data Governance and Analytics
      Veteran population projection model (VetPop). From Veterans Benefits Administration; Veterans Health Administration, Office of the Assistant Deputy Under Secretary for Health for Policy and Planning.
      ), so high-quality research related to the health and health care of women veterans will be critical for evidence-based practice.
      In our recent evidence map of women veterans’ health and health care research published from 2008 to 2015, we found a significant increase in the amount of research conducted with women veteran participants over time (
      • Danan E.
      • Ensrud K.
      • Krebs E.
      • Koeller E.
      • Greer N.
      • Velasquez T.
      • Wilt T.
      An evidence map of the women veterans’ health research literature (2008–2015).
      ,
      • Danan E.
      • Krebs E.
      • Ensrud K.
      • Koeller E.
      • MacDonald R.
      • Velasquez T.
      • Wilt T.
      An evidence map of the women veterans’ health research literature (2008–2015).
      ). In that report, we included 440 articles that reported results by sex or gender, including 8 randomized, controlled trials (RCTs). We excluded 424 articles that included women veterans but did not report results by sex or gender or included very few women; as a result, we were unable to examine study characteristics associated with reporting results by sex or gender.

       Objectives

      The main purpose of the current study was to compare characteristics of RCTs that included women veterans and did or did not report results by sex or gender (Q1). We focused on RCTs due to the importance of intervention research in advancing women veterans’ health. Secondary goals were to appraise how studies with sex/gender results adhere to currently proposed guidelines or best practices (Q2) and to assess how studies without sex/gender results address sex or gender, if at all (Q3). We aimed to identify targets for researchers and policy-makers to improve the quality of future research involving women veterans. Our key questions were:
      • Q1: How do RCTs that include women veterans and report results by sex or gender differ from RCTs that include women veterans but do not report results by sex or gender?
      • Q2: Among RCTs with women veterans that report results by sex or gender, do sex/gender analyses and reporting adhere to currently proposed best practices?
      • Q3: Among RCTs with women veterans that do not report results by sex or gender, how are sex/gender addressed in publications, if at all?

      Methods

      We used systematic review procedures to create a targeted evidence map comparing characteristics of RCTs published from 2008 to 2018 that included women veterans and did or did not report results by sex or gender. A recent systematic review identified 5 key characteristics of evidence maps: 1) systematic process, 2) capture a broad field, 3) identify gaps or future research needs, and 4) produce a user-friendly product with a 5) visual depiction (
      • Miake-Lye I.M.
      • Hempel S.
      • Shanman R.
      • Shekelle P.G.
      What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products.
      ). Although initially developed in parallel, evidence maps are increasingly considered a type of scoping review, for which a new checklist (the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews, or PRISMA-ScR) can be applied (
      • Tricco A.C.
      • Lillie E.
      • Zarin W.
      • O'Brien K.K.
      • Colquhoun H.
      • Levac D.
      • Straus S.E.
      PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation.
      ). We report this review according to the PRISMA-ScR checklist.

       Protocol, Information Sources, and Search Strategy

      The current article is a focused extension of a recent systematic review, for which we created a broad evidence map of the literature related to women veterans' health and health care from 2008 to 2015. Full methods for the prior review have been described (
      • Danan E.
      • Ensrud K.
      • Krebs E.
      • Koeller E.
      • Greer N.
      • Velasquez T.
      • Wilt T.
      An evidence map of the women veterans’ health research literature (2008–2015).
      ,
      • Danan E.
      • Krebs E.
      • Ensrud K.
      • Koeller E.
      • MacDonald R.
      • Velasquez T.
      • Wilt T.
      An evidence map of the women veterans’ health research literature (2008–2015).
      ). Briefly, we searched MEDLINE (Ovid), CINAHL, and the VA Health Services Research and Development database for articles published between January 2008 and December 2015 using the MeSH terms: Women; Women's Health; Women's Health Services; Veterans; Veterans Health; and Hospitals, Veterans. That review included 440 articles that provided results by sex or gender, of which 8 were RCTs and 12 were secondary analyses of RCTs (3 of which described trials with male and female participants). The prior review identified and excluded 424 articles related to women veterans' health that did not provide results by sex or gender or included a very small proportion of women veterans. These 424 previously excluded articles formed the basis for the current review. For the current review, we also updated our results by repeating the identical MEDLINE search for articles published January 2015 to May 2018 (Appendix 1).

       Eligibility Criteria

      The current review differed from the prior review by including only RCTs, excluding studies in which participants were VA clinicians or clinics, and eliminating the minimum threshold for number of women veteran participants. Other study eligibility criteria were unchanged from the prior review. We excluded studies that were not related to health/health care, did not include women U.S. veterans, or included only active duty military. Studies with less than 75% veterans (e.g., multisite trials with one VA site) were excluded if they did not report results stratified by veteran status.

       Selection of Sources of Evidence

      Abstracts were dual-reviewed for exclusion criteria by an investigator (E.D.) and research associate (K.U.). Agreement on exclusion was 97% (Cohen's κ = 0.81) and conflicts were resolved by consensus. Trials or interventions that were uncontrolled (single-arm) or used nonrandom treatment assignment were identified (using abstracts or full text if needed), but did not undergo data extraction. Each RCT was counted only once, using either the primary outcomes paper (with or without results by sex or gender) or a secondary analysis paper that provided results by sex or gender (if not presented in the parent trial). Agreement on classification as an RCT was 86% (κ = 0.63) and conflicts were resolved by consensus.

       Data Charting Process

      Original RCTs and secondary analyses meeting eligibility criteria underwent data abstraction for 12 study characteristics by one investigator or research associate (E.D., E.Y., R.K., or K.U.). Each article abstraction was checked by a second reviewer and conflicts were resolved by consensus. Study characteristics of interest were selected a priori based on our prior review and opinion of participating investigators, who have extensive professional experience with the women veterans' health and health care literature. Throughout the initial review process, group consensus was used to refine or adjust extracted characteristics as needed to address feasibility issues.

       Data Items

      The characteristics abstracted were publication year, health care topic (grouped under 4 headings: 1] mental health, 2] physical health, 3] health care organization and delivery, and 4] access, use, and postdeployment health), sample size (n randomized), number and proportion of women participants, study location(s) (single site or multisite, VA Cooperative Studies Program or VA Women's Health Practice Based Research Network studies, non-VA/community based), time to longest follow-up (converted to days), intervention type (pharmacologic or behavioral), control type (active comparator, placebo, attention control, usual care, waitlist control), outcome type(s) (patient reported, objective clinical measure, resource use or cost), and funding source(s) (VA, U.S. Department of Defense, the NIH, industry, university, foundation, or not reported).
      We recorded journal titles and cross-referenced them with an online list of journals that report adhering to the
      International Committee of Medical Journal Editors (ICJME)
      Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals.
      recommendations for scholarly work (2017). The ICMJE recommendations state that “separate reporting of data by demographic variables, including age and sex … should be routine, unless there are compelling reasons not to stratify reporting.”
      We determined whether results were reported by sex or gender for any outcomes. This was defined as subgroup results for women for the primary study outcome and/or an explicit assessment of the interaction between the treatment and sex or gender. Interrater agreement on the presence of results by sex or gender was 94% (κ = 0.80). Conflicts were resolved by consensus.

       Synthesis of Results by Key Question

      • Q1: How do RCTs that report results by sex or gender differ from RCTs that do not? Across all included trials, we used descriptive statistics and visual graphing to describe trends in study characteristics. Among trials that included both male and female veterans, we compared characteristics of those that did provide results by sex or gender with those that did not (Figure 1; Table 1).
        Figure thumbnail gr1
        Figure 1Sources of evidence for key questions. RCTs, randomized, controlled trials.
        Table 1Characteristics of Included Trials
        Trial CharacteristicsFemale and Male Veteran Participants100% Women Veteran Participants (n = 5)
        No Results by Sex or Gender (n = 30)Results by Sex or Gender (n = 10)
        Randomized participants125.5 (56.5–295.3)343.5 (113.5–765.3)86 (58.5–2813)
        Women participants (%)11.2 (4.9–21.7)16.8 (13–40.4)100
        Time to longest follow-up (d)182.4 (91.2–365)228 (144.4–365)182.4 (45.6–456.2)
        Health care topic
         Mental health18 (60)4 (40)4 (80)
         Physical health9 (30)4 (40)1 (20)
         Health care delivery3 (10)0 (0)0 (0)
         Access, use, PDH0 (0)2 (20)0 (0)
        Study location(s)
         Single site20 (67)4 (40)3 (60)
         Multisite10 (33)3 (30)1 (20)
         VA Cooperative study3 (10)0 (0)0 (0)
         WH PBRN study0 (0)0 (0)0 (0)
         Non-VA or community based2 (7)2 (20)2 (40)
        Intervention type
         Pharmacologic8 (27)1 (10)0 (0)
         Behavioral11 (37)7 (70)4 (80)
         Health services9 (30)2 (20)1 (20)
         Device or physical treatment2 (7)0 (0)0 (0)
        Control type
         Active comparator13 (43)5 (50)4 (80)
         Placebo6 (20)0 (0)0 (0)
         Attention control2 (7)1 (10)0 (0)
         Usual care6 (20)3 (30)0 (0)
         Waitlist control3 (10)1 (10)1 (20)
        Outcome type(s)
         Patient-reported measure24 (80)10 (100)5 (100)
         Objective clinical measure11 (37)4 (40)1 (20)
        Funding source(s)
         VA24 (80)9 (90)2 (40)
         DOD4 (13)1 (10)2 (40)
         NIH/other government3 (10)2 (20)1 (20)
         Industry3 (10)0 (0)0 (0)
         University0 (0)0 (0)0 (0)
         Foundation3 (10)1 (10)0 (0)
         Not reported0 (0)0 (0)1 (20)
        Mention of sex/gender
         Methods12 (40)8 (80)
         Results20 (67)10 (100)
         Discussion14 (47)9 (90)
        Abbreviations: DOD, Department of Defense; NIH, National Institutes of Health; PDH, postdeployment health; VA, U.S. Department of Veterans Affairs; WH PBRN, Women's Health Practice Based Research Network.
        Note: Values are number (%) or median (IQR).
      • Q2: Among RCTs that report sex/gender results, do sex/gender analyses and reporting adhere to currently proposed best practices? The principal investigator (E.D.) reviewed studies classified as reporting results by sex or gender for attention to sex/gender in hypothesis, design, analysis, reporting, and acknowledgement of limitations (Table 2). Criteria for appraisal were adapted using the Sex and Gender Equity in Research (SAGER) guidelines (
        • Heidari S.
        • Babor T.F.
        • De Castro P.
        • Tort S.
        • Curno M.
        Sex and gender equity in research: Rationale for the SAGER guidelines and recommended use.
        ), with additional attention to relevant literature (
        • Aulakh A.K.
        • Anand S.S.
        Sex and gender subgroup analyses of randomized trials.
        ,
        • Johnson J.L.
        • Greaves L.
        • Repta R.
        Better science with sex and gender: Facilitating the use of a sex and gender-based analysis in health research.
        ,
        • McGregor A.J.
        • Hasnain M.
        • Sandberg K.
        • Morrison M.F.
        • Berlin M.
        • Trott J.J.
        How to study the impact of sex and gender in medical research: A review of resources.
        ,
        • Nieuwenhoven L.
        • Klinge I.
        Scientific excellence in applying sex-and gender-sensitive methods in biomedical and health research.
        ). In the absence of widely accepted guidelines or requirements for reporting results by sex or gender, this assessment was intended to be descriptive.
        Table 2Attention to Sex/Gender among 10 Trials that Reported Results by Sex or Gender
        Sex/Gender Reporting CriterionArticle ID Number
        12345678910
        Publication year20102015201620172018
        Hypothesis
         Explicitly stated hypothesisXX
         Suggested relationship or prior sex-specific findings citedXXX
        Study design
         Explicitly an article about sex/gender differencesXX
         Oversampling or enhanced recruitment of womenXX
         Sex/gender-specific inclusion/exclusion criteriaXX
         Randomization stratified or blocked by sexX
         Sex/gender balanced between treatment armsXXXXXXXX
        Statistical analysis
         Power calculation for interactionX
         Interaction test (sex/gender by treatment group)XXXXXXXX
        Reporting
         Gender of patients lost/withdrawn postrandomization reportedXX
         Sex/gender analysis described in introduction or methodsXXXXXXX
         Statistically significant sex/gender by treatment interactionXXX
         Any differential treatment effect by sex/gender reportedXXXXX
         Full sex-disaggregated results reported for primary outcomeXXXXXX
        Limitations acknowledged
         Small proportion of women limits generalizabilityXXX
         Subgroup analysis lacks power, interpret with caution, replicateXXX
        For full citations see Appendix 3.
      • Q3: Among RCTs that do not report results by sex or gender, how are sex/gender addressed in publications, if at all? For studies that included men and women but did not report results by sex or gender, two reviewers identified and described mentions of sex or gender in the methods, results, or discussion sections (Appendix 2). If articles referenced online supplemental materials, these were also reviewed for the presence of results by sex or gender (online tables) and/or sex-specific recruitment or eligibility criteria (online protocols).

      Results

       Selection of Sources of Evidence

      From the prior review (2008–2015), we identified 424 articles that did not report results by sex or gender (n = 361) or included very few women (n = 63; Figure 2). Our updated search (2015–2018) produced 1,461 citations. After removing duplicates, 1,820 abstracts from the combined searches underwent dual review and 1,668 were excluded. Next, we added 11 studies from the prior review that did report results by sex or gender for a total of 163 intervention studies, and then excluded 98 ineligible studies. Of 24 secondary analyses, 22 did not meet inclusion criteria. Two of the ineligible secondary analyses described parent trials that had not been previously captured by our search due to lack of keywords in the title or abstract. These parent trials were reviewed and included. The final sample included 45 trials: 43 papers reporting primary outcomes and 2 secondary analysis papers focused on sex/gender differences in the original trial (Appendix 3).
      Figure thumbnail gr2
      Figure 2Literature flow diagram: screening and selection of included articles. RCTs, randomized, controlled trials; VA, Veterans Administration.

       Characteristics and Results of Sources of Evidence

       All included trials (N = 45)

      Of the 45 included trials, 5 included only women veteran participants and 40 included both male and female veteran participants (Table 1, Figure 3, Appendix 3). Twenty-six trials (58%) covered mental health topics, including post-traumatic stress disorder (PTSD; n = 9), comorbid mental and physical health problems (n = 6), substance abuse (n = 5), and military sexual trauma (n = 2). Fourteen trials (31%) covered physical health topics including chronic pain (n = 4), diabetes (n = 2), hypertension (n = 2), and long-term care and aging (n = 2). Three trials (7%) addressed health care organization and delivery (all telehealth interventions) and two studies addressed health care use and postdeployment health. These 45 articles were published in 36 distinct medical journals, 15 of which reported adherence to the ICMJE recommendations.
      Figure thumbnail gr3
      Figure 3Evidence map: randomized controlled trials with veteran participants, by proportion women, reporting of results by sex or gender, and publication year.
      Four of the five trials that included all women veteran participants were small local trials (n = 51–126) about mental health topics that were published since 2014. The fifth, a large study of mammography promotion (n = 5,500), was a multisite trial published in 2008. Only one of the five, a substance abuse trial, received VA funding.
      Across 40 trials that included male and female participants, there were a total of 1,928 women; the median proportion of women participants was 14.3%. One-third of the trials (13 of 40) included less than 10% women, and one-half (20 of 40) had 10% to 33% women; only seven trials included more than 33% women. The number of trials published per year increased substantially in the latter half of our time period: from one to three per year (2008–2012) to four to nine per year (2013–2018; Figure 3).

       Synthesis of Results by Key Question

       Q1: How do RCTs that report results by sex or gender differ from RCTs that do not?

       Trials that reported results by sex or gender (n = 10)

      Ten of 40 trials that included both male and female participants reported results by sex or gender (25%; Table 1). Nine of the 10 trials that reported results by sex or gender were published in 2015 or later (Figure 3). Trials that reported results by sex or gender were larger and included a greater proportion of women than those that did not. These articles were published in 10 different journals, one-half of which reported adherence to the ICMJE recommendations.

       Trials that did not report results by sex or gender (n = 30)

      Several characteristics were observed more frequently among the 30 trials that did not report results by sex or gender (Table 1). They were more likely to be single-site studies. Notably, however, the three multisite VA Cooperative Studies Program trials did not report results by sex or gender. Most pharmacologic or device trials, all six studies with a placebo control, all three studies with industry funding, and three of four studies with foundation funding did not report results by sex or gender. Fourteen of 30 articles were published in journals that reported adherence to the ICMJE recommendations.

       Q2: Among RCTs that report sex/gender results, do sex/gender analyses and reporting adhere to currently proposed best practices?

      The 10 trials that reported results by sex or gender paid varied attention to sex and gender with respect to hypothesis, design, analysis, reporting, and acknowledgement of limitations (Table 2).

       Q3: Among RCTs that do not report results by sex or gender, how are sex/gender addressed in publications, if at all?

      The 30 trials that did not report results by sex or gender paid limited attention to sex and gender (Appendix 2).

      Discussion

       Summary of Evidence

      In this evidence map of 45 RCTs with women veteran participants published in the past decade, we found 10 studies that reported results by sex or gender, 30 that did not, and 5 studies with only female participants. To answer our first key question, we identified study characteristics that differed between RCTs that did or did not report results by sex or gender, including sample size, proportion of women, and intervention type. Publication in a journal that reports adherence to ICMJE recommendations, or use of a multisite research mechanism such as the VA Cooperative Studies program, did not predict reporting of results by sex or gender. For our second key question, we found that important features of appropriate sex and gender reporting were often missing from studies reporting results by sex or gender. For our third key question, we found that most studies that did not report results by sex or gender paid little attention to the role of sex or gender in published methods, results, or discussion sections. We review the significance of each of these findings below, with suggestions for action by research stakeholders.

       Participation of women veterans in trials

      Among 40 trials that included both male and female veterans, only 14.3% of trial participants were women, a far smaller proportion than in published general population trials (37%–41% women;
      • Avery E.
      • Clark J.
      Sex-related reporting in randomised controlled trials in medical journals.
      ,
      • Geller S.E.
      • Koch A.
      • Pellettieri B.
      • Carnes M.
      Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: Have we made progress?.
      ). However, participation by women veterans in trials was double their representation among VA users (7%) and higher than their representation among living veterans (10%;
      Department of Veterans Affairs, Office of Data Governance and Analytics
      Veteran population projection model (VetPop). From Veterans Benefits Administration; Veterans Health Administration, Office of the Assistant Deputy Under Secretary for Health for Policy and Planning.
      ,
      • Frayne S.M.
      • Phibbs C.S.
      • Saechao F.
      • Maisel N.C.
      • Friedman S.A.
      • Finlay A.
      • Berg E.
      • Balasubramanian V.
      • Dally S.K.
      • Ananth L.
      • Romodan Y.
      • Lee J.
      • Iqbal S.
      • Hayes P.M.
      • Zephyrin L.
      • Whitehead A.
      • Torgal A.
      • Katon J.G.
      • Haskell S.
      Sourcebook: Women Veterans in the Veterans Health Administration. Volume 3. Sociodemographics, Utilization, Costs of Care, and Health Profile.
      ). Only 1 of 13 trials that included less than 10% women reported results by sex or gender. In contrast, among 27 trials with at least 10% women, one-third reported results by sex or gender. Achieving higher participation rates by women is a necessary, although not sufficient, first step toward improving the generalizability of veterans’ health research.
      Outside the VA, funding agencies have contributed to the increased enrollment of women in clinical trials. The NIH's Advisory Committee on Research on Women's Health requires the inclusion of women and publicly reports the proportion of women enrolled in NIH-funded research, which has approached 50% for the past decade (
      NIH Tracking/Inclusion Committee
      Tracking of human subjects research as reported in fiscal year 2006 and fiscal year 2007.
      ;
      National Institutes of Health (NIH)
      Amendment: NIH Policy and Guidelines on the Inclusion of Women and Minorities as Subjects in Clinical Research.
      ). The Canadian Institutes of Health Research has created a cross-cutting sex and gender platform within large research consortia to provide guidance to research teams (
      • Duchesne A.
      • Tannenbaum C.
      • Einstein G.
      Funding agency mechanisms to increase sex and gender analysis.
      ).
      The VA Office of Research and Development, a major funder of veterans' health research and the sponsor of this special issue, has long required that “special efforts must be made when scientifically appropriate to include women veterans” in research studies. Although data on women's participation in VA trials have been collected and reported to oversight groups such as Institutional Review Boards or Data Monitoring Committees, these data have not been publicly available. Since 2013, VA Office of Research and Development has required that results from its registered clinical trials be submitted to ClinicalTrials.gov, including the number of men and women enrolled. This requirement was retroactively applied to trials that were initiated and/or actively enrolling since 2007 (
      • Huang G.D.
      • Altemose J.K.
      • O'Leary T.J.
      Public access to clinical trials: Lessons from an organizational implementation of policy.
      ). This public registry will enable VA and other interested stakeholders to track the inclusion of women in VA clinical trials over time.
      In addition, since 2010, VA Health Services Research & Development has invested in the Women's Health Research Network, which includes a Practice Based Research Network that supports investigators seeking to recruit more women in multisite observational and intervention studies and a Research Consortium that supports conduct of women's health research within VA. We did not identify any RCTs that reported using the Practice Based Research Network in this review. Given the long lag time between planning and publishing multisite studies, it is probably too soon to draw conclusions about effects of these initiatives on women veterans' research participation.

       Reporting of results by sex or gender

      The proportion of trials that reported results by sex or gender in this review (25%) was within the range of previously published reviews of non–veteran-specific RCTs from a similar time frame (13%–48%;
      • Avery E.
      • Clark J.
      Sex-related reporting in randomised controlled trials in medical journals.
      ,
      • Geller S.E.
      • Adams M.G.
      • Carnes M.
      Adherence to federal guidelines for reporting of sex and race/ethnicity in clinical trials.
      ,
      • Geller S.E.
      • Koch A.
      • Pellettieri B.
      • Carnes M.
      Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: Have we made progress?.
      ,
      • Phillips S.P.
      • Hamberg K.
      Doubly blind: A systematic review of gender in randomised controlled trials.
      ). Certain study types, such as those evaluating pharmacologic and device interventions, may be particularly important targets for sex and gender research equity (
      • Carey J.L.
      • Nader N.
      • Chai P.R.
      • Carreiro S.
      • Griswold M.K.
      • Boyle K.L.
      Drugs and medical devices: Adverse events and the impact on women’s health.
      ). In 2001, the U.S. General Accounting Office reported that 8 of 10 drugs removed from the market in the preceding years had more significant (and some potentially fatal) adverse effects for women than men (
      • Heinrich J.
      • Gahart M.
      • Rowe E.
      • Bradley L.
      A letter to The Honorable Tom Harkin. Drug safety: Most drugs withdrawn in recent years had greater health risks for women.
      ). Since 2001, the NIH and U.S. Food and Drug Administration have required phase III pharmaceutical and device trials to incorporate sex-specific analyses (
      National Institutes of Health (NIH)
      NIH policy and guidelines on the inclusion of women and minorities as subjects in clinical research (amendment to the 1994 Federal Register).
      ); a November 2017 amendment states these results must be reported on ClinicalTrials.gov (
      National Institutes of Health (NIH)
      Amendment: NIH Policy and Guidelines on the Inclusion of Women and Minorities as Subjects in Clinical Research.
      ).
      Among 11 pharmacologic or device studies, only 1 presented results by sex or gender and 7 included very few women. For example, a VA Cooperative Studies Program trial that evaluated prazosin for PTSD and reported a potentially practice-changing finding included only 7 women among 304 participants (
      • Raskind M.A.
      • Peskind E.R.
      • Chow B.
      • Harris C.
      • Davis-Karim A.
      • Holmes H.A.
      • Huang G.D.
      Trial of prazosin for post-traumatic stress disorder in military veterans.
      ). This is troubling because PTSD treatment needs may vary by sex, given evidence that military-related sexual trauma has a much stronger relationship with PTSD for women veterans (
      • Kimerling R.
      • Street A.E.
      • Pavao J.
      • Smith M.W.
      • Cronkite R.C.
      • Holmes T.H.
      • Frayne S.M.
      Military-related sexual trauma among Veterans Health Administration patients returning from Afghanistan and Iraq.
      ). Likewise, the efficacy and side effects of prazosin likely vary by sex; indeed, the prazosin study used different dosing protocols for men and women. Nonetheless, the publication did not describe the under-representation of women as a limitation.
      Although the ICMJE recommends routine reporting of sex-stratified results, we found little evidence that medical journal editors are enforcing this expectation (2017). Of 19 included trials published in 16 journals that state they follow the ICMJE guidelines, only 5 reported results by sex or gender. In 2016, the European Association of Science Editors proposed guidelines for Sex and Gender Equity in Research (SAGER) that can be incorporated into journal instructions to authors and peer-review forms (
      • Heidari S.
      • Babor T.F.
      • De Castro P.
      • Tort S.
      • Curno M.
      Sex and gender equity in research: Rationale for the SAGER guidelines and recommended use.
      ). However, the SAGER guidelines have not yet been widely implemented. The most widely used set of guidelines for randomized trial reporting, the CONSORT statement, makes no specific recommendation about sex/gender results reporting.
      Without prioritization of sex/gender-stratified reporting, competing reporting requirements and strict journal word limits may contribute to the omission of sex/gender results. The single pharmacologic intervention trial in this review that reported results by sex or gender, led by an author of the current report (E.K.) and published in a journal with a 3,000-word limit, included sex-stratified results in an Online Supplemental Appendix (
      • Krebs E.E.
      • Gravely A.
      • Nugent S.
      • Jensen A.C.
      • DeRonne B.
      • Goldsmith E.S.
      • Noorbaloochi S.
      Effect of opioid vs nonopioid medications on pain-related function in patients with chronic back pain or hip or knee osteoarthritis pain: The SPACE randomized clinical trial.
      ). The only in-text mention of the sex subgroup analysis was cut in the prepublication editorial process. Because we did not speak with the authors of the other trials included in this review, we do not know how often omissions of sex/gender results are due to authors' versus editors’ decisions.

       Areas for improved attention to sex and gender

      Across 10 trials that reported results by sex or gender, we identified targets for improvement with respect to study design, analysis, and reporting. Few trials presented a hypothesis for the relationship between sex/gender and the intervention to justify the analysis, and most trials did not block or stratify randomization by sex, or report the sex of participants who withdrew early. Only one of the five trials that reported a nonsignificant sex/gender by treatment interaction also provided full results disaggregated by sex.
      Sex-disaggregated outcomes should be routinely reported regardless of statistical test results, because interaction tests for subgroup analyses are often underpowered and, conversely, because statistically significant findings can be due to chance. Further, the routine reporting of sex-disaggregated results, regardless of findings, allows for meta-analyses by sex in the future and may reduce unnecessary repetition in research (
      • Clayton J.A.
      • Tannenbaum C.
      Reporting sex, gender, or both in clinical research?.
      ). In the VA, where women represent an extreme minority of the population, techniques that combine data from multiple studies to overcome sample size limitations (
      Institute of Medicine (US) Committee on Understanding the Biology of Sex and Gender Differences.
      ) may be especially relevant.
      Among 30 trials that did not report results by sex or gender, fewer than one-half remarked on sex/gender limitations in the discussion. Investigators who do not incorporate sex or gender analyses into the design, execution, or analysis of their trial must at least critique this omission (
      • Johnson J.L.
      • Greaves L.
      • Repta R.
      Better science with sex and gender: Facilitating the use of a sex and gender-based analysis in health research.
      ).
      Shortcomings we identified are not restricted to veteran research; previous literature reviews have also found methodologic shortcomings in subgroup analyses by sex (
      • Aulakh A.K.
      • Anand S.S.
      Sex and gender subgroup analyses of randomized trials.
      ). Inattention to sex and gender decreases research equity, but poorly executed sex and gender analyses also have risks. For example, overinterpretation of post hoc subgroup analyses as conclusive, rather than hypothesis generating, can lead providers to withhold potentially beneficial treatments from women (
      • Aulakh A.K.
      • Anand S.S.
      Sex and gender subgroup analyses of randomized trials.
      ) or overuse potentially harmful treatments (
      • Rothwell P.M.
      Subgroup analysis in randomised controlled trials: Importance, indications, and interpretation.
      ). Most investigators lack training in sex/gender equity in research design, conduct, analysis, and reporting. Although workshops and publications are available for individual investigators seeking to build this skillset (
      • Johnson J.L.
      • Greaves L.
      • Repta R.
      Better science with sex and gender: Facilitating the use of a sex and gender-based analysis in health research.
      ,
      • Nieuwenhoven L.
      • Klinge I.
      Scientific excellence in applying sex-and gender-sensitive methods in biomedical and health research.
      ), uniform requirements for training tied to research funding may reach more researchers (
      • Duchesne A.
      • Tannenbaum C.
      • Einstein G.
      Funding agency mechanisms to increase sex and gender analysis.
      ). Within the VA, the Women's Health Research Network can provide expert guidance for investigators to facilitate improved methodology and design (
      • Rohrer L.D.
      • Gierisch J.M.
      • Fish L.J.
      • Blakeney J.K.
      • Bastian L.A.
      A five-step guide for moving from observational studies to interventional research for women veterans.
      ), but investigators do not always take advantage of this resource.

       Limitations

      Limitations of this review include search criteria specific to women, which led us to miss some RCTs that did not reference women in the abstract or MeSH terms. As a result, we may have overestimated the proportion of all RCTs that report results by sex or gender. Our analysis is limited by the information provided by study authors in published manuscripts or linked online materials. We restricted our reporting to published data, but some trials listed eligibility criteria related to sex/gender on ClinicalTrials.gov that were not mentioned in the published article. Finally, results apply only to RCTs. Our exclusion of single-arm pilots and implementation/evaluation projects created a more uniform cohort of included studies for comparison purposes, but omitted research that is also important to women veterans’ health and deserves similar scrutiny.

       Implications for Practice and/or Policy

      Several actions by research stakeholders could improve sex and gender sensitivity in veteran research. Funders could set more specific expectations for women's inclusion, track and publicly report women's participation, and offer additional support to increase participation of women in multisite VA trials. Journal editors could adopt and enforce sex/gender reporting guidelines, including a requirement to report results of clinical trials disaggregated by sex. Investigators should target a greater proportion of female study subjects and/or a larger overall study size to make reporting results disaggregated by sex/gender feasible. They should acknowledge the role of sex/gender in their study, including the limitations of not providing results by sex or gender. To improve the quality of reporting, investigators can seek training and use existing resources for the conduct, analysis, and reporting of sex- and gender-equitable research.

       Conclusions

      This systematic review found that women veterans are increasingly participating in RCTs, but reporting of results by sex or gender remains infrequent. In particular, trials with less than 10% women and trials testing pharmacologic and device interventions were unlikely to report results by sex or gender. Trials that did report results by sex or gender often omitted important details related to study design. Reporting of full sex-disaggregated results, regardless of statistical test outcomes, was rare. Implementing the suggested approaches will increase applicability of the knowledge gained from veteran research to the care of women.

      Acknowledgments

      The authors gratefully acknowledge the assistance of Nancy Greer, PhD, ESP Program Manager for the Minneapolis VA Medical Center. Dr. Danan had full access to all the data in the study and takes full responsibility for the integrity of the data and the accuracy of the data analysis.

      Supplementary Data

      References

        • Aulakh A.K.
        • Anand S.S.
        Sex and gender subgroup analyses of randomized trials.
        Womens Health Issues. 2007; 17: 342-350
        • Avery E.
        • Clark J.
        Sex-related reporting in randomised controlled trials in medical journals.
        Lancet. 2016; 388: 2839-2840
        • Carey J.L.
        • Nader N.
        • Chai P.R.
        • Carreiro S.
        • Griswold M.K.
        • Boyle K.L.
        Drugs and medical devices: Adverse events and the impact on women’s health.
        Clinical Therapeutics. 2017; 39: 10-22
        • Clayton J.A.
        • Tannenbaum C.
        Reporting sex, gender, or both in clinical research?.
        JAMA. 2016; 316: 1863-1864
        • Danan E.
        • Ensrud K.
        • Krebs E.
        • Koeller E.
        • Greer N.
        • Velasquez T.
        • Wilt T.
        An evidence map of the women veterans’ health research literature (2008–2015).
        (Available:)
        • Danan E.
        • Krebs E.
        • Ensrud K.
        • Koeller E.
        • MacDonald R.
        • Velasquez T.
        • Wilt T.
        An evidence map of the women veterans’ health research literature (2008–2015).
        Journal of General Internal Medicine. 2017; 32: 1359-1376
        • Department of Veterans Affairs, Office of Data Governance and Analytics
        Veteran population projection model (VetPop). From Veterans Benefits Administration; Veterans Health Administration, Office of the Assistant Deputy Under Secretary for Health for Policy and Planning.
        (Available:)
        www.va.gov/vetdata/index.asp
        Date: 2016
        Date accessed: January 4, 2019
        • Duchesne A.
        • Tannenbaum C.
        • Einstein G.
        Funding agency mechanisms to increase sex and gender analysis.
        Lancet. 2017; 389: 699
        • Frayne S.M.
        • Phibbs C.S.
        • Saechao F.
        • Maisel N.C.
        • Friedman S.A.
        • Finlay A.
        • Berg E.
        • Balasubramanian V.
        • Dally S.K.
        • Ananth L.
        • Romodan Y.
        • Lee J.
        • Iqbal S.
        • Hayes P.M.
        • Zephyrin L.
        • Whitehead A.
        • Torgal A.
        • Katon J.G.
        • Haskell S.
        Sourcebook: Women Veterans in the Veterans Health Administration. Volume 3. Sociodemographics, Utilization, Costs of Care, and Health Profile.
        Women's Health Evaluation Initiative, Women's Health Services, Veterans Health Administration, Department of Veterans Affairs, Washington, DC2014
        • Geller S.E.
        • Adams M.G.
        • Carnes M.
        Adherence to federal guidelines for reporting of sex and race/ethnicity in clinical trials.
        Journal of Womens Health (Larchmont). 2006; 15: 1123-1131
        • Geller S.E.
        • Koch A.
        • Pellettieri B.
        • Carnes M.
        Inclusion, analysis, and reporting of sex and race/ethnicity in clinical trials: Have we made progress?.
        Journal of Womens Health (Larchmt). 2011; 20: 315-320
        • Heidari S.
        • Babor T.F.
        • De Castro P.
        • Tort S.
        • Curno M.
        Sex and gender equity in research: Rationale for the SAGER guidelines and recommended use.
        Research Integrity and Peer Review. 2016; 1: 2
        • Heinrich J.
        • Gahart M.
        • Rowe E.
        • Bradley L.
        A letter to The Honorable Tom Harkin. Drug safety: Most drugs withdrawn in recent years had greater health risks for women.
        2001 (Washington, DC Available:)
        www.gao.gov/products/GAO-01-286R
        Date accessed: January 4, 2019
        • Huang G.D.
        • Altemose J.K.
        • O'Leary T.J.
        Public access to clinical trials: Lessons from an organizational implementation of policy.
        Contemporary Clinical Trials. 2017; 57: 87-89
      1. Institute of Medicine (US) Committee on Understanding the Biology of Sex and Gender Differences.
        in: Wizemann T.M. Pardue M.L. Exploring the Biological Contributions to Human Health: Does Sex Matter? National Academies Press (US), Washington (DC)2001 (Available from:)
        • International Committee of Medical Journal Editors (ICJME)
        Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals.
        (Available:)
        www.ICMJE.org
        Date: 2017
        Date accessed: January 4, 2019
        • Johnson J.L.
        • Greaves L.
        • Repta R.
        Better science with sex and gender: Facilitating the use of a sex and gender-based analysis in health research.
        International Journal for Equity in Health. 2009; 8: 14
        • Kimerling R.
        • Street A.E.
        • Pavao J.
        • Smith M.W.
        • Cronkite R.C.
        • Holmes T.H.
        • Frayne S.M.
        Military-related sexual trauma among Veterans Health Administration patients returning from Afghanistan and Iraq.
        American Journal of Public Health. 2010; 100: 1409-1412
        • Krebs E.E.
        • Gravely A.
        • Nugent S.
        • Jensen A.C.
        • DeRonne B.
        • Goldsmith E.S.
        • Noorbaloochi S.
        Effect of opioid vs nonopioid medications on pain-related function in patients with chronic back pain or hip or knee osteoarthritis pain: The SPACE randomized clinical trial.
        JAMA. 2018; 319: 872-882
        • McGregor A.J.
        • Hasnain M.
        • Sandberg K.
        • Morrison M.F.
        • Berlin M.
        • Trott J.J.
        How to study the impact of sex and gender in medical research: A review of resources.
        Biology of Sex Differences. 2016; 7: 46
        • Miake-Lye I.M.
        • Hempel S.
        • Shanman R.
        • Shekelle P.G.
        What is an evidence map? A systematic review of published evidence maps and their definitions, methods, and products.
        Systematic Reviews. 2016; 5: 28
        • National Institutes of Health (NIH)
        NIH policy and guidelines on the inclusion of women and minorities as subjects in clinical research (amendment to the 1994 Federal Register).
        (Available:)
        • National Institutes of Health (NIH)
        Amendment: NIH Policy and Guidelines on the Inclusion of Women and Minorities as Subjects in Clinical Research.
        (Available:)
        • Nieuwenhoven L.
        • Klinge I.
        Scientific excellence in applying sex-and gender-sensitive methods in biomedical and health research.
        Journal of Women's Health. 2010; 19: 313-321
        • NIH Tracking/Inclusion Committee
        Tracking of human subjects research as reported in fiscal year 2006 and fiscal year 2007.
        2008 (Available:)
        • Office of Research on Women’s Health
        Report of the advisory committee on research on women's health, fiscal years 2015-2016: Office of Research on Women s Health and NIH support for research on women’s health.
        U.S. Department of Health and Human Services, Public Health Service, National Institutes of Health, Bethesda, MD2017
        • Phillips S.P.
        • Hamberg K.
        Doubly blind: A systematic review of gender in randomised controlled trials.
        Global Health Action. 2016; 9: 29597
        • Raskind M.A.
        • Peskind E.R.
        • Chow B.
        • Harris C.
        • Davis-Karim A.
        • Holmes H.A.
        • Huang G.D.
        Trial of prazosin for post-traumatic stress disorder in military veterans.
        New England Journal of Medicine. 2018; 378: 507-517
        • Rohrer L.D.
        • Gierisch J.M.
        • Fish L.J.
        • Blakeney J.K.
        • Bastian L.A.
        A five-step guide for moving from observational studies to interventional research for women veterans.
        Women's Health Issues. 2011; 21: S98-S102
        • Rothwell P.M.
        Subgroup analysis in randomised controlled trials: Importance, indications, and interpretation.
        Lancet. 2005; 365: 176-186
        • Tricco A.C.
        • Lillie E.
        • Zarin W.
        • O'Brien K.K.
        • Colquhoun H.
        • Levac D.
        • Straus S.E.
        PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation.
        Annals of Internal Medicine. 2018; 169: 467-473
        • U.S. Department of Health and Human Services
        NIH guidelines on the inclusion of women and minorities as subjects in clinical research.
        1994 (23(11). Available: https://grants.nih.gov/grants/guide/notice-files/not94-100.html. Accessed May 17, 2019)

      Biography

      Elisheva R. Danan, MD, MPH, is a Core Investigator, Center for Care Delivery and Outcomes Research (CCDOR), and Assistant Professor, Department of Medicine, University of Minnesota. She is a general internist and health services researcher interested in equity and women's health.
      Kristen Ullman, MPH, is a Health Science Specialist at CCDOR. She has broad interests in health services research and evidence synthesis. She has experience working with the Minneapolis VA Evidence Synthesis Program and the University of Minnesota Evidence-based Practice Center.
      Ruth S. Klap, PhD, is a Core Investigator, Center for the Study of Healthcare Innovation, Implementation and Policy (CSHIIP) and Program Manager, VA Women's Health Research Consortium. Her work focuses on intimate partner violence and gender/ethnic disparities in access to care.
      Elizabeth M. Yano, PhD, is Director of CSHIIP and Adjunct Professor of Health Policy & Management, UCLA Fielding School of Public Health. She leads the VA Women's Health Research Network and is PI of the VA Women's Health Research Consortium.
      Erin E. Krebs, MD, MPH, is a Core Investigator at CCDOR and Associate Professor, Department of Medicine, University of Minnesota. She is a general internist and health services researcher with a primary goal to improve chronic pain management in primary care.