Advertisement

Designing Studies for Sex and Gender Analyses: How Research Can Derive Clinically Useful Knowledge for Women's Health

  • Ruth Klap
    Correspondence
    Correspondence to: Ruth Klap, PhD, VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Healthcare System, 11301 Wilshire Blvd (151) Building 206, Los Angeles, CA 90073. Phone: (310) 478-3711x44636; fax: (818) 895-5838.
    Affiliations
    VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy, VA Greater Los Angeles Health Care System, Los Angeles, California

    Department of Psychiatry and Biobehavioral Sciences, David Geffen School of Medicine, University of California Los Angeles, Los Angeles, California
    Search for articles by this author
  • Keith Humphreys
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, California
    Search for articles by this author
      It was once considered acceptable for clinical researchers to draw conclusions about women's health based on studies that only enrolled men (
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). Fortunately, in recent decades, government, research funders, and other groups have promoted greater inclusion of women participants in clinical research (
      • Freeman A.
      • Stanko P.
      • Berkowitz L.N.
      • Parnell N.
      • Zuppe A.
      • Bale T.L.
      • Epperson C.N.
      Inclusion of sex and gender in biomedical research: Survey of clinical research proposed at the University of Pennsylvania.
      ,
      Society for Women's Health Research
      Institute of medicine report validates the science of sex differences.
      ). As a result, the participation of women in clinical research has increased (
      • Freeman A.
      • Stanko P.
      • Berkowitz L.N.
      • Parnell N.
      • Zuppe A.
      • Bale T.L.
      • Epperson C.N.
      Inclusion of sex and gender in biomedical research: Survey of clinical research proposed at the University of Pennsylvania.
      ,
      • Lippman A.
      The inclusion of women in clinical trials: Are we asking the right questions?.
      ,
      Society for Women's Health Research
      Institute of medicine report validates the science of sex differences.
      ), and the articles in this special issue of Women's Health Issues are some of the fruits of that progress. Yet many studies within and outside Veterans Affairs (VA) do not report results by sex or gender
      The term “sex” is used to refer to biological or physiological factors and “gender” is used to refer to psychosocial, gender identity, or cultural factors. It should be noted that these terms are not mutually exclusive (
      • Clayton J.A.
      • Tannenbaum C.
      Reporting sex, gender, or both in clinical research?.
      ).
      1The term “sex” is used to refer to biological or physiological factors and “gender” is used to refer to psychosocial, gender identity, or cultural factors. It should be noted that these terms are not mutually exclusive (
      • Clayton J.A.
      • Tannenbaum C.
      Reporting sex, gender, or both in clinical research?.
      ).
      (
      • Avery E.
      • Clark J.
      Sex-related reporting in randomised controlled trials in medical journals.
      ,
      • Clayton J.A.
      • Tannenbaum C.
      Reporting sex, gender, or both in clinical research?.
      ,
      • Duan-Porter W.
      • Goldstein K.M.
      • McDuffie J.R.
      • Hughes J.M.
      • Clowse M.E.
      • Klap R.S.
      • Gierisch J.M.
      Reporting of sex effects by systematic reviews on interventions for depression, diabetes, and chronic pain.
      ,
      • Vidaver R.M.
      • Lafleur B.
      • Tong C.
      • Bradshaw R.
      • Marts S.A.
      Women subjects in NIH-funded clinical research literature: Lack of progress in both representation and analysis by sex.
      ). Some scholars propose increasing the frequency of such analyses by having journals or funders mandate them (
      • Mazure C.M.
      • Jones D.P.
      Twenty years and still counting: Including women as participants and studying sex and gender in biomedical research.
      ,
      • Sugimoto C.R.
      • Ahn Y.-Y.
      • Smith E.
      • Macaluso B.
      • Larivière V.
      Factors affecting sex-related reporting in medical research: A cross-disciplinary bibliometric analysis.
      ).
      In this commentary, we express reservations about that approach and instead suggest that more clinically relevant lessons about women's health could be derived from research if 1) studies are adequately designed and powered for testing sex and/or gender interactions in advance, 2) study exclusion criteria are chosen in light of their sex and gender impact, 3) researchers use research designs and statistical techniques that can illuminate sex and gender differences, and 4) researchers routinely make data publicly available after publication.

      Mandatory Analysis of Clinical Research Outcomes by Sex and Gender: Some Reservations

      Clinical trials estimate average treatment effects (
      • Kravitz R.L.
      • Duan N.
      • Braslow J.
      Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.
      ), and ample evidence highlights the hazards of assuming such effects apply equally to male and female participants (
      • Bailey K.R.
      Reporting of sex-specific results: A statistician's perspective.
      ,
      • Hawkes S.
      • Haseen F.
      • Aounallah-Skhiri H.
      Measurement and meaning: Reporting sex in health research.
      ,
      • Wallach J.D.
      • Sullivan P.G.
      • Trepanowski J.F.
      • Sainani K.L.
      • Steyerberg E.W.
      • Ioannidis J.P.
      Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials.
      ). Yet we doubt the usefulness of a policy of funding agencies or journals mandating or endorsing analysis by sex and gender, for all studies, for several reasons.
      Only a minority of studies are designed and conducted in a manner that adequately supports the study of outcome differences among sample subgroups (
      • Bucholz E.M.
      • Krumholz H.M.
      Women in clinical research: What we need for progress.
      ,
      • Freeman A.
      • Stanko P.
      • Berkowitz L.N.
      • Parnell N.
      • Zuppe A.
      • Bale T.L.
      • Epperson C.N.
      Inclusion of sex and gender in biomedical research: Survey of clinical research proposed at the University of Pennsylvania.
      ), in part because no benchmark defining adequate enrollment by sex has been established by the U.S. Food and Drug Administration or the National Institutes of Health (
      • Bucholz E.M.
      • Krumholz H.M.
      Women in clinical research: What we need for progress.
      ). Most clinical trials are only powered to identify overall treatment effects. A trial that has 80% power to detect an overall treatment effect will only have 29% power to detect an interaction effect of the same magnitude; sample size would have to increase four-fold for the study to have 80% power to detect a sex- or gender-specific effect (
      • Hernández A.V.
      • Boersma E.
      • Murray G.D.
      • Habbema J.D.F.
      • Steyerberg E.W.
      Subgroup analyses in therapeutic cardiovascular clinical trials: Are most of them misleading?.
      ). False negatives are always a risk when conducting underpowered analyses; as the number of statistical comparisons conducted within a data set increases, the likelihood of finding statistically significant but clinically meaningless differences increases.
      • Peto R.
      Current misconception 3: That subgroup-specific trial mortality results often provide a good basis for individualising patient care.
      puckishly illustrated the perils of running multiple unplanned comparisons by demonstrating that although daily aspirin was clearly beneficial overall, it was harmful for people born under Libra and Gemini astrological signs.
      Journal editors or research funders mandating reanalysis of data to account for sex and gender will not address the problems associated with the design and conduct of a study (
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). Few studies provide a rationale or a hypothesis related to the subgroup analyses (
      • Aulakh A.K.
      • Anand S.S.
      Sex and gender subgroup analyses of randomized trials: The need to proceed with caution.
      ,
      • Avery E.
      • Clark J.
      Sex-related reporting in randomised controlled trials in medical journals.
      ). Evidence suggests that most published subgroup analyses are misleading (
      • Aulakh A.K.
      • Anand S.S.
      Sex and gender subgroup analyses of randomized trials: The need to proceed with caution.
      ,
      • Avery E.
      • Clark J.
      Sex-related reporting in randomised controlled trials in medical journals.
      ,
      • Petticrew M.
      • Tugwell P.
      • Kristjansson E.
      • Oliver S.
      • Ueffing E.
      • Welch V.
      Damned if you do, damned if you don't: Subgroup analysis and equity.
      ,
      • Wallach J.D.
      • Sullivan P.G.
      • Trepanowski J.F.
      • Sainani K.L.
      • Steyerberg E.W.
      • Ioannidis J.P.
      Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials.
      ). Attempts at replicating subgroup findings are rare, but when done, the original findings are not generally supported (
      • Wallach J.D.
      • Sullivan P.G.
      • Trepanowski J.F.
      • Sainani K.L.
      • Steyerberg E.W.
      • Ioannidis J.P.
      Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials.
      ), and many subgroup analyses noted in published abstracts are not supported by their own data (
      • Wallach J.D.
      • Sullivan P.G.
      • Trepanowski J.F.
      • Sainani K.L.
      • Steyerberg E.W.
      • Ioannidis J.P.
      Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials.
      ). Researchers often incorrectly conclude that a statistically significant effect in one subgroup but not another (stratified analyses) establishes evidence of a subgroup affect when, in fact, such analyses require the conduct of a formal interaction test (
      • Wallach J.D.
      • Sullivan P.G.
      • Trepanowski J.F.
      • Sainani K.L.
      • Steyerberg E.W.
      • Ioannidis J.P.
      Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials.
      ). Furthermore, subgroup analyses of one variable at a time often fail to detect treatment effect differences because patients have many attributes that can influence treatment effectiveness (
      • Kent D.M.
      • Hayward R.A.
      Limitations of applying summary results of clinical trials to individual patients: The need for risk stratification.
      ). In light of these issues, we are wary of requiring sex and gender analysis as a general matter in clinical research or gauging progress in the inclusion of women in research through the existence of reported subgroup analyses. We share the concern of
      • Springer K.W.
      • Stellman J.M.
      • Jordan-Young R.M.
      Beyond a catalogue of differences: A theoretical frame and good practice guidelines for researching sex/gender in human health.
      that this practice could lead to an ever-growing catalogue of differences (that are likely spurious) that will distract from needed research into specific mechanisms that lead to sex or gender differences and the relationship between sex and gender differences. Furthermore, such requirements are at odds with statistical and methodological guidance regarding planning for subgroup analyses.

      Designing Studies for Sex and Gender Subgroup Analyses

      Planning for subgroup analysis is infinitely superior to implementing unplanned post hoc analyses (
      • Aulakh A.K.
      • Anand S.S.
      Sex and gender subgroup analyses of randomized trials: The need to proceed with caution.
      ,
      • Yusuf S.
      • Wittes J.
      • Probstfield J.
      • Tyroler H.A.
      Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials.
      ). Proper planning requires having an explicit rationale for performing subgroup analyses, specifying sex- and gender-related hypotheses a priori, ensuring sufficient statistical power, stratifying randomizations by sex, designing a formal test of the interaction, and adjusting for multiple statistical tests.
      Achieving a sample size with sufficient power for subgroup analyses is resource intensive and thus requires a strong rationale (
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). There are costs and benefits associated with powering studies for sex and gender subgroup analyses in the general population, and these tradeoffs are exacerbated in the VA where women represent only 7.5% of the patient population (
      • Frayne S.M.
      • Phibbs C.
      • Saechao F.
      • Friedman S.
      • Shaw J.
      • Romodan Y.
      • Haskel S.
      Sourcebook: Women Veterans in the Veterans Health Administration.
      ). Although the VA Women's Health Practice-Based Research Network can increase the representation of women veterans in VA research (
      • Frayne S.M.
      • Carney D.V.
      • Bastian L.
      • Bean-Mayberry B.
      • Sadler A.
      • Klap R.
      • Yee E.F.
      The VA women’s health practice-based research network: Amplifying women veterans’ voices in VA research.
      ), even with a multisite approach it can be challenging to sufficiently power studies of VA users for sex and gender comparisons, depending on the research question and study design.
      It is therefore critical to identify research questions where sex or gender is likely to matter and adequately design and power studies for subgroup analyses in these situations. “A starting assumption that there is a sex difference for any association being studied could lead to publication of false conclusions,”
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      cautions. Researchers need to consider what is known about sex and gender differences at the research design stage. Information on how groups based on sex and age metabolize medications, for instance, could allow for the identification of questions where subgroup differences are possible and should be studied and those where they are unlikely to occur (
      • Allmark P.
      Should research samples reflect the diversity of the population?.
      ).

      Choosing Exclusion Criteria Wisely in Light of Disparate Sex and Gender Effects

      Clinical researchers often undermine the clinical relevance of their data by choosing trial exclusion criteria that disproportionately prevent certain subsets of women from enrolling in studies. The most obvious example is a demographic one, namely excluding the elderly (who are disproportionately women), a practice of 40% of clinical trials in medicine (
      • Zulman D.M.
      • Sussman J.B.
      • Chen X.
      • Cigolle C.T.
      • Blaum C.S.
      • Hayward R.A.
      Examining the evidence: A systematic review of the inclusion and analysis of older adults in randomized controlled trials.
      ). Exclusion of participants with disorders more common in women (e.g., depression;
      • Humphreys K.
      • Williams L.M.
      What can treatment research offer general practice?.
      ), or social conditions more common in women (e.g., unemployment;
      • Humphreys K.
      • Weingardt K.R.
      • Harris A.H.
      Influence of subject eligibility criteria on compliance with National Institutes of Health guidelines for inclusion of women, minorities, and children in treatment research.
      ) can have the same effect.
      Importantly, although oversampling women can improve power for subgroup analyses, differences in the sex and gender impact of an exclusion criterion cannot be compensated for by oversampling. This method only produces a larger relatively unrepresentative sample of women, allowing more precision in estimating the wrong answer. For example, a study that excluded individuals under 5′7″ tall would have a more representative sample of men than women, no matter whether women were oversampled or not.

      Research Designs and Statistical Techniques that Can Illuminate Sex and Gender Differences

      Because the conduct of well-planned, sufficiently powered subgroup analyses is often not feasible, alternative approaches are needed. Although it is risky to act on findings from exploratory subgroup analyses, based on interaction tests, they can be used, if interpreted cautiously, to generate hypotheses that are tested in future studies (see
      • Brown M.C.
      • Sims K.J.
      • Gifford E.
      • Goldstein K.M.
      • Johnson M.R.
      • Williams C.
      • Provenzale D.
      Gender-based differences among 1990-91 Gulf War Era veterans: Demographics, lifestyle behaviors, and health conditions.
      and
      • Danan E.R.
      • Sherman S.E.
      • Clothier B.
      • Burgess D.J.
      • Pinsker E.
      • Joseph A.M.
      • Fu S.F.
      Smoking cessation among female and male veterans before and after a randomized trial of proactive outreach.
      ). Furthermore, the risks of producing type 1 errors (false positives) can be mitigated by correcting for multiple comparisons (see
      • Benjamini Y.
      • Hochberg Y.
      Controlling the false discovery rate: A practical and powerful approach to multiple testing.
      and
      • Naylor J.C.
      • Wagner H.R.
      • Johnston C.
      • Elbogen E.E.
      • Brancu M.
      • Marx C.E.
      • Strauss J.L.
      Pain intensity and pain interference in male and female Iraq/Afghanistan-era veterans.
      ). Although stratified analyses are not good indicators of whether sex or gender differences exist, if all studies routinely presented findings stratified by sex or gender, perhaps as supplementary materials with warnings about overinterpretation of findings, power issues could be addressed through meta-analysis techniques (
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). Faced with selective reporting by gender, however, meta-analyses will be seriously compromised by publication bias (
      • Bailey K.R.
      Reporting of sex-specific results: A statistician's perspective.
      ).
      Suitable ways of addressing sex and gender subgroup differences include using underutilized statistical techniques (
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). Bayesian approaches, for instance, can be used to reduce sample size requirements in traditional frequentist analyses (
      • Berry D.A.
      Introduction to Bayesian methods III: Use and interpretation of Bayesian tools in design and analysis.
      ). In addition, Bayesian and adaptive analytic trial designs can be used to update trial information as results accumulate, allowing for a reduction in sample size requirements and a focus on clinically relevant subgroups (
      • Berry D.A.
      Introduction to Bayesian methods III: Use and interpretation of Bayesian tools in design and analysis.
      ,
      • Kravitz R.L.
      • Duan N.
      • Braslow J.
      Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.
      ). The use of risk stratification is another promising alternative to subgroup analyses. Specifically in situations where an externally developed risk prediction tool is available, multivariate risk models often have greater power to detect treatment differences than individual subgroup analyses (
      • Hayward R.A.
      • Kent D.M.
      • Vijan S.
      • Hofer T.P.
      Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis.
      ,
      • Kent D.M.
      • Hayward R.A.
      Limitations of applying summary results of clinical trials to individual patients: The need for risk stratification.
      ,
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). By combining multiple patient attributes into a single score that describes a single dimension of risk upon which a treatment effect is likely to vary, risk-stratified analyses minimize difficulties associated with multiple comparisons and poor statistical power (
      • Kent D.M.
      • Hayward R.A.
      Limitations of applying summary results of clinical trials to individual patients: The need for risk stratification.
      ).

      Data Sharing Can Help

      If researchers make raw data available, then postpublication data could be pooled across multiple studies that address the same condition. Data merging could be especially valuable if common measures were included in studies (
      • Wizeman T.M.
      Sex-specific reporting of scientific research: A workshop summary.
      ). Researchers could increase statistical power by pooling data across studies that individually enrolled small numbers of women veterans.

      Conclusion

      The importance of gathering high-quality data on women's responses to health care interventions cannot be overstated from a scientific, clinical, or ethical viewpoint. However, mandating post hoc sex and gender analysis may just as easily result in misleading rather than useful data on women's health. A better path forward involves a priori considerations of sex and gender analysis where warranted, choosing exclusion criteria wisely, exploiting innovative statistical techniques, and broadly sharing data so that other researchers can analyze large pools of data on women's response to treatments.

      Supplementary Data

      References

        • Allmark P.
        Should research samples reflect the diversity of the population?.
        Journal of Medical Ethics. 2004; 30: 185-189
        • Aulakh A.K.
        • Anand S.S.
        Sex and gender subgroup analyses of randomized trials: The need to proceed with caution.
        Women's Health Issues. 2007; 17: 342-350
        • Avery E.
        • Clark J.
        Sex-related reporting in randomised controlled trials in medical journals.
        Lancet. 2016; 388: 2839-2840
        • Bailey K.R.
        Reporting of sex-specific results: A statistician's perspective.
        Mayo Clinic Proceedings. 2007; 82: 158
        • Benjamini Y.
        • Hochberg Y.
        Controlling the false discovery rate: A practical and powerful approach to multiple testing.
        Journal of the Royal Statistical Society: Series B (Methodological). 1995; 57: 289-300
        • Berry D.A.
        Introduction to Bayesian methods III: Use and interpretation of Bayesian tools in design and analysis.
        Sage, Thousand Oaks, CA2005
        • Brown M.C.
        • Sims K.J.
        • Gifford E.
        • Goldstein K.M.
        • Johnson M.R.
        • Williams C.
        • Provenzale D.
        Gender-based differences among 1990-91 Gulf War Era veterans: Demographics, lifestyle behaviors, and health conditions.
        Womens Health Issues. 2019; 29: S47-S55
        • Bucholz E.M.
        • Krumholz H.M.
        Women in clinical research: What we need for progress.
        Circulation. Cardiovasc Quality and Outcomes. 2015; 8: S1-S3
        • Clayton J.A.
        • Tannenbaum C.
        Reporting sex, gender, or both in clinical research?.
        JAMA. 2016; 316: 1863-1864
        • Danan E.R.
        • Sherman S.E.
        • Clothier B.
        • Burgess D.J.
        • Pinsker E.
        • Joseph A.M.
        • Fu S.F.
        Smoking cessation among female and male veterans before and after a randomized trial of proactive outreach.
        Womens Health Issues. 2019; 29: S15-S23
        • Duan-Porter W.
        • Goldstein K.M.
        • McDuffie J.R.
        • Hughes J.M.
        • Clowse M.E.
        • Klap R.S.
        • Gierisch J.M.
        Reporting of sex effects by systematic reviews on interventions for depression, diabetes, and chronic pain.
        Annals of Internal Medicine. 2016; 165: 184-193
        • Frayne S.M.
        • Carney D.V.
        • Bastian L.
        • Bean-Mayberry B.
        • Sadler A.
        • Klap R.
        • Yee E.F.
        The VA women’s health practice-based research network: Amplifying women veterans’ voices in VA research.
        Journal of General Internal Medicine. 2013; 28: 504-509
        • Frayne S.M.
        • Phibbs C.
        • Saechao F.
        • Friedman S.
        • Shaw J.
        • Romodan Y.
        • Haskel S.
        Sourcebook: Women Veterans in the Veterans Health Administration.
        in: Volume 4: Longitudinal Trends in Sociodemographics, Utilization, Health Profile, and Geographic Distribution. Women’s Health Evaluation Initiative. Women’s Health Services, Veterans Health Administration, Department of Veterans Affairs, Washington, DC2018
        • Freeman A.
        • Stanko P.
        • Berkowitz L.N.
        • Parnell N.
        • Zuppe A.
        • Bale T.L.
        • Epperson C.N.
        Inclusion of sex and gender in biomedical research: Survey of clinical research proposed at the University of Pennsylvania.
        Biology of Sex Differences. 2017; 8: 22
        • Hawkes S.
        • Haseen F.
        • Aounallah-Skhiri H.
        Measurement and meaning: Reporting sex in health research.
        Lancet. 2019; 393: 497-499
        • Hayward R.A.
        • Kent D.M.
        • Vijan S.
        • Hofer T.P.
        Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis.
        BMC Medical Research Methodology. 2006; 6: 18
        • Hernández A.V.
        • Boersma E.
        • Murray G.D.
        • Habbema J.D.F.
        • Steyerberg E.W.
        Subgroup analyses in therapeutic cardiovascular clinical trials: Are most of them misleading?.
        American Heart Journal. 2006; 151: 257-264
        • Humphreys K.
        • Weingardt K.R.
        • Harris A.H.
        Influence of subject eligibility criteria on compliance with National Institutes of Health guidelines for inclusion of women, minorities, and children in treatment research.
        Alcoholism: Clinical and Experimental Research. 2007; 31: 988-995
        • Humphreys K.
        • Williams L.M.
        What can treatment research offer general practice?.
        Lancet Psychiatry. 2018; 5: 295-297
        • Kent D.M.
        • Hayward R.A.
        Limitations of applying summary results of clinical trials to individual patients: The need for risk stratification.
        JAMA. 2007; 298: 1209-1212
        • Kravitz R.L.
        • Duan N.
        • Braslow J.
        Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages.
        Milbank Quarterly. 2004; 82: 661-687
        • Lippman A.
        The inclusion of women in clinical trials: Are we asking the right questions?.
        Women and Health Protection, Toronto, Canada2006
        • Mazure C.M.
        • Jones D.P.
        Twenty years and still counting: Including women as participants and studying sex and gender in biomedical research.
        BMC Womens Health. 2015; 15: 94
        • Naylor J.C.
        • Wagner H.R.
        • Johnston C.
        • Elbogen E.E.
        • Brancu M.
        • Marx C.E.
        • Strauss J.L.
        Pain intensity and pain interference in male and female Iraq/Afghanistan-era veterans.
        Women' Health Issues. 2019; 29: S24-S31
        • Peto R.
        Current misconception 3: That subgroup-specific trial mortality results often provide a good basis for individualising patient care.
        British Journal of Cancer. 2011; 104: 1057-1058
        • Petticrew M.
        • Tugwell P.
        • Kristjansson E.
        • Oliver S.
        • Ueffing E.
        • Welch V.
        Damned if you do, damned if you don't: Subgroup analysis and equity.
        Journal of Epidemiology and Community Health. 2012; 66: 95-98
        • Society for Women's Health Research
        Institute of medicine report validates the science of sex differences.
        Journal of Womens Health and Gender Based Medicine. 2001; 10: 303-304
        • Springer K.W.
        • Stellman J.M.
        • Jordan-Young R.M.
        Beyond a catalogue of differences: A theoretical frame and good practice guidelines for researching sex/gender in human health.
        Social Science & Medicine. 2012; 74: 1817-1824
        • Sugimoto C.R.
        • Ahn Y.-Y.
        • Smith E.
        • Macaluso B.
        • Larivière V.
        Factors affecting sex-related reporting in medical research: A cross-disciplinary bibliometric analysis.
        Lancet. 2019; 393: 550-559
        • Vidaver R.M.
        • Lafleur B.
        • Tong C.
        • Bradshaw R.
        • Marts S.A.
        Women subjects in NIH-funded clinical research literature: Lack of progress in both representation and analysis by sex.
        Journal of Womens Health and Gender Based Medicine. 2000; 9: 495-504
        • Wallach J.D.
        • Sullivan P.G.
        • Trepanowski J.F.
        • Sainani K.L.
        • Steyerberg E.W.
        • Ioannidis J.P.
        Evaluation of evidence of statistical support and corroboration of subgroup claims in randomized clinical trials.
        JAMA Internal Medicine. 2017; 177: 554-560
        • Wizeman T.M.
        Sex-specific reporting of scientific research: A workshop summary.
        National Academies Press, Washington, DC2012
        • Yusuf S.
        • Wittes J.
        • Probstfield J.
        • Tyroler H.A.
        Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials.
        JAMA. 1991; 266: 93-98
        • Zulman D.M.
        • Sussman J.B.
        • Chen X.
        • Cigolle C.T.
        • Blaum C.S.
        • Hayward R.A.
        Examining the evidence: A systematic review of the inclusion and analysis of older adults in randomized controlled trials.
        Journal of General Internal Medicine. 2011; 26: 783-790

      Biography

      Ruth Klap, PhD, is a Research Health Scientist, VA HSR&D Center for the Study of Healthcare Innovation, Implementation and Policy, and National Consortium Program Manager, VA Women's Health Research Network and an Associate Research Sociologist at the UCLA David Geffen School of Medicine.

      Biography

      Keith Humphreys, PhD, is a Senior Research Career Scientist at the VA HSR&D Center on Innovation to Implementation, VA Palo Alto Health Care System, and is the Esther Ting Memorial Professor in the Department of Psychiatry and Behavioral Sciences at Stanford University.