Abstinence Education: Efforts to Assess the Accuracy and
Effectiveness of Federally Funded Programs (03-OCT-06,
GAO-07-87).
Reducing the incidence of sexually transmitted diseases and
unintended pregnancies is one objective of the Department of
Health and Human Services (HHS). HHS provides funding to states
and organizations that provide abstinence-until-marriage
education as one approach to address this objective. GAO was
asked to describe the oversight of federally funded
abstinence-until-marriage education programs. GAO is reporting on
(1) efforts by HHS and states to assess the scientific accuracy
of materials used in these programs and (2) efforts by HHS,
states, and researchers to assess the effectiveness of these
programs. GAO reviewed documents and interviewed HHS officials in
the Administration for Children and Families (ACF) and the Office
of Population Affairs (OPA) that award grants for these programs.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-87
ACCNO: A61784
TITLE: Abstinence Education: Efforts to Assess the Accuracy and
Effectiveness of Federally Funded Programs
DATE: 10/03/2006
SUBJECT: Adolescent health
Education program evaluation
Educational grants
Educational research
Evaluation criteria
Federal funds
Federal grants
Locally administered programs
Policy evaluation
Sexually transmitted diseases
Teenage pregnancy
Assessments
Community and supportive services
programs
Education programs
Government agency oversight
HHS Abstinence Education Program
HHS Adolescent Family Life Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-87
* [1]Results in Brief
* [2]Background
* [3]The State Program
* [4]The Community-Based Program
* [5]The AFL Program
* [6]Funding for Abstinence-until-Marriage Education Has Increase
* [7]Federal Agency Responsibilities Related to Abstinence-until-
* [8]ACF
* [9]OPA
* [10]CDC
* [11]ASPE
* [12]NIH
* [13]Federal and State Efforts to Assess the Scientific Accuracy
* [14]ACF Does Not Review Program Materials for Scientific Accurac
* [15]OPA Reviews Materials Used by AFL Program Grantees for Scien
* [16]A Variety of Efforts Have Been Made to Assess the Effectiven
* [17]HHS, States, and Researchers Have Made a Variety of Efforts
* [18]ACF and OPA Have Required Grantee Reporting of Data on
Outco
* [19]ACF Uses National Data on Adolescent Behavior as a
Measure o
* [20]Some States Have Made Additional Efforts to Assess the
Effec
* [21]ASPE, CDC, and NIH Are Funding Studies Designed to
Assess th
* [22]Other Researchers Have Also Made Efforts to Assess the
Effec
* [23]Several Factors Limit the Conclusions That Can Be Drawn abou
* [24]Experts Have Developed Criteria to Evaluate Efforts to
Asses
* [25]Most Efforts to Assess the Effectiveness of
Abstinence-until
* [26]Most Efforts to Assess the Effectiveness of
Abstinence-until
* [27]Results of Some Scientifically Valid Assessment Efforts
Have
* [28]Conclusions
* [29]Recommendation for Executive Action
* [30]Agency Comments and Our Evaluation
* [31]HRSA Used Full and Open Competition Procedures to Award the
* [32]HRSA Identified No Actual or Potential Organizational Confli
* [33]GAO Contact
* [34]Acknowledgments
* [35]GAO's Mission
* [36]Obtaining Copies of GAO Reports and Testimony
* [37]Order by Mail or Phone
* [38]To Report Fraud, Waste, and Abuse in Federal Programs
* [39]Congressional Relations
* [40]Public Affairs
Report to Congressional Requesters
United States Government Accountability Office
GAO
October 2006
ABSTINENCE EDUCATION
Efforts to Assess the Accuracy and Effectiveness of Federally Funded
Programs
GAO-07-87
Contents
Letter 1
Results in Brief 5
Background 7
Federal and State Efforts to Assess the Scientific Accuracy of Materials
Used in Abstinence-until-Marriage Education Programs Have Been Limited 14
A Variety of Efforts Have Been Made to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs, but a Number of Factors
Limit the Conclusions That Can Be Drawn 19
Conclusions 39
Recommendation for Executive Action 40
Agency Comments and Our Evaluation 40
Appendix I HRSA's Technical Assistance Contract for Abstinence Education
44
Appendix II Methodology for Identifying and Reviewing Research Studies 49
Appendix III Comments from the Department of Health and Human Services 51
Appendix IV GAO Contact and Staff Acknowledgments 57
Tables
Table 1: Definition of Abstinence Education 9
Table 2: Funding Provided by HHS for the Three Main
Abstinence-until-Marriage Education Programs 12
Table 3: ACF's Reporting Requirements for the State Program and the
Community-Based Program, Fiscal Year 2006 21
Abbreviations
ACF Administration for Children and Families AFL Adolescent Family Life
ASPE Office of the Assistant Secretary for Planning and Evaluation CDC
Centers for Disease Control and Prevention FAR Federal Acquisition
Regulation HHS Department of Health and Human Services HIV human
immunodeficiency virus HPV human papillomavirus HRSA Health Resources and
Services Administration NAC National Abstinence Clearinghouse NIH National
Institutes of Health OMB Office of Management and Budget OPA Office of
Population Affairs RFP request for proposal STD sexually transmitted
disease TANF Temporary Assistance for Needy Families
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
October 3, 2006
Congressional Requesters
Preventing sexually transmitted diseases (STD) and unintended pregnancies
among adolescents is an important public health challenge. Although
pregnancy and birth rates among female adolescents in the United States
have been declining since the early 1990s, the rates continue to be high
when compared with those in other industrialized nations. The Centers for
Disease Control and Prevention (CDC) reports that about 141,000 children
were born to girls 17 years old and younger in the United States in 2003.
CDC also reports that STDs disproportionately affect adolescents, with
adolescents and young adults ages 15 to 24 acquiring almost half of the
estimated 19 million new infections each year.
Reducing the incidence of STDs and unintended pregnancies among
adolescents is an important objective for the Department of Health and
Human Services (HHS), which identifies as one of its goals the need to
reduce major threats to the health and well-being of Americans. Among the
efforts it supports to reduce the incidence of STDs and unintended
pregnancies among adolescents, HHS funds abstinence-until-marriage
education programs. Abstinence-until-marriage education programs, also
referred to as abstinence-only education programs, teach adolescents to
abstain from sexual activity until marriage in order to avoid risks of
unintended pregnancy, STDs, and related health problems.^1 The content of
federally funded abstinence-until-marriage programs is required to be
consistent with several principles, such as teaching that a mutually
faithful monogamous relationship in the context of marriage is the
expected standard of human sexual activity, and that abstinence from
sexual activity is the only certain way to avoid STDs.
Abstinence-until-marriage education programs are delivered by a variety of
entities, including schools, human service agencies, faith-based
organizations, youth development groups, and pregnancy crisis centers.
Instructors can incorporate a variety of educational materials into their
abstinence-until-marriage education programs, including textbooks, student
manuals, brochures, slide presentations, and videos.
^1Abstinence-until-marriage education programs also support HHS's
objective to promote family formation and healthy marriages.
The three main federally funded abstinence-until-marriage programs are the
Abstinence Education Program (State Program), which is administered by
HHS's Administration for Children and Families (ACF); the Community-Based
Abstinence Education Program (Community-Based Program), which is also
administered by ACF; and the Adolescent Family Life (AFL) Program, which
is administered by HHS's Office of Population Affairs (OPA) within the
Office of Public Health and Science. Funding provided by HHS for the three
abstinence-until-marriage programs increased from about $73 million in
fiscal year 2001 to about $158 million in fiscal year 2005.
Recent studies have raised concerns about the accuracy of educational
materials used in abstinence-until-marriage education programs and about
the effectiveness of these programs. These studies have reported that some
of the materials used in abstinence-until-marriage education programs
contain, for example, scientifically inaccurate information about anatomy
and physiology as they relate to reproductive health as well as misleading
information about contraceptive failure rates and STDs.^2 State and
federal agencies have also documented inaccuracies in
abstinence-until-marriage educational materials. Further, studies
examining the effectiveness of these programs have reported varied
results. For example, some researchers have reported that
abstinence-until-marriage education programs have resulted in adolescents
reporting less frequent sexual intercourse or fewer sexual partners, while
other researchers have reported that these types of programs did not
affect the frequency of sexual intercourse or were ineffective in delaying
the initiation of sexual intercourse.^3
You asked us to describe certain aspects of the oversight of federally
funded abstinence-until-marriage education programs. Our objectives were
to report on (1) efforts by HHS and states to assess the scientific
accuracy of materials used in abstinence-until-marriage education programs
and (2) efforts by HHS, states, and researchers to assess the
effectiveness of abstinence-until-marriage education programs. You also
asked us to describe how HHS selected a contractor for the
abstinence-until-marriage technical assistance contract that was awarded
in September 2002. This information is provided in appendix I.
^2See K. L. Wilson et al., "A Review of 21 Curricula for
Abstinence-Only-Until-Marriage Programs," The Journal of School Health,
vol. 75, no. 3 (2005), and The Content of Federally Funded Abstinence-Only
Education Programs, United States House of Representatives, Committee on
Government Reform-Minority Staff, Special Investigations Division (2004).
^3See, for example, E. A. Borawski et al., "Effectiveness of
Abstinence-only Intervention in Middle School Teens," American Journal of
Health Behavior, vol. 29, no. 5 (2005), and J. B. Jemmott III, L. S.
Jemmott, and G. T. Fong, "Abstinence and Safer Sex HIV Risk-Reduction
Interventions for African American Adolescents: A Randomized Controlled
Trial," Journal of the American Medical Association, vol. 279, no. 19
(1998).
To describe the efforts by HHS and states to assess the scientific
accuracy of materials used in abstinence-until-marriage education
programs, we reviewed published reports, program announcements, Federal
Register notices, agency Web sites, and other documents related to
abstinence-until-marriage education. We focused our review on efforts
related to the three main federally funded abstinence-until-marriage
education programs administered by HHS, as well as efforts to review the
accuracy of scientific facts included in abstinence-until-marriage
education materials. We did not assess the criteria used to determine the
scientific accuracy of education materials or the quality of the reviews.
We interviewed officials from ACF, the Health Resources and Services
Administration (HRSA), OPA, and CDC. We also interviewed officials from
the 10 states that received the largest share of federal funding (together
accounting for 51 percent of the total funding in fiscal year 2005)
through the State Program for abstinence-until-marriage education.^4
To describe efforts by HHS, states, and researchers to assess the
effectiveness of abstinence-until-marriage education programs, we focused
on efforts that examined the extent to which these programs achieved their
program goals. In general, these goals include teaching adolescents to
abstain from sexual activity until marriage in order to avoid unintended
pregnancies, STDs, and related health problems. As part of our review, we
compared these efforts to the design characteristics that experts have
identified as important for a scientifically valid study of program
effectiveness.^5 We reviewed journal articles and other published reports,
agency budget submissions, program announcements, agency and grantee
performance reports, Federal Register notices, agency Web sites, and other
documents related to abstinence-until-marriage education. (For a more
detailed description of our literature review methodology, see app. II).
We also interviewed officials from ACF, HRSA, OPA, CDC, the National
Institutes of Health (NIH), the Office of the Assistant Secretary for
Planning and Evaluation (ASPE), and 10 states that received the largest
share of federal funding for abstinence-only education through the State
Program in fiscal year 2005. We also interviewed individuals from the
National Campaign to Prevent Teen Pregnancy, The Brookings Institution,
ETR Associates, The Heritage Foundation, and Advocates for Youth, and
researchers from Case Western Reserve University and Columbia University
to obtain general information regarding the state of the research on
abstinence-until-marriage education. We focused our review on efforts to
assess the scientific accuracy of materials and the effectiveness of the
programs during fiscal year 2006, and also reviewed the administration of
the programs back to fiscal year 2001. We also attended conferences
organized by ACF and OPA to learn about training that is provided to
grantees on scientific accuracy and program evaluations.
^4The 10 states that received the largest share of funding in fiscal year
2005 through the State Program were Arizona, Florida, Georgia, Illinois,
Louisiana, Michigan, New York, North Carolina, Ohio, and Texas.
To describe how HHS selected a contractor for the
abstinence-until-marriage technical assistance contract that was awarded
in September 2002, we reviewed the Request for Proposals and other related
contract documents. We also interviewed officials at HRSA, ACF, and the
National Abstinence Clearinghouse about the technical assistance contract.
We performed our work from October 2005 through September 2006 in
accordance with generally accepted government auditing standards.
^5See Douglas Kirby, Emerging Answers: Research Findings on Programs to
Reduce Teen Pregnancy (Washington, D.C.: National Campaign to Prevent Teen
Pregnancy, 2001). The experts identifying the design characteristics of a
scientifically valid study for the National Campaign to Prevent Teen
Pregnancy were drawn from institutions that include the National
Institutes of Health, the Medical Institute for Sexual Health, the Alan
Guttmacher Institute, the Institute for Research and Evaluation, and
various universities. See David Satcher, The National Consensus Process on
Sexual Health and Responsible Sexual Behavior: Interim Report (Atlanta:
Morehouse School of Medicine, 2006). The panel convened by David Satcher
included experts from a variety of organizations, including the Medical
Institute for Sexual Health, the Alan Guttmacher Institute, and the
American Academy of Pediatrics. In addition, characteristics of a
scientifically valid study have been identified by other experts in the
field of evaluation research. For example, see Carol H. Weiss, Evaluation
(Upper Saddle River: Prentice Hall, 1998).
Results in Brief
Efforts by HHS and states to assess the scientific accuracy of materials
used in abstinence-until-marriage education programs have been limited.
This is because ACF--which awards grants through two programs that account
for the largest portion of federal spending on abstinence-until-marriage
education--does not review its grantees' education materials for
scientific accuracy and does not require grantees of either program to
review their own materials for scientific accuracy. In addition, not all
states that receive funding through ACF's State Program have chosen to
review their program materials for scientific accuracy. In particular, 5
of the 10 states in our review conduct such reviews. Officials from these
states reported using a variety of approaches in their reviews. In
contrast, OPA does review the scientific accuracy of AFL grantees'
proposed educational materials and any inaccuracies found must be
corrected before the materials can be used. While the extent to which
federally funded abstinence-until-marriage education materials are
inaccurate is not known, in the course of their reviews OPA and some
states reported that they have found some inaccuracies in
abstinence-until-marriage education materials. For example, OPA has
required that a grantee correct several statements in a true/false
quiz--including statements about STDs and condom use--in order for the
quiz to be approved for use in its curriculum. In addition, one state
official described an instance in which abstinence-until-marriage
materials incorrectly suggested that HIV can pass through condoms because
the latex used in condoms is porous.
HHS, states, and researchers have made a variety of efforts to assess the
effectiveness of abstinence-until-marriage education programs; however, a
number of factors limit the conclusions that can be drawn about the
effectiveness of abstinence-until-marriage education programs. To assess
the effectiveness of their abstinence-until-marriage education programs,
ACF and OPA have required their grantees to report on various outcomes
that the agencies use to measure the effectiveness of grantees'
abstinence-until-marriage education programs. For example, as of fiscal
year 2006, states that receive funding through the State Program are
required to report annually on four measures of the prevalence of
adolescent sexual behavior in their state, such as the rate of pregnancy
among adolescents aged 15 to 17 years. To assess the effectiveness of both
its State and Community-Based Programs, ACF also analyzes trends in
adolescent behavior, as reflected in national data on birth rates among
teens and the proportion of surveyed high school students reporting that
they have had sexual intercourse. OPA requires grantees of the AFL Program
to develop and report on outcome measures that demonstrate the extent to
which grantees' programs are having an effect on program participants. In
addition, other HHS agencies and offices--ASPE, CDC and NIH--are making
efforts to assess the effectiveness of abstinence-until-marriage education
programs. Further, 6 of the 10 states in our review that receive funding
through the State Program have worked with third-party evaluators to
assess the effectiveness of abstinence-until-marriage education programs
in their states. Several factors, however, limit the conclusions that can
be drawn about the effectiveness of abstinence-until-marriage education
programs. Most of the efforts to evaluate the effectiveness of
abstinence-until-marriage education programs that we describe in our
review have not met certain minimum criteria--such as random assignment of
participants and sufficient follow-up periods and sample sizes--that
experts have concluded are necessary in order for assessments of program
effectiveness to be scientifically valid, in part because such designs can
be expensive and time-consuming to carry out. In addition, the results of
efforts that meet the criteria of a scientifically valid assessment have
varied and two key studies funded by HHS that meet these criteria have not
yet been completed. When completed, these HHS-funded studies may add
substantively to the body of research on the effectiveness of
abstinence-until-marriage education programs.
To address concerns about the scientific accuracy of materials used in
abstinence-until-marriage education programs, we recommend that the
Secretary of HHS develop procedures to help assure the accuracy of such
materials used in the State and Community-Based Programs. To help provide
such assurance, the Secretary could consider alternatives such as (1)
extending the approach currently used by OPA to review the scientific
accuracy of the factual statements included in abstinence-until-marriage
education to materials used by grantees of ACF's Community-Based Program
and requiring grantees of ACF's State Program to conduct such reviews or
(2) requiring grantees of both programs to sign written assurances in
their grant applications that the materials they propose using are
accurate.
In commenting on a draft of this report, HHS agreed to consider requiring
grantees of both ACF programs to sign written assurances in grant
applications that the materials they use are accurate. In addition, HHS
noted that all federal grant applicants attest on a standard form that
information in their applications is correct. However, it is not clear
that this serves the purpose of assuring the scientific accuracy of the
educational materials. Further, the curricula to be used are not required
to be included with states' applications. HHS's written comments also
stated that ACF requires that the Community-Based Program curricula
conform to standards that are grounded in scientific literature by
requiring certain types of information. However, the inclusion of certain
types of information does not necessarily ensure the accuracy of the
scientific facts included in the abstinence-until-marriage materials. In
addition, HHS noted in its written comments that we did not define the
term scientific accuracy and stated that it disagreed with certain
findings of the report because it was difficult to precisely determine the
criteria employed by GAO in making the recommendation as to scientific
accuracy. However, the objective of our work was to focus on efforts by
HHS and states to review the accuracy of scientific facts included in
abstinence-until-marriage education materials and not to perform an
independent assessment of the criteria used or the quality of the reviews.
With regard to effectiveness, HHS agreed that it may be too soon to draw
conclusions about the effectiveness of ACF's and OPA's programs.
Background
Recent statistics from CDC show that many high school students engage in
sexual behavior that places them at risk for unintended pregnancy and
STDs. In 2005, 46.8 percent of high school students reported that they
have ever had sexual intercourse, with 14.3 percent of students reporting
that they had had sexual intercourse with four or more persons. The
likelihood of ever having sexual intercourse varied by grade, with the
highest rate among 12th grade students (63.1 percent) and the lowest rate
among 9th grade students (34.3 percent). CDC also has reported that the
prevalence of certain STDs--including the rate of chlamydia infection, the
most frequently reported STD in the United States--peaks in adolescence
and young adulthood. According to CDC, in 2004 the chlamydia rates among
adolescents 15 to 19 years old (1,579 cases per 100,000 adolescents) and
young adults 20 to 24 years old (1,660 cases per 100,000) were each more
than twice the rates among all other age groups.
HHS's current strategic plan includes the objectives to reduce the
incidence of STDs and unintended pregnancies and to promote family
formation and healthy marriages. These two objectives support HHS's goals
to reduce the major threats to the health and well-being of Americans and
to improve the stability and healthy development of American children and
youth, respectively. Abstinence-until-marriage education programs are one
of several types of programs that support these objectives.^6
The three main federal abstinence-until-marriage education programs--the
State Program, the Community-Based Program, and the AFL Program--provide
grants to support the recipients' own efforts to provide
abstinence-until-marriage education at the local level.^7 These programs
must comply with the statutory definition of abstinence education (see
table 1).^8
6HHS's Family Planning Program, for example, also supports the objective
to reduce the incidence of STDs and unintended pregnancies by providing
access to contraceptive supplies and family planning information,
especially for low-income persons, at community health clinics. This
program is authorized under Title X of the Public Health Service Act.
^7There are other federal sources of funding that are used for abstinence
education, such as the Temporary Assistance for Needy Families (TANF)
Program that is administered by ACF. Some states have allocated some of
their TANF funding for abstinence education programs. For example, Florida
has used TANF funds to provide community-based and faith-based
organizations with contracts to carry out abstinence education. Other
sources of funding that are used for abstinence education include ACF's
Compassion Capital Fund and CDC's Division of Adolescent and School Health
grants.
^842 U.S.C. S 710(b)(2). This definition is also referred to as the A-H
definition. This statutory provision defines abstinence education for
purposes of the State Program. Annual appropriations acts and program
announcements have extended this definition to the Community-Based and AFL
Programs. See, e.g , Departments of Labor, Health and Human Services, and
Education, and Related Agencies Appropriations Act, 2006,Pub. L. No.
109-149,119 Stat. 2833, 2855-56.
Table 1: Definition of Abstinence Education
Abstinence education refers to an educational or motivational program
that:
A. has as its exclusive purpose, teaching the social, psychological, and
health gains to be realized by abstaining from sexual activity;
B. teaches abstinence from sexual activity outside marriage as the
expected standardfor all school age children;
C. teaches that abstinence from sexual activity is the only certain way to
avoid out-of-wedlock pregnancy, sexually transmitted diseases, and other
associated health problems;
D. teaches that a mutually faithful monogamous relationship in context of
marriage is the expected standard of human sexual activity;
E. teaches that sexual activity outside of the context of marriage is
likely to have harmful psychological and physical effects;
F. teaches that bearing children out-of-wedlock is likely to have harmful
consequences for the child, the child's parents, and society;
G. teaches young people how to reject sexual advances and how alcohol and
drug use increases vulnerability to sexual advances; and
H. teaches the importance of attaining self-sufficiency before engaging in
sexual activity.
Source: Social Security Act, S 510(b)(2) (codified at 42 U.S.C. S
710(b)(2)).
The State Program
The State Program, administered by ACF, provides funding to its
grantees--states--for the provision of abstinence-until-marriage education
to those most likely to have children outside of marriage.^9 States that
receive grants through the State Program have discretion in how they use
their funding to provide abstinence-until-marriage education. Some require
that organizations apply for funds and use them to administer
abstinence-until-marriage education programs. Others may directly
administer such programs. At their discretion, states may also provide
mentoring, counseling, and adult supervision to adolescents to promote
abstinence from sexual activity until marriage.
The Personal Responsibility and Work Opportunity Reconciliation Act of
1996 established the State Program, and states were awarded grants
beginning in fiscal year 1998.^10 Funds are allotted to each state that
submits the required annual application based on the ratio of the number
of low-income children in the state to the total number of low-income
children in all states. States are required to match every $4 they receive
in federal money with $3 of nonfederal money and are required to report
annually on the performance of the abstinence-until-marriage education
programs that they support or administer. In fiscal year 2005, 47 states,
the District of Columbia, and 3 insular areas were awarded funding.^11,12
^9Through the State Program funds are also provided to insular areas and
the District of Columbia.
^10Pub. L. No. 104-193, S 912; 110 Stat. 2353-54 (codified at 42 U.S.C. S
710).
The Community-Based Program
The Community-Based Program, which is also administered by ACF, is focused
on funding public and private entities that provide
abstinence-until-marriage education for adolescents from 12 to 18 years
old, with the purpose of creating an environment within communities that
supports adolescent decisions to postpone sexual activity until marriage.
The Community-Based Program provides grants for school-based programs,
adult and peer mentoring, and parent education groups. The Community-Based
Program first awarded grants in fiscal year 2001.^13 Grantees of the
Community-Based Program are selected through a competitive process and are
evaluated according to several criteria, such as the extent to which they
have demonstrated that a need exists for abstinence-until-marriage
education for a targeted population or in a specific geographic location.
Grantees are required to report to ACF, on a semiannual basis, on the
performance of their programs. For fiscal year 2005, 63 grants were
awarded to organizations and other entities.^14
11In this report, we refer to U.S. territories and commonwealths as
"insular areas."
^12Some states and insular areas have not applied for funding under the
State Program. California, Maine, and Pennsylvania did not apply for
funding under the State Program in fiscal year 2005. In this report, when
we refer to "states," we are referring to all grantees of the State
Program--including states, insular areas, and the District of Columbia.
^13The Community-Based Program is conducted under section 1110 of the
Social Security Act. See 42 U.S.C. S 1310.
^14In addition to the 63 grants awarded in fiscal year 2005, ACF is also
responsible for other grants that the agency awarded before 2005.
The AFL Program
The AFL Program supports programs that provide abstinence-until-marriage
education.^15 The primary purpose of these programs is to find effective
means of reaching preadolescents and adolescents before they become
sexually active in order to encourage them to abstain from sexual activity
and other risky behaviors. Under the AFL Program, OPA awards competitive
grants to public or private nonprofit organizations or agencies, including
community-based and faith-based organizations, to facilitate
abstinence-until-marriage education in a variety of settings, including
schools and community centers. Established in 1981, the AFL Program began
awarding grants in fiscal year 1982. AFL Program grantees include school
districts, youth development groups, and medical centers. Grant applicants
are evaluated based on several criteria, such as the extent to which they
provide a clear statement of mission, goals, measurable objectives, and a
reasonable method for achieving their objectives. Grantees are required to
conduct evaluations of certain aspects of their programs and report
annually on their performance. As of August 2006, OPA funded 58
abstinence-until-marriage education programs, and most of these were
focused on reaching young adolescents from the ages of 9 to 14.
Funding for Abstinence-until-Marriage Education Has Increased
Funding provided by HHS for abstinence-until-marriage education programs
has increased steadily since 2001 (see table 2). For the three main
programs combined--the State Program, the Community-Based Program, and the
AFL Program--the amount of agency funding increased from about $73 million
in fiscal year 2001 to about $158 million in fiscal year 2005. Nearly all
of this increase was for the Community-Based program; funding under this
program increased by about $84 million from fiscal years 2001 through
2005. In fiscal year 2005, agency funding for the Community-Based Program
constituted the largest share of the total funding (about 66 percent) for
the three main programs combined.
^15See 42 U.S.C. S 300z et seq. The AFL Program also supports other
projects for pregnant and parenting adolescents, their infants, male
partners, and family members. The purpose of these projects is to improve
the outcomes of early childbearing for teen parents, their infants, and
their families. However, in this report, when we use the term "AFL
Program," we are referring only to the abstinence-until-marriage component
of the AFL Program.
Table 2: Funding Provided by HHS for the Three Main
Abstinence-until-Marriage Education Programs
Dollars in millions
Fiscal year State Program^a,b Community-Based Program AFL Program
2001 $43 $20 $10
2002 43 40 12
2003 44 55 13
2004 41 75 13
2005 41 104 13
Sources: ACF, OPA, and HRSA.
Notes: Figures are rounded to nearest $1,000,000. Funding levels represent
the total amount of grants awarded and funding for program-related
support, such as technical assistance and evaluation.
aStates that receive funding are required to match every $4 they receive
of federal funds with $3 of nonfederal money.
bThe amount of funding provided by HHS for the State Program has generally
varied by year because the states that have applied for funding each year
have varied.
Within each of the three main abstinence-until-marriage education
programs, the amount of individual grants varied.^16 In fiscal year 2005,
the State Program's annual grants ranged from $57,057 to $4,777,916 and
the median annual grant amount was $569,675. That same year, the
Community-Based Program's annual grants ranged from $213,276 to $800,000
and the median grant amount was $642,250. In fiscal year 2006, the AFL
Program's annual grants ranged from $95,676 to $300,000 and the median
grant amount was $225,000.
Federal Agency Responsibilities Related to Abstinence-until-Marriage Education
Five organizational units located within HHS--ACF, OPA, CDC, ASPE, and
NIH--have responsibilities related to abstinence-until-marriage education.
ACF and OPA administer the three main federal abstinence-until-marriage
education programs. CDC supports abstinence-until-marriage education at
the national, state, and local levels. CDC, ASPE, and NIH are sponsoring
research on the effectiveness of abstinence-until-marriage programs.
^16ACF awards formula grants under the State Program each year, and states
have 2 years to spend the funds they are awarded. In the Community-Based
Program and AFL Program, grantees develop multiyear projects--up to 5
years--for which the first year of funding is provided through competitive
grants; for subsequent years, grantees may obtain funding through
noncompetitive continuation grants.
ACF
ACF is responsible for federal programs that promote the economic and
social well-being of families, children, individuals, and communities. ACF
administers and provides oversight of both the State Program and the
Community-Based Program by, among other things, awarding grants, providing
training and technical assistance to grantees, and requiring annual
performance reporting from grantees. ACF has been responsible for the
State Program since June 2004 and the Community-Based Program since
October 2005. HRSA previously administered these programs.
OPA
OPA has responsibility for advising the Secretary of HHS on a wide range
of reproductive health topics, including adolescent pregnancy and family
planning. The office is also responsible for administering programs that
provide services for pregnant and parenting teens and prevention programs,
such as abstinence-until-marriage education programs. OPA administers and
provides oversight of the AFL Program by awarding grants, providing
training and technical assistance to grantees, and requiring annual
performance reporting from grantees.
CDC
CDC is primarily responsible for the prevention and control of infectious
and chronic diseases, including STDs. CDC provides funding to state and
local education agencies in their efforts to support comprehensive school
health education and HIV/STD prevention education programs, and CDC
officials told us that some of these are focused on abstinence. CDC also
provides funding to several state education agencies to implement various
abstinence projects, such as collaboration-building among agencies to
increase the impact of their efforts to encourage abstinence. Further, CDC
develops tools to assist state and local education agencies with their
health education programs. CDC provides funding to several national
organizations to build the capacity of abstinence-until-marriage education
providers. Organizations' activities include, but are not limited to, the
development and distribution of educational materials. CDC is also
sponsoring research on the effectiveness of an abstinence-until-marriage
education program.
ASPE
ASPE advises the Secretary of HHS in several areas, including policy
development in health, human services, data, and science. ASPE is
responsible for the development of policy analyses and it conducts
research and evaluation studies in several areas, including the health of
children and adolescents. ASPE is currently sponsoring research on the
effectiveness of abstinence-until-marriage education programs.
NIH
NIH is the primary federal agency that conducts and supports medical and
behavioral research among various populations, including children and
adolescents. NIH is currently sponsoring research on the effectiveness of
abstinence-until-marriage education programs.
Federal and State Efforts to Assess the Scientific Accuracy of Materials Used in
Abstinence-until-Marriage Education Programs Have Been Limited
Efforts by HHS and states to assess the scientific accuracy of materials
used in abstinence-until-marriage education programs have been limited.
ACF--which awards grants to two programs that account for the largest
portion of federal spending on abstinence-until-marriage education--does
not review its grantees' education materials for scientific accuracy and
does not require grantees of either program to review their own materials
for scientific accuracy. In addition, not all states funded through the
State Program have chosen to review their program materials for scientific
accuracy. In contrast to ACF, OPA has reviewed the scientific accuracy of
grantees' proposed educational materials and corrected inaccuracies in
these materials.
ACF Does Not Review Program Materials for Scientific Accuracy and Does Not
Require Grantees to Do So, though Some State Grantees Have Conducted Such
Reviews
There have been limited efforts to review the scientific accuracy of
educational materials used in ACF's State and Community-Based
Programs--the two programs that account for the largest portion of federal
spending on abstinence education. ACF does not review materials for
scientific accuracy in either reviewing grant applications^17 or in
overseeing grantees' performance. Prior to fiscal year 2006, State Program
and Community-Based Program applicants were not required to submit copies
of their proposed educational materials with their applications. While ACF
required grantees of the Community-Based Program--but not the State
Program--to submit their educational materials with their fiscal year 2006
applications, ACF officials told us that grantee applications and
materials are only reviewed to ensure that they address all aspects of the
scope of the Community-Based Program, such as the A-H definition of
abstinence education.^18 Further, documents provided to us by ACF indicate
that the agency does not review grantees' educational materials for
scientific accuracy as a routine part of its oversight activities. In
addition, ACF also does not require its grantees to review their own
materials for scientific accuracy. Similarly, when HRSA was responsible
for the State and Community-Based Programs, the agency did not review
materials used by grantees for scientific accuracy or require grantees to
review their own materials.
^17In reviewing grantees applications, ACF does examine several issues,
including applicants' stated program goals and need for assistance, their
compliance with the A-H definition of abstinence education, their intended
approach in carrying out their objectives, and their budget plan.
^18HHS officials told us that if ACF finds inaccurate statements during
this more general review process or if inaccuracies are brought to their
attention at any time during the grant period, ACF officials work with the
grantees to take corrective action.
Not all grantees of the State Program have chosen to review the scientific
accuracy of their educational materials. Officials from 5 of the 10 states
in our review reported that their states have chosen to conduct such
reviews.^19,20 Officials in these states identified a variety of reasons
why their states reviewed abstinence-until-marriage educational materials,
including program requirements, state education laws and guidelines, and
past lawsuits, to ensure that materials used in abstinence-until-marriage
programs were accurate. For example, Michigan's Revised School Code states
that materials and instruction in the sex education curricula, including
information on abstinence, "shall not be medically inaccurate,"^21 and
Ohio's fiscal year 2007 abstinence-until-marriage education program
guidance states that abstinence-until-marriage educational materials
"should be medically accurate in all assertions."^22
The five states we contacted that review abstinence-until-marriage
educational materials for scientific accuracy have used a variety of
approaches in their reviews. Some states contracted with medical
professionals--such as nurses, gynecologists, and pediatricians--to serve
as medical advisors who review program materials and use their expertise
to determine what is and is not scientifically accurate. Some states have
created checklists or worksheets to guide their staff conducting the
review and document findings of inaccuracy or verification of a statement.
All five states use medical professionals in conducting these reviews. One
of the states requires that all statistics or scientific statements cited
in a program's materials are sourced to CDC or a peer-reviewed medical
journal. Officials from this state told us that if statements in these
materials cannot be attributed to these sources, the statements are
required to be removed until citations are provided and materials are
approved. Officials from this state told us they have also supplemented
their review of program materials with on-site classroom observations to
assess the scientific accuracy of the information presented to students.
^19In addition to reviewing materials for accuracy, one state requires
abstinence-until-marriage providers to sign a written assurance that their
materials are scientifically accurate. Officials from this state also
reported providing abstinence-until-marriage education programs with
public health consultants to provide technical assistance and training to
help ensure the accuracy of their educational materials.
^20In addition, some state officials we interviewed told us that review
committees for local school districts may review the scientific accuracy
of educational materials that include information about HIV and other
STDs, including abstinence-until-marriage education materials.
^21Mich. Comp. Laws Ann. S 380.1507b(2)(West 2004).
^22See Ohio Department of Health, "Abstinence Education Program Request
for Proposals for Fiscal Year 2007," (program announcement, 2005).
Officials from two of the five states reported that they have found
inaccuracies as a result of their reviews. For example, one state official
stated that because information is constantly evolving, state officials
have had to correct out-of-date scientific information. In addition, this
official cited an instance where materials incorrectly suggested that HIV
can pass through condoms because the latex used in condoms is porous. In
addition, this official provided documentation that the state has had to
correct a statement indicating that when a person is infected with the
human papillomavirus,^23 the virus is "present for life" because, in
almost all cases, this is not true. State officials who have identified
inaccuracies told us that they informed their grantees of inaccuracies so
that they could make corrections in their individual programs. One state
official added that she contacted the authors of the materials to report
an inaccuracy.
Some of the educational materials that states have reviewed are materials
that are commonly used in the Community-Based Program. Officials from four
of the five states that review materials for scientific accuracy told us
that they have each reviewed at least one of the five curricula most
commonly used in the Community-Based Program because programs in their
state were using them: Choosing the Best, WAIT Training, Sex Can Wait,
A.C. Green's Game Plan Abstinence Program, and Worth the Wait. Based on
ACF documents, we found that there were 58 different curricula used by
grantees of the Community-Based Program in fiscal year 2005. However, more
than half of the grantees of the Community-Based Program reported using at
least one of these five curricula.^24
While there has been limited review of materials used in the State and
Community-Based Programs, grantees of these programs have received some
technical assistance designed to improve the scientific accuracy of their
materials. For example, ACF officials reported that the agency provided a
conference for grantees of the Community-Based Program in February 2006
that included a presentation focused on medical accuracy, including a
discussion of state legislative proposals that would require medical
accuracy in abstinence-until-marriage education, and how to identify
reliable data. In addition, in 2002, HRSA awarded a contract to the
National Abstinence Clearinghouse requiring, among other things, that the
contractor develop and implement a program to provide medically accurate
information and training to grantees of the State and Community-Based
Programs.^25 (See app. I for a description of HRSA's process for awarding
this contract). The portion of the contract that focused on providing
medically accurate information to grantees was subcontracted to the
Medical Institute for Sexual Health (Medical Institute),^26 which has
conducted presentations at regional educational conferences to provide
grantees with medical and scientific information, such as updated
information on condoms and STD transmission. The Medical Institute has
also provided consultative services to grantees by responding to medical
and scientific questions.
^23The human papillomavirus (HPV) causes an STD called genital HPV
infection.
^24Some grantees of the Community-Based Program reported using more than
one of these curricula in fiscal year 2005.
OPA Reviews Materials Used by AFL Program Grantees for Scientific Accuracy
In contrast to ACF, OPA reviews for scientific accuracy the educational
materials used by AFL Program grantees. Specifically, OPA reviews its
grantees' proposed educational materials for scientific accuracy before
they are used. Agency officials stated that they began to review these
materials while litigation concerning the AFL Program was ongoing. OPA
continued to review these materials as part of a 1993 settlement to this
lawsuit.^27 The settlement agreement expired in 1998, though the agency
has continued to review grantees' proposed educational materials for
accuracy as a matter of policy. OPA officials told us that grant
applicants submit summaries of materials they propose to use, though the
materials are not reviewed for scientific accuracy until after grantees
have been selected. OPA officials said that after grants are awarded, a
medical education specialist (in consultation with several part-time
medical experts) reviews the grantees' printed materials and other
educational media, such as videos. OPA officials explained that the
medical education specialist must approve all materials before they are
used. On many occasions, OPA grantees have proposed using--and therefore
OPA has reviewed--materials commonly used in the Community-Based Program.
For example, an OPA official told us that the agency had reviewed three of
the Community-Based Program's commonly used curricula--Choosing the Best,
Sex Can Wait, and A.C. Green's Game Plan Abstinence Program--and is also
currently reviewing another curriculum commonly used by Community-Based
Program grantees, WAIT Training.^28
25The administration of this contract was transferred to ACF in May 2005.
^26The Medical Institute is a nonprofit organization that provides
educational resources, conferences, and seminars to educators, health
professionals, pregnancy care centers, and faith-based groups about
behaviors to decrease STDs and out-of-wedlock pregnancies, including
abstinence.
^27See generally Bowen v. Kendrick, 487 U.S. 589 (1988), 657 F. Supp 1547
(D.D.C. 1987).
OPA officials stated that the medical education specialist has
occasionally found and addressed inaccuracies in grantees' proposed
educational materials. OPA officials stated that these inaccuracies are
often the result of information being out of date because, for example,
medical and statistical information on STDs changes frequently. OPA has
addressed these inaccuracies by either not approving the materials in
which they appeared or correcting the materials through discussions with
the grantees and, in some cases, the authors of the materials. In fiscal
year 2005, OPA disapproved of a grantee using a specific pamphlet about
STDs because the pamphlet contained statements about STD prevention and
HIV transmission that were considered incomplete or inaccurate. For
example, the pamphlet stated that there was no cure for hepatitis B, but
the medical education specialist required the grantee to add that there
was a preventive vaccine for hepatitis B. In addition, OPA required that a
grantee correct several statements in a true/false quiz--including
statements about STDs and condom use--in order for the quiz to be approved
for use. For example, the medical education specialist changed a sentence
from "The only 100% effective way of avoiding STDs or unwanted pregnancies
is to not have sexual intercourse." to "The only 100% effective way of
avoiding STDs or unwanted pregnancies is to not have sexual intercourse
and engage in other risky behaviors."
^28In addition, a CDC official told us that some of its grantees are
producing educational materials with CDC funds to be used by
abstinence-until-marriage education programs, which are likely to include
State and Community-Based Program grantees. These materials are required
to be reviewed for scientific accuracy. CDC officials told us that they
have made corrections to some of these materials. Materials used in
school-based HIV prevention education programs that are supported with CDC
funds are also reviewed for scientific accuracy. A CDC official told us
that some of these programs are abstinence-until-marriage education
programs.
While OPA and some states have reviewed their grantees'
abstinence-until-marriage education materials for scientific accuracy,^29
these types of reviews have the potential to affect
abstinence-until-marriage education providers more broadly. Such efforts
may create an incentive for authors of abstinence-until-marriage education
materials to ensure they are accurate. Thus, some authors of
abstinence-until-marriage education materials have recently updated
materials in their curricula following reports that questioned their
accuracy. For example, one of the most widely used curricula used by
grantees of the Community-Based Program--WAIT Training--has been recently
updated and provides the updated information on its Web site. A
representative from WAIT Training stated that the company recently revised
its curriculum, in part, in response to a congressional review that found
inaccuracies in its abstinence-until-marriage education materials.
A Variety of Efforts Have Been Made to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs, but a Number of Factors Limit the
Conclusions That Can Be Drawn
HHS, states, and researchers have made a variety of efforts to assess the
effectiveness of abstinence-until-marriage education programs; however, a
number of factors limit the conclusions that can be drawn about the
effectiveness of these programs. ACF and OPA have required their grantees
to report on various outcomes used to measure the effectiveness of
grantees' abstinence-until-marriage education programs,^30 though the
reporting requirements for each of the three abstinence-until-marriage
programs differ. In addition, to assess the effectiveness of the State and
Community-Based Programs, ACF has analyzed national data on adolescent
birth rates and the proportion of adolescents who report having had sexual
intercourse. Other organizational units within HHS--ASPE, CDC, and
NIH--are funding studies designed to assess the effectiveness of
abstinence-until-marriage education programs in delaying sexual
initiation, reducing pregnancy and STD rates, and reducing the frequency
of sexual activity. Despite these efforts, several factors limit the
conclusions that can be drawn about the effectiveness of
abstinence-until-marriage education programs. Most of the efforts to
evaluate the effectiveness of abstinence-until-marriage education programs
that we describe in our review have not met certain minimum criteria that
experts have concluded are necessary in order for assessments of program
effectiveness to be scientifically valid, in part because such designs can
be expensive and time-consuming to carry out. In addition, the results of
some efforts that meet the criteria of a scientifically valid assessment
have varied, and two key studies that meet these criteria have not yet
been completed.
^29In addition to OPA and some states, others have also reviewed the
scientific accuracy of abstinence-until-marriage education materials. See,
for example, Wilson et al.
^30This reporting is a part of ACF's efforts to collect evaluative
information about these programs. These efforts include both performance
measurement--the ongoing monitoring and reporting of program
accomplishments toward preestablished goals--and program
evaluation--individual systematic studies to assess how well a program is
working. Both types of assessments aim to support decisions to improve
service delivery and program effectiveness. See GAO, Performance
Measurement and Evaluation: Definitions and Relationships,
[41]GAO-05-739SP (Washington, D.C.: May 2005), for more information on
types of assessments.
HHS, States, and Researchers Have Made a Variety of Efforts to Assess the
Effectiveness of Abstinence-until-Marriage Education Programs
Efforts of HHS, states, and researchers to assess the effectiveness of
abstinence-until-marriage education programs have included ACF and OPA
requiring grantees to report data on outcomes of their
abstinence-until-marriage education programs; ACF analyzing national data
on adolescent behavior and birth rates; and other HHS agencies, states,
and researchers funding or conducting studies to assess the effectiveness
of abstinence-until-marriage education programs.
ACF and OPA Have Required Grantee Reporting of Data on Outcomes
ACF has made efforts to assess the effectiveness of
abstinence-until-marriage education programs funded by the State Program
and the Community-Based Program. One of ACF's efforts has been to require
grantees of both programs to report data on outcomes, though the two
programs have different requirements for the outcomes grantees must
report. For the State Program, as of fiscal year 2006, grantees must
report annually on four measures of the prevalence of adolescent sexual
behavior in their states, such as the rate of pregnancy among adolescents
aged 15 to 17 years, and compare these data to program targets over 5
years. To report on these four measures, states may choose the data
sources they will use.^31 States must also develop and report on two
additional performance measures that are related to the goals of their
programs.^32 (See table 3 for a list of ACF's fiscal year 2006 reporting
requirements for the State Program.)
^31Previously, to report on the four measures, states have relied on
either state or national data sources, such as CDC's Youth Risk Behavior
Surveillance System.
^32For example, in fiscal year 2002, state grantees developed such
measures as the percentage of teens surveyed who show an increase in
participating in structured activities after school hours; the percentage
of live births to women younger than 18, fathered by men age 20 and older;
the percentage of program participants proficient in refusal skills; the
percentage of high school students who reported using drugs or alcohol
before intercourse; and the percentage of high school students who had
sexual intercourse for the first time before age 13.
State Program Community-Based Program
o Report on four o Report on program goals that are
performance measures: (1) developed by grantees with a third-party
rate of pregnancy among evaluator. Such outcomes could include
female teenagers aged 15 changes in knowledge about abstinence or
to 17, (2) proportion of declared behavior among participants of
adolescents who have abstinence-until-marriage programs.
engaged in sexual
intercourse, (3) incidence
of youths 15 to 19 years
old who have contracted
selected STDs, and (4)
rate of births among
female teenagers aged 15
to 17.
o Develop and report on o Report on program "outputs": the number
two additional performance of youth served, the hours of service
measures that are related provided to each youth, and the number of
to individual program youths who complete the program. Grantees
goals. Past examples of choose additional outputs that allow for
these additional measures effective monitoring and management of
have included the the project. The additional outputs may
percentage of high school include tracking the number of staff
students who reported trained to provide services, the number
using drugs or alcohol of events hosted, number of marketing
before intercourse and the materials distributed, and so forth.
percentage of high school
students who had sexual
intercourse for the first
time before age 13.
Table 3: ACF's Reporting Requirements for the State Program and the
Community-Based Program, Fiscal Year 2006
Sources: State and Community-Based Programs' announcements, fiscal year
2006.
As of fiscal year 2006, ACF requires Community-Based Program grantees to
develop and report on outcome measures designed to demonstrate the extent
to which grantees' community-based abstinence education programs are
accomplishing their program goals.^33 ACF requires grantees of the
Community-Based Program to contract with third-party evaluators, who are
responsible for both helping grantees develop the outcome measures and
monitoring grantee performance against the measures,^34 but because this
is a new requirement established for fiscal year 2006 grantees, ACF has
not yet received the results of these evaluations. In addition to outcome
reporting, ACF requires grantees of the Community-Based Program to report
on program "outputs," which measure the quantity of program activities and
other deliverables, such as the number of participants who are served by
the abstinence-until-marriage education programs. According to ACF
officials, the agency requires grantees of both the State Program and the
Community-Based Program to report on program outcomes in order to monitor
grantees' performance, target training, and technical assistance, and help
grantees improve service delivery. (See table 3 for a list of ACF's fiscal
year 2006 reporting requirements for the Community-Based Program.)
^33The fiscal year 2006 program announcement for the Community-Based
Program provides examples of outcome measures that grantees could use,
including increased knowledge of the benefits of abstinence, the number of
youths who commit to abstaining from premarital sexual activity, and
increased knowledge of how to avoid high-risk situations and risk
behaviors.
^34Fiscal year 2006 Community-Based Program grantees are required to
devote a minimum of 15 percent of their requested budgets to performance
monitoring by third-party contractors.
ACF's fiscal year 2006 reporting requirements for grantees of the State
Program are the same as HRSA's when it administered the State Program. In
contrast, ACF's fiscal year 2006 reporting requirements for the
Community-Based Program differ from HRSA's reporting requirements for the
program. For example, for Community-Based Program grants awarded in fiscal
year 2001, HRSA required^35 grantees to report on the effectiveness of
their programs, as measured by program participation as well as behavioral
and biological outcomes.^36 These performance measures were modified for
fiscal year 2002, in part HHS officials explained, because of concerns
expressed by members of the abstinence-education community that the
original performance measures did not accurately reflect the efforts of
the grantees of the Community-Based Program. For grants awarded from
fiscal years 2002 through 2004, HRSA required grantees of the
Community-Based Program to report on a combination of program outputs,
such as the proportion of adolescents who completed an
abstinence-until-marriage education program, and measures of adolescent
intentions, such as the proportion of adolescents who committed to
abstaining from sexual activity until marriage.^37 For grants awarded in
fiscal year 2005, when ACF assumed responsibility for the Community-Based
Program from HRSA, grantees were not required to report on any specific
performance measures.
^35Some grantees of the Community-Based Program may have to meet reporting
requirements established by HRSA. Grants under this program are awarded
for projects that may extend over a period of several years. Grantees that
were awarded grants when HRSA administered the program and have since
received noncompetitive continuation grants for these projects are
required to meet the reporting requirements in place at the time they
first received the competitively awarded grants.
^36In fiscal year 2001, HRSA required grantees of the Community-Based
Program to report on the following four performance measures: the
proportion of program participants who successfully complete or remain
enrolled in an abstinence-only education program; the proportion of
program participants who have engaged in sexual intercourse; the
proportion of program participants who report a reduction in risk
behaviors, such as tobacco, alcohol, and drug use; and the rate of births
to female program participants.
OPA has also made efforts to assess the effectiveness of the AFL Program.
Specifically, OPA requires grantees of the AFL Program to develop and
report on outcome measures that are used to help demonstrate the extent to
which grantees' programs are having an effect on program
participants.^38,39 According to OPA officials, the agency recommends that
grantees report on outcome measures, such as participants' knowledge of
the benefits of abstinence and their reported intentions to abstain from
sexually activity, reported beliefs in their ability to remain abstinent,
and reported parental involvement in their lives. To collect data on these
outcome measures and any others, OPA requires all grantees funded in
fiscal year 2004 and beyond to administer, at a minimum, a standardized
questionnaire--developed by OPA--to their program participants, both when
participants begin an abstinence-only education program and after the
program's completion.^40 The standardized questionnaire includes questions
intended to obtain information on participants' reported involvement in
extracurricular activities, behaviors linked to health risks, attitudes
and intentions about abstinence, and opinions about the consequences of
premarital sexual activity. Like ACF, OPA requires its grantees to
contract with independent evaluators, such as colleges or universities,
which are responsible for evaluating the effectiveness of grantees'
individual abstinence-until-marriage education programs.^41 In addition to
evaluating the extent to which grantees are meeting their goals, OPA
officials stated that the independent evaluators may also provide input to
grantees of the AFL Program on other aspects of the program to improve
their service delivery. Unlike ACF, OPA requires that the third-party
evaluations incorporate specific methodological characteristics, such as
control groups or comparison groups^42 and sufficient sample sizes.^43 In
addition, OPA requires that the evaluations for grantees funded in fiscal
year 2004 and beyond account for baseline and follow-up data obtained from
the standardized questionnaires.
^37Specifically, HRSA required grantees of the Community-Based Program to
report annually on the following six performance measures: the proportion
of program participants who successfully completed or remained enrolled in
an abstinence-only education program; the proportion of adolescents who
understood that abstinence from sexual activity is the only certain way to
avoid out-of-wedlock pregnancy and STDs; the proportion of adolescents who
indicated an understanding of the social, psychological, and health gains
to be realized by abstaining from premarital sexual activity; the
proportion of participants who reported that they had the skills necessary
to resist sexual urges and advances; the proportion of youth who committed
to abstaining from sexual activity until marriage; and the proportion of
participants who intended to avoid situations and risk behaviors, such as
drug use and alcohol consumption, which make them more vulnerable to
sexual advances and urges.
^38In addition to these outcomes, grantees of the AFL Program are required
to report on program outputs, such as the number of program participants,
the average number of participants per session, and the average number of
sessions attended by participants. Agency officials stated that OPA has
implemented a new format for its grantees' reports, which is intended to
standardize their reporting on these outputs.
^39OPA's grantees are required to perform evaluations of their programs
that are directly tied to their program objectives. For these evaluations,
OPA requires grantees to develop research hypotheses that reflect the
outcomes the grantees intend to achieve. This type of evaluation is
generally considered to be an outcome evaluation--which assesses the
extent to which a program achieves its outcome-oriented objectives. These
evaluations focus on outputs and outcomes to judge program effectiveness
but may also assess program process to understand how outcomes are
produced. In addition, grantees of the AFL Program are required to perform
implementation evaluations.
OPA's requirement that grantees use a standardized set of questionnaires,
with data from these questionnaires used in evaluations, differs from
OPA's previous requirements. Previously, grantees of the AFL Program were
not required to use a standardized method for collecting data that could
be used to assess the effectiveness of their programs; instead, grantees
chose their own data collection instruments. As a result, an OPA official
explained, the collected data varied from one project to another. OPA
officials said that the agency developed the standardized questionnaire to
ensure uniformity in the data collected and allow the agency to more
effectively aggregate the data reported in evaluations of individual
abstinence-until-marriage education programs.
^40OPA officials stated that grantees may also supplement the standardized
questionnaire with additional data collection instruments to obtain
information on the effectiveness of their abstinence-until-marriage
education programs.
^41OPA has required that its grantees perform independent evaluations of
their programs since the program first awarded grants in 1982, and
requires that grantees devote from 1 percent to 5 percent of grant funds
to the evaluation of their programs. In cases where a more rigorous or
comprehensive evaluation is proposed, OPA may allow these grantees to use
up to 25 percent of their grant funds.
^42A control group is a group of individuals or communities in a study
that is compared to an intervention group--a group in a study that is
receiving or participating in the program being studied. A control group
is a randomly assigned group that does not receive the program. A
comparison group is not randomly assigned like a control group. However,
individuals or communities in well-matched comparison groups should have
similar characteristics.
^43Specifically, OPA requires that third-party evaluations of grantees of
the AFL Program compare, when possible, randomized control or matched
comparison groups with groups receiving abstinence-until-marriage
education. In addition, OPA requires that these evaluations include a
sufficient sample size to ensure that any observed differences between the
groups are statistically valid and that the evaluations include a
follow-up assessment of program participants at least 6 months after the
abstinence-until-marriage intervention has been tested.
OPA officials told us that they plan to aggregate information from certain
questions in the standardized set of questionnaires in order to report on
certain performance measures as part of the agency's annual performance
reports.^44 The measures include the extent of parental involvement in
adolescents' lives and the extent to which adolescents understand the
benefits of abstinence. An agency official stated that the agency expects
to begin receiving data from grantees that are using these questionnaires
in January 2007. OPA did not previously have long-term measures of the
performance of the AFL Program. Its current measures were developed in
collaboration with the Office of Management and Budget (OMB) in response
to an OMB review in 2004 that found that the AFL Program did not have any
annual performance measures for measuring progress toward long-term goals.
In addition to requiring their grantees to report on outcomes used to
assess program effectiveness, both ACF and OPA have provided technical
assistance and training to their grantees in order to support grantees'
own program evaluation efforts. For example, in November 2005 the two
agencies sponsored an evaluation conference for abstinence-until-marriage
grantees that included presentations about evaluations and their
methodology. Similarly, ACF's Office of Planning, Research, and Evaluation
sponsors annual evaluation conferences, and an ACF official told us that a
recent conference placed "a significant emphasis" on the evaluation of
abstinence-until-marriage education programs. In addition, HHS officials
told us that ACF, along with ASPE, is funding a multiyear project that is
designed to identify gaps in abstinence education evaluation and technical
assistance needs, develop materials on abstinence education evaluation,
deliver technical assistance and capacity-building activities related to
program evaluation, and develop research reports related to abstinence
education. OPA officials also told us that they attempt to help ensure
grantees' progress and effectiveness by offering various technical
assistance workshops and conferences. For example, in May 2006 OPA
provided a 2-day training conference to its grantees on the importance of
program evaluations and administering evaluation instruments. In addition,
OPA officials stated that the agency contracts with evaluation
consultants, who review grantees' evaluation tools and activities. OPA
officials explained that these consultants provide in-depth technical
assistance to grantees on how to improve grantees' evaluations.
^44OPA prepares annual performance reports as a part of HHS's
responsibilities under the Government Performance and Results Act, which
include program performance measures to help link funding decisions with
performance and review of related outcome measures.
ACF Uses National Data on Adolescent Behavior as a Measure of the Overall
Effectiveness of Its Two Programs
Requiring outcome reporting from state and community-based grantees is not
ACF's only effort to assess the effectiveness of its two programs. ACF
also analyzes trends in adolescent behavior, as reflected in national data
on birth rates among teens and the proportion of surveyed high school
students reporting that they have had sexual intercourse.^45 ACF uses
these national data as a measure of the overall effectiveness of its State
and Community-Based Programs, comparing the national data to program
targets. In its annual performance reports, the agency summarizes the
progress being made toward lowering the rate of births to unmarried
teenage girls and the proportion of students (grades 9-12) who report
having ever had sexual intercourse.
ACF's use of national data to assess the effectiveness of the State and
Community-Based Programs represents a change from how HRSA assessed the
overall effectiveness of these programs. Whereas ACF compares national
data on adolescent behavior to program targets, HRSA aggregated data from
its state and community-based grantees. HRSA's state grantees were allowed
to select the data sources used to gauge their progress against certain
performance measures. For example, in its annual performance reports on
the State Program, HRSA reported information on the percentage of its
state grantees meeting target rates for reducing the proportion of
adolescents who have engaged in sexual intercourse, the incidence of
youths aged 15 to 19 who have contracted selected STDs, and the rate of
births among youths aged 15 to 17. To determine their progress in meeting
their target rates, some state grantees, for example, reported national
data from the Youth Risk Behavior Surveillance System, while other
grantees reported state-collected data. After ACF assumed responsibility
for the State and Community-Based Programs from HRSA, ACF began using
national data on adolescent behavior as a measure of the programs'
effectiveness. According to ACF officials, the agency changed how it
assessed its programs out of concern over the quality of the data state
grantees were using in their performance reporting and because the agency
wanted to use parallel measures of effectiveness for both programs. For
example, according to state performance reports for fiscal year 2001 that
we reviewed, two reports did not include adolescent pregnancy rates that
year because the states did not collect data on abortions among this
population.^46 In addition, ACF officials told us that they decided not to
use national data on STDs as a measure of program effectiveness because
the goal of reducing STD rates is not as central to the State and
Community-Based Programs as reducing sexual activity and birth rates among
teens. However, one official stated that reducing STDs is an important
"by-product" of the programs.
^45Data on teen birth rates and adolescents' reported sexual behavior are
contained in the National Vital Statistics System and the Youth Risk
Behavior Surveillance System, respectively. The former is a national data
set of public health statistics reported by states to CDC, and the latter
is a national data set based on nationwide surveys administered to high
school students by CDC.
Some States Have Made Additional Efforts to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs
Some states have made additional efforts to assess the effectiveness of
abstinence-until-marriage education programs, although they are not
required by ACF.^47 Specifically, we found that 6 of the 10 states in our
review that receive funding through ACF's State Program have made efforts
to conduct evaluations of selected abstinence-until-marriage programs in
their state. All 6 of the states worked with third-party evaluators, such
as university researchers or private research firms, to perform the
evaluations, which in general measure self-reported changes in program
participants' behavior and attitudes related to sex and abstinence as
indicators of program effectiveness. To obtain this information, the
third-party evaluators have typically relied on surveys administered to
program participants at the start of a program, its conclusion, and during
a follow-up period anywhere from 3 months to almost 3 years after the
conclusion. The third-party evaluations for 4 of the 6 states in our
review have been completed as of February 2006, and the results of these
studies have varied.^48
46In order to estimate pregnancy rates among adolescents, states use data
on both birth rates and abortions among adolescents.
^47ACF does require grantees of the State Program to describe, in their
grant applications, their plans for "tracking activities and measuring
achievement" of their program goals and objectives.
Among the 4 states that have completed third-party evaluations, 3 states
require the abstinence programs in their state to measure reported changes
in participants' behavior as an indicator of program effectiveness--both
at the start of the program and after its completion. The 3 states require
their programs to track participants' reported incidence of sexual
intercourse. In addition, 2 states require their programs to track
biological outcomes, such as pregnancies, births, or STDs. In addition, 6
of the 10 states in our review require their programs to track
participants' attitudes about abstinence and sex, such as the number of
participants who make pledges to remain abstinent.
Some states also provide technical assistance to the
abstinence-until-marriage programs they support in their state. This
assistance is designed to help programs evaluate and improve their
effectiveness. Officials from 5 of the 10 states in our review either told
us or provided documentation that they provide technical assistance on
evaluations to abstinence programs in their state. One state official said
that the abstinence-until-marriage programs supported by the state were
found to be ill-prepared to conduct evaluations themselves, and that she
now requires these programs to dedicate a portion of their grants to
contract with a third-party or state evaluator to assist them in
program-level evaluations. Officials from another state told us that they
contract with a private organization of public health professionals in
order to provide evaluation consultation and technical assistance for the
abstinence-until-marriage programs the state supports.
^48See, for example, LeCroy & Milligan Associates, Inc., Abstinence Only
Education Program: Fifth Year Evaluation Report, a report prepared for the
Arizona Department of Health Services, 2003; Patricia Goodson et al.,
Abstinence Education Evaluation: Phase 6, a report prepared for the Texas
Department of State Health Services, 2005; MGT of America, Evaluation of
Georgia Abstinence Education Programs Funded Under Title V, Section 510, a
report prepared for the Georgia Department of Human Resources, 2005;
Thomas E. Smith, It's Great to Wait: An Interim Evaluation, a report
prepared for the Florida Department of Health, 2001.
ASPE, CDC, and NIH Are Funding Studies Designed to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs
In addition to ACF and OPA, other organizational units within HHS have
made efforts to assess the effectiveness of abstinence-until-marriage
education programs. ASPE is currently sponsoring a study of the
Community-Based Program and a study of the State Program.^49 For the
former program, ASPE has contracted with Abt Associates to help design the
study, and an ASPE official told us that once the agency selects an
appropriate design, it will competitively award a contract to conduct the
study.^50 For the latter program, ASPE has contracted with Mathematica
Policy Research, Inc. (Mathematica), which is in the process of examining
the impact of five programs funded through the State Program on
participants' attitudes and behaviors related to abstinence and sex.^51 As
of August 2006, Mathematica has published two reports on findings from its
study--an interim report documenting the experiences of schools and
communities that receive abstinence-until-marriage education funding, and
a report on the first-year impacts of selected state
abstinence-until-marriage education programs.^52,53 Mathematica's final
report, which has not been completed, will examine the impact of the State
Program on behavioral outcomes, including abstinence, sexual activity,
risk of STDs, risk of pregnancy, and drug and alcohol use.^54 An ASPE
official told us that the agency expects a final report to be published in
2007.
^49According to the House Appropriations Committee report accompanying the
fiscal year 2005 appropriation act for Labor, HHS, Education, and related
agencies, the effectiveness of abstinence education programs should be
determined by measures that include the prevention and reduction of
adolescent pregnancies and STD infections, age at first sexual activity
and intercourse, frequency of sexual activity and intercourse, and numbers
who postpone sexual activity or intercourse through adolescence. See H.R.
Rep. No. 108-636, at 139-140 (2004).
^50According to ASPE officials, one factor that has contributed to delays
in the initiation of this study is the difficulty in recruiting schools to
participate.
^51The five abstinence-until-marriage education programs being studied are
My Choice, My Future! in Powhatan, Virginia; ReCapturing the Vision in
Miami, Florida; Teens in Control in Clarksdale, Mississippi; Families
United to Prevent Teen Pregnancy in Milwaukee, Wisconsin; and Heritage
Keepers in Edgefield, South Carolina.
^52See B. Devaney et al., The Evaluation of Abstinence Education Programs
Funded Under Title V Section 510: Interim Report, a report prepared for
ASPE, 2002.
^53See R. Maynard et al., First-Year Impacts of Four Title V, Section 510
Abstinence Education Programs, a report prepared for ASPE, 2005.
Mathematica's report on the first-year impacts of selected state
abstinence-until-marriage education programs focused on intermediate
outcomes, including attitudes about abstinence, teen sex, and marriage;
perceived consequences of teen and nonmarital sex; and expectations to
abstain from sexual intercourse.
Like ASPE, CDC has made its own effort to assess the effectiveness of
abstinence-until-marriage education. CDC is sponsoring a study to evaluate
the effectiveness of two middle school curricula--one that complies with
abstinence education program requirements and one that teaches a
combination of abstinence and contraceptive information and skills.^55 In
CDC's study, five middle schools chosen at random will receive a program
consisting of abstinence-until-marriage education exclusively; five
schools will receive comprehensive sex education, which also includes
information on contraception; and five schools will be assigned to a
control group. The study will examine the relative effectiveness of the
programs on behavioral outcomes such as reported sexual risk behaviors and
changes in attitudes related to abstinence and sex. CDC plans to recruit
approximately 1,500 seventh grade students into its study and will follow
them over a 2-year period. The agency expects to complete the study in
2009.
NIH has funded studies comparing the effectiveness of education programs
that focus only on abstinence with the effectiveness of sex education
programs that teach both abstinence and information about contraception.
As of August 2006, NIH is funding five studies, which in general are
comparing the effects of these two types of programs on the sexual
behavior and related attitudes among groups of either middle school or
high school students. For example, in one NIH study, researchers are using
groups of seventh and eighth grade adolescents to assess the impact of a
variety of programs on, among other issues, adolescents' reported sexual
activities, knowledge, and beliefs. For this study, researchers are
comparing these outcomes among students who received
abstinence-until-marriage education; students who received a combination
of abstinence and contraceptive education; and students who participated
in a general health class, who serve as a comparison group. NIH expects
both this study and its other four studies to be competed in 2006.
^54An impact evaluation assesses the net effect of a program by comparing
program outcomes with an estimate of what would have happened in the
absence of the program. This form of evaluation is employed when external
factors are known to influence the program's outcomes, in order to isolate
the program's contribution to achievement of its objectives.
^55HHS officials told us that the two curricula being tested are intended
to be comparable in length, intensity, and other characteristics.
Other Researchers Have Also Made Efforts to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs
In addition to the efforts of researchers working on behalf of HHS and
states, other researchers--such as those affiliated with universities and
various advocacy groups--have made efforts to study the effectiveness of
abstinence-until-marriage education programs. This work includes studies
of the outcomes of individual programs and reviews of other studies on the
effectiveness of individual abstinence-until-marriage education programs.
In general, research studies on the effectiveness of individual
abstinence-until-marriage education programs have examined the extent to
which they changed participants' demonstrated knowledge, declared
intentions, and reported behavior related to sexual activity and
abstinence. For example, some studies examined the impact of
abstinence-until-marriage education programs on participants' knowledge of
concepts taught in the programs, as well as participants' declared
attitudes about abstinence and teen sex. Some studies examined the impact
of these programs on such outcomes as participants' declared commitment to
abstain from sex until marriage, participants' understanding of the
potential consequences of having intercourse, and participants' reported
ability to resist pressures to engage in sexual activity. Some of the
studies we reviewed examined the impact of abstinence-until-marriage
programs on participants' sexual behavior, as measured, for example, by
the proportion of participants who reported having had sexual intercourse
and the frequency of sexual intercourse reported by participants. In
general, the efforts to study and build a body of research on the
effectiveness of most abstinence education programs have been under way
for only a few years, in part because grants under the two programs that
account for the largest portion of federal spending on abstinence
education--the State Program and the Community-Based Program--were not
awarded until 1998 and 2001, respectively.
Several Factors Limit the Conclusions That Can Be Drawn about the Effectiveness
of Abstinence-until-Marriage Education Programs
Most of the efforts of HHS, states, and other researchers to evaluate the
effectiveness of abstinence-until-marriage education programs included in
our review have not met certain minimum criteria that experts have
concluded are necessary in order for assessments of program effectiveness
to be scientifically valid. For example, most of the efforts included in
our review did not include experimental or quasi-experimental designs, nor
did they measure behavioral or biological outcomes. In addition, the
results of some assessment efforts that meet the criteria of a
scientifically valid assessment have varied, and two key studies that meet
these criteria have not yet been completed.
Experts Have Developed Criteria to Evaluate Efforts to Assess
Abstinence-until-Marriage Education Programs
In an effort to better assess the merits of the studies that have been
conducted on the effectiveness of sexual health programs--including
abstinence-until-marriage education programs--scientific experts have
developed criteria that can be used to gauge the scientific rigor of these
evaluations. For example, in 2001, the National Campaign to Prevent Teen
Pregnancy--an organization focused on reducing teen pregnancy--published a
report by a panel of scientific experts that assessed the evidence
reported on abstinence-until-marriage education programs in peer-reviewed
journals and other literature.^56 The panel developed criteria that an
evaluation of a program's effectiveness must meet in order for the
program's results to be considered scientifically valid. In addition, in
2004, former U.S. Surgeon General David Satcher convened a panel of
experts to discuss, among other things, best practices for evaluating the
effectiveness of sexual health education programs--including
abstinence-until-marriage education programs.^57 This panel published a
report in 2006 that describes similar scientific criteria that assessments
of program effectiveness need to meet in order for their results to be
scientifically valid. Further, experts we interviewed agreed that these
criteria are important for ensuring that the results of a study support
valid conclusions. In general, these panels, as well as the experts we
interviewed, agreed that scientifically valid studies of a program's
effectiveness should include the following characteristics:
o An experimental design that randomly assigns individuals or
schools to either an intervention group or control group, or a
quasi-experimental design that uses nonrandomly assigned but
well-matched comparison groups. According to the panel of
scientific experts convened by the National Campaign to Prevent
Teen Pregnancy, experimental designs or quasi-experimental designs
with well-matched comparison groups have at least three important
strengths that are typically not found in other studies, such as
those that use aggregated data: they evaluate specific programs
with known characteristics, they can clearly distinguish between
participants who did and did not receive an intervention, and they
control for other factors that may affect study outcomes.
Therefore, experimental and quasi-experimental study designs have
a greater ability to assess the causal impact of specific programs
than other types of studies.^58 According to scientific experts,
studies that include experimental or quasi-experimental designs
should also collect follow-up data for a minimum number of months
after subjects receive an intervention.^59 Experts reported that
follow-up periods are important in order to identify the effects
of a program that are not immediately apparent or to determine
whether these effects diminish over time. In addition, experts
have reported that studies should have a sample size of at least
100 individuals for study results to be considered scientifically
valid.^60
o Studies should assess or measure changes in biological outcomes
or reported behaviors instead of attitudes or intentions.
According to scientific experts, biological outcomes--such as
pregnancy rates, birth rates, and STD rates--and reported
behaviors--such as reported initiation and frequency of sexual
activity--are better measures of the effectiveness of
abstinence-until-marriage programs, because adolescent attitudes
and intentions may or may not be indicative of actual behavior.
For example, adolescents may report that they intend to abstain
from sexual intercourse but may not actually do so.
Most Efforts to Assess the Effectiveness of Abstinence-until-
Marriage Education Programs Have Not Used an Experimental or
Quasi-Experimental Design
Many of the efforts by HHS, states, and other researchers that we
identified in our review lack at least one of the characteristics
of a scientifically valid study of program effectiveness. That is,
most of the efforts to assess the effectiveness of these programs
have not used experimental or quasi-experimental designs with
sufficient follow-up periods and sample sizes to make their
conclusions scientifically valid. For example, ACF--and before it,
HRSA--used, according to ACF officials, grantee reporting on
outcomes in order to monitor grantees' performance, target
training and technical assistance, and help grantees improve
service delivery. However, because the outcomes reported by
grantees have not been produced through experimentally or
quasi-experimentally designed studies, such information cannot be
causally attributed to any particular abstinence-until-marriage
education program. While ACF requires its fiscal year 2006
grantees of the Community-Based Program to contract with
third-party evaluators to select and monitor outcomes for their
programs, ACF is not specifically requiring these grantees to use
experimental or quasi-experimental designs. Therefore, it is not
clear whether these evaluations will include such designs.
Similarly, ACF's use of national data on adolescent behavior and
birth rates to assess its State and Community-Based Programs is of
limited value because these data do not distinguish between those
who participated in abstinence-until-marriage education programs
and those who did not. Consequently, these national data sets,
which represent state-reported vital statistics and a nationwide
survey of high school students, cannot be used to causally link
declines in birth rates and adolescent sexual activity to the
effects of specific abstinence-until-marriage education
programs.^61
Similarly, the efforts we identified by states and researchers to
assess the effectiveness of abstinence-until-marriage education
programs often did not include experimental or quasi-experimental
designs. None of the state evaluations we reviewed that have been
completed included randomly assigned control groups. For instance,
one state evaluation that we reviewed only included students who
volunteered to participate in the study. This evaluation report
stated that the absence of a randomly assigned control group in
the evaluation did not allow the evaluators to determine whether
observed changes in participants' reported sexual behavior--as
indicated through surveys administered at the beginning and end of
a program--could be attributed to the abstinence-until-marriage
education program.^62 Similarly, some of the journal articles that
we reviewed described studies to assess the effectiveness of
abstinence-until-marriage programs that did not include
experimental or quasi-experimental designs needed to support
scientifically valid conclusions about the programs'
effectiveness. In these studies, researchers administered
questionnaires to study participants before and after they
completed an abstinence-until-marriage education program and
assessed the extent to which the responses of participants
changed.^63 These studies did not compare the responses of study
participants with a group that did not participate in an
abstinence-until-marriage education program. In addition, some of
the studies used insufficient follow-up periods, thereby limiting
the conclusions that can be drawn about the effectiveness of the
abstinence-until-marriage education programs being studied. For
example, two journal articles that we reviewed described studies
that measured the effectiveness of abstinence-until-marriage
programs in delaying the initiation of sexual activity from 1 to 2
months after completion of the program.^64 Scientific experts
consider this follow-up period too short to assess whether the
programs had a valid effect.
According to scientific experts, HHS, states, and other
researchers face a number of challenges in designing experimental
or quasi-experimental studies of program effectiveness. According
to these experts, experimental or quasi-experimental studies can
be expensive and time-consuming to carry out, and many grantees of
abstinence-until-marriage education programs have insufficient
time and funding to support these types of studies. Moreover, it
can be difficult for researchers assessing
abstinence-until-marriage education programs to convince school
districts to participate in randomized intervention and control
groups, in part because of sensitivities to surveying attitudes,
intentions, and behaviors related to abstinence and sex. For
example, in a third-party evaluation of its program, one grantee
of the State Program originally planned to administer follow-up
surveys 1 year after participants finished their abstinence
education program, but the evaluators decided not to conduct this
follow-up because of confidentiality concerns and the difficulty
of locating students. In addition, the contractors hired to design
ASPE's study of the effectiveness of the Community-Based Program
have reported difficulties finding school districts that are
willing to participate in randomly assigned intervention and
control groups receiving either abstinence-until-marriage
education or comprehensive sex education. An ASPE official told us
that although a "randomized approach" is the best design for
assessing the effectiveness of a program, the approach is also the
most difficult to conduct.
Most Efforts to Assess the Effectiveness of Abstinence-until-
Marriage Education Programs Have Not Measured Behavioral or
Biological Outcomes
Another factor that limits the conclusions that can be drawn about
the effectiveness of abstinence-until-marriage education programs
is the fact that most efforts in our review to study the
effectiveness of these programs did not measure changes in
behavioral or biological outcomes among participants. Instead,
most of the efforts we identify in our review used reported
intentions and attitudes in order to assess the effectiveness of
abstinence-until-marriage programs. For example, neither ACF's
community-based grantees nor OPA's AFL grantees are required to
report on behavioral or biological outcomes, such as rates of
intercourse or pregnancy. Similarly, the journal articles we
reviewed were more likely to use reported attitudes and
intentions--such as study participants' reported attitudes about
premarital sexual activity or their reported intentions to remain
abstinent until marriage--rather than their reported behaviors or
biological outcomes to assess the effectiveness of
abstinence-until-marriage programs. For example, in one journal
article we reviewed, participants were asked to rate the
likelihood that they would have sexual intercourse as unmarried
teenagers; another journal article described a study in which
participants rated the likelihood that they would have sexual
intercourse in the next year, before finishing high school, and
before marriage.^65
Experts, as well as state and HHS officials, have reported that it
can be difficult to obtain scientifically valid information on
biological outcomes and sexual behaviors. Specifically, experts
have reported that when measuring an abstinence-until-marriage
education program's affect on biological outcomes--such as
reducing pregnancy or birth rates--it is necessary to have large
sample sizes in order to determine whether a small change in
biological outcomes is the result of the abstinence-until-marriage
education program. In addition, state and federal officials told
us that they have experienced difficulties obtaining information
on sexual behaviors because of the sensitive nature of the
information they were trying to collect. For example, one state
official told us that her state's effort to evaluate
abstinence-until-marriage education programs was only able to
measure changes in participants' reported attitudes, instead of
behaviors, because the evaluators needed to obtain consent from
the parents of the program participants in order to ask them about
their sexual behavior. The state official explained that the
requirement to obtain consent from parents raised issues of
self-selection, and therefore state officials decided to
ultimately halt the study and only report on the attitudes that
they had measured. In another example, ACF's fiscal year 2006
budget justification reports that ACF has had some difficulty in
obtaining reliable data from state grantees, in part because
questions about teenage sexual behavior are sensitive. OPA
officials also acknowledged that many communities will not allow
grantees to ask program participants questions about their sexual
behavior because the communities believe such questions are too
intrusive. One OPA official said that such restrictions affect the
agency's ability to measure behavioral outcomes, explaining that
OPA cannot measure what it cannot ask about.
Results of Some Scientifically Valid Assessment Efforts Have
Varied, and Other Key Studies Have Not Been Completed
Among the assessment efforts we identified are some studies that
meet the criteria of a scientifically valid effectiveness study.
However, results of these studies have varied, and this limits the
conclusions that can be drawn about the effectiveness of
abstinence-until-marriage education programs. Some researchers
have reported that abstinence-until-marriage education programs
have resulted in adolescents reporting having less frequent sexual
intercourse or fewer sexual partners.^66 For example, in one study
of middle school students, participants in an
abstinence-until-marriage education program who had sexual
intercourse during the follow-up period were 50 percent less
likely to report having two or more sexual partners when compared
with their nonparticipant peers.^67 In contrast, other studies
have reported that abstinence-until-marriage education programs
did not affect the reported frequency of sexual intercourse or
number of sexual partners.^68 For example, one study of middle
school students found that participants of an
abstinence-until-marriage program were not less likely than
nonparticipants at the 1 year follow-up to report less frequent
sexual intercourse or fewer sexual partners.^69 In addition to
these varied findings, one study found that an
abstinence-until-marriage program was effective in delaying the
initiation of sexual intercourse in the short term but not long
term.^70 Experts with whom we spoke emphasized that there are
still too few scientifically valid studies completed to date that
can be used to determine conclusively which, if any,
abstinence-until-marriage programs are effective.
Additionally, among the assessment efforts we identified are some
studies that experts anticipate will meet the criteria of a
scientifically valid effectiveness study but are not yet
completed. One of these key studies is the final Mathematica
report, contracted by ASPE, on the State Program.^71 The final
report was originally slated for publication in 2005, but an ASPE
official stated that the final report has been delayed until 2007
so that researchers can extend the follow-up period to improve
their response rate and the reliability of the information they
collect. Another key study is CDC's research on middle school
programs, which is not expected to be completed until 2009.
Experts and federal officials we interviewed stated that they
expect the results of these two federally funded studies to add
substantively to the body of research on the effectiveness of
abstinence-until-marriage education programs. One expert with whom
we spoke said that she expects the final Mathematica report on
participants' behaviors to provide the groundwork for the field.
Another expert we interviewed stated that the CDC study was very
well-designed and she expects the results to contribute to the
development of effective abstinence-until-marriage education
curricula.
Conclusions
There have been various efforts--by HHS, states, and others--to
assess the scientific accuracy of educational materials used in
abstinence-until-marriage education programs and the effectiveness
of these programs. However, efforts to evaluate both the accuracy
and effectiveness of abstinence-until-marriage education programs
have been, in various ways, limited. ACF, which administers the
two programs that account for the largest portion of federal
spending on abstinence-until-marriage education, does not review
or require its grantees to review program materials for scientific
accuracy. In addition, not all grantees of the State Program have
chosen to review their materials. Because of these limitations,
ACF cannot be assured that the materials used in its State and
Community-Based Programs are accurate. Moreover, OPA, which
reviews all grantees' proposed abstinence-until-marriage
educational materials, and states that review educational
materials have found inaccuracies in some educational materials
used by abstinence-until-marriage programs.
Similarly, most of the efforts described in our review to assess
the effectiveness of abstinence-until-marriage programs have not
met minimum scientific criteria needed to draw valid conclusions
about their effectiveness. Specifically, most efforts by agencies,
states, and other researchers have not included experimental or
quasi-experimental designs that can establish whether changes in
behaviors or biological outcomes can be causally linked to
specific abstinence-until-marriage education programs. While these
types of studies are time-consuming and expensive, experts said
that they are the only definitive way to draw valid conclusions
about the effectiveness of these programs. In addition, among the
assessment efforts we identified are some studies funded by HHS
that experts anticipate will meet the criteria of a scientifically
valid effectiveness study but are not yet completed. When
completed, these HHS-funded studies may add substantively to the
body of research on the effectiveness of abstinence-until-marriage
education programs.
Recommendation for Executive Action
To address concerns about the scientific accuracy of materials
used in abstinence-until-marriage education programs, we recommend
that the Secretary of HHS develop procedures to help assure the
accuracy of such materials used in the State and Community-Based
Programs. To help provide such assurances, the Secretary could
consider alternatives such as (1) extending the approach currently
used by OPA to review the scientific accuracy of the factual
statements included in abstinence-until-marriage education to
materials used by grantees of ACF's Community-Based Program and
requiring grantees of ACF's State Program to conduct such reviews
or (2) requiring grantees of both programs to sign written
assurances in their grant applications that the materials they
propose using are accurate.
Agency Comments and Our Evaluation
HHS provided written comments on a draft of this report. (See app.
III.)
In its written comments, HHS stated that it will consider
requiring grantees of both ACF programs to sign written assurances
in grant applications that the materials they use are accurate.
Regarding accuracy, HHS's written comments also noted that all
applicants for federal assistance attest on the application
form--Standard Form 424--that all data in their applications are
"true and correct," and that in the view of HHS, this applies to
information presented in curricula funded by federal grants.
However, as we stated in the draft report, grantees of the State
Program are not required to submit curricula as a part of their
applications; therefore, the attestation in Standard Form 424
would not apply to curricula used by those grantees. In addition,
as stated in the draft report, some states have reviewed materials
used in abstinence-until-marriage education programs, but these
reviews occurred after they received funding from ACF. Further,
while grantees of the Community-Based Program were required to
submit copies of their curricula and a Standard Form 424 in fiscal
year 2006 as part of their applications, none of the materials
specifically require an assurance of scientific accuracy. Further,
OPA and states have found inaccuracies in educational materials
used by abstinence-until-marriage programs.
HHS's written comments also stated that ACF requires that
curricula conform to HHS's standards grounded in scientific
literature. HHS's comments refer to the curriculum standards for
this program that detail what types of information must be
included in abstinence-until-marriage curricula, and the comments
stated that the curricula must provide supporting references for
this information. Further, HHS's comments stated that ACF staff
review the curricula to ensure compliance with these standards.
The draft report stated this. However, a requirement that
curricula include certain types of information does not
necessarily ensure the accuracy of the scientific facts included
in the abstinence-until-marriage materials. For example, while
education materials may include information on failure rates
associated with contraceptives or STD infections, this information
may be outdated or otherwise inaccurate or incomplete. HHS's
written comments also stated that if it finds inaccurate
statements during the review process or at any time during the
grant period, ACF works with grantees to take corrective action.
To ensure completeness, we have added this statement to the
report. Further, HHS stated that 2 inaccuracies cited in the draft
report had been corrected before our work began. We believe HHS is
referring to inaccuracies identified by OPA during its review of
materials for scientific accuracy and this reinforces the need for
review of materials used by ACF's grantees.
As HHS noted in its written comments, we did not define the term
scientific accuracy. HHS stated that it disagreed with certain
findings of the report because it was difficult to precisely
determine the criteria we employed in making the recommendation as
to scientific accuracy. As we stated in the scope and methodology
section of the draft report, the objective of our work was to
focus on efforts by HHS and states to review the accuracy of
scientific facts included in abstinence-until-marriage education
materials. Performing an independent assessment of the criteria
used by these entities to determine the scientific accuracy of
education materials or the quality of the reviews was beyond the
scope of the work.
Regarding effectiveness, HHS's written comments also described a
number of actions it is taking to determine program effectiveness
and improve the quality of programs and research. Specifically,
HHS's comments described (1) studies undertaken or funded by ASPE,
CDC, and NIH; (2) technical assistance provided by OPA and ACF;
(3) grantee evaluation requirements; and (4) ACF and OPA
requirements for the amount of grant funds to be spent on
evaluations. All of this information was included in our draft
report. HHS's comments also described a new effort funded by ACF
and ASPE that is designed to build capacity for quality research
in the field of abstinence education. We added information on this
effort to the report. HHS's written comments also describe
evaluations that resulted from an Abstinence Education Evaluation
Conference sponsored by ACF and OPA. While this conference was
described in the draft report, we added more detail regarding the
content of the conference. HHS's written comments also describe
OPA's efforts to assess the effectiveness of the AFL Program. We
had included this information in the draft report.
HHS's written comments stated that it may be too soon to draw
conclusions about the effectiveness of ACF's and OPA's programs,
in part, because key studies have not been completed. We agree and
discussed this in the draft report. As we noted in the draft
report, key studies funded by HHS that experts anticipate will
meet the criteria of a scientifically valid effectiveness study
are not yet completed, but when completed these HHS funded studies
may add substantively to the body of research on the effectiveness
of abstinence-until-marriage education programs.
In addition, the comments stated that having an inadequate amount
of scientifically valid and conclusive evaluation studies is not
unique to abstinence-until-marriage education programs, and a
recent ASPE review of comprehensive sex education programs found
mixed results on their effectiveness. However, the scope of our
report was focused on abstinence-until-marriage education
programs, and we did not review comprehensive sex education
programs or make any comparisons between the two types of
programs.
HHS also provided technical comments, which we incorporated into
the report as appropriate.
As agreed with your office, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 30 days after its date. We will then send copies of this
report to the Secretary of HHS and to other interested parties. In
addition, this report is available at no charge on GAO's Web site
at http://www.gao.gov. We will also make copies available to
others upon request.
If you or your staff have any questions about this report, please
call me at (202) 512-3407 or [email protected]. Contact points for
our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix IV.
Marcia Crosse
Director, Health Care
List of Requesters
The Honorable Henry A. Waxman
Ranking Minority Member Committee on Government Reform
House of Representatives
The Honorable Pete Stark
Ranking Minority Member
Subcommittee on
Health Committee on Ways and Means
House of Representatives
The Honorable Sherrod Brown
Ranking Minority Member Subcommittee on Health Committee on Energy and Commerce
House of Representatives
The Honorable Barbara Boxer
The Honorable Maria Cantwell
The
Honorable Richard J. Durbin
The Honorable Russell D. Feingold
The Honorable Dianne Feinstein
The Honorable Tom Harkin
The Honorable James M. Jeffords
The Honorable Edward M. Kennedy
The Honorable Frank R. Lautenberg
The Honorable Patrick Leahy
The Honorable Patty Murray
United States Senate
The Honorable Howard L. Berman
The Honorable Lois Capps
The Honorable Jay Inslee
The Honorable Barbara Lee
The Honorable Nita M. Lowey
The Honorable Betty McCollum
House of Representatives
Appendix I: HRSA�s Technical Assistance Contract for
Abstinence Education
The Health Resources and Services Administration (HRSA) awarded a
contract to the National Abstinence Clearinghouse (NAC) in 2002 to
provide assistance with its Community-Based Abstinence Education
Program (Community-Based Program) and Abstinence Education Program
(State Program).^1 NAC is a nonprofit educational organization
whose mission is to promote the appreciation for and practice of
sexual abstinence until marriage through the distribution of age
appropriate, factual, and medically referenced materials. The
purpose of the contract was (1) to develop national criteria for
the review of abstinence-until-marriage educational materials and
to create a directory of approved materials; (2) to provide
medical accuracy training to grantees; and (3) to provide
technical support to grantees, such as assistance with program
evaluation.^2
We are reporting on the steps that HRSA took to award the contract
to NAC in response to concerns that have been raised by a
congressional requester. In general, these concerns centered on
the extent to which the selection process was competitive and
whether HRSA identified the potential for an organizational
conflict of interest.
HRSA awarded the contract to address three concerns it had with
the Community-Based Program during 2001, the first year of its
implementation. First, HRSA officials needed guidance to determine
whether abstinence-until-marriage education materials conformed to
the definitional requirements of the Social Security Act.^3
Second, many grantees lacked the medical background and training
to ensure that they would provide medically accurate,
science-based information in their programs. Third, grantees also
lacked experience with the technical management of federal grants,
including how to conduct evaluations of their programs.
HRSA used full and open competition procedures to award the
contract to NAC. In doing so, HRSA (1) publicly solicited
proposals from potential contractors; (2) conducted technical
evaluations of both the original proposals and the revised
proposals for those considered to be in the competitive range; and
(3) determined that NAC's proposal represented the best overall
value to the government. This process, which took place from May
2002 through September 2002, resulted in HRSA awarding NAC the
contract with a potential value of nearly $2.7 million.
HRSA Used Full and Open Competition Procedures to Award the
Contract to NAC
HRSA issued a notice on May 20, 2002, on the FedBizOpps Web site,
the government point of entry for notifying potential contractors
of federal contract opportunities, indicating its intent to
publicly request proposals from prospective contractors in June
2002.^4 On June 20, 2002, HRSA posted the solicitation on the
FedBizOpps Web site indicating that the abstinence contract would
be awarded using full and open competition procedures, that is,
all responsible prospective contractors would be provided the
opportunity to compete.^5 The solicitation, which was a Request
for Proposals (RFP), described the contract objectives, which
included (1) the development of national criteria for the review
of abstinence-until-marriage educational materials and the
development of a directory of approved materials; (2) the
provision of medical accuracy training to grantees; and (3) the
provision of technical support to grantees, such as assistance
with program evaluation. The RFP stated that HRSA intended to
award a cost-reimbursement contract with fixed fee for a 1-year
base period and 2 option years.^6 This was a best value
procurement; that is, HRSA reserved the right in the RFP to select
for award the proposal that HRSA determined offered the best value
to the government, even if it did not offer the lowest cost.
Further, the RFP stated that the technical evaluation of the
prospective contractors' proposals would receive paramount
consideration in the selection of the contractor. According to the
RFP, this evaluation would include an assessment of the
prospective contractor's technical approach, the organizational
experience and expertise of the prospective contractor, the plans
for personnel and management of the work, and the prospective
contractor's statement and understanding of the project purpose.
Other factors, such as the estimated cost, past performance under
other contracts for similar services, and the subcontracting plan
would also be considered in the selection process. Five
prospective contractors submitted proposals to HRSA by July 31,
2002, when proposals were due.
HRSA established a review committee to conduct the technical
evaluation of the five proposals. This committee included three
voting members and a nonvoting chairperson. The Director of HRSA's
Community-Based and State Programs and two analysts from other
programs within the Department of Health and Human Services (HHS)
served as the voting members, and the chairperson of the review
committee was a project officer of HRSA's Community-Based Program.
The committee members conducted the technical evaluation of the
proposals, according to the criteria in the RFP, as described
above. Three proposals with the highest technical scores were
determined to be in the competitive range,^7 with NAC's proposal
receiving the highest technical score. HRSA requested in writing
that the competitive range offerors address certain technical and
cost issues and submit revised proposals to HRSA by September 17,
2002.^8 For example, HRSA requested that one of the prospective
contractors other than NAC clearly describe its proposed
management of day-to-day tasks of the contract and provide
justification for several labor and travel expenditures. HRSA did
not have oral discussions with the competitive range offerors.
HRSA's review committee evaluated the revised proposals and again
gave NAC's revised proposal the highest technical score.
Although NAC's estimated cost was not the lowest among the
proposals in the competitive range, HRSA determined that NAC had
proposed a realistic cost estimate for the contract. Accordingly,
and in light of the NAC proposal's high technical rating and the
RFP's evaluation criteria giving paramount consideration to the
technical evaluation, HRSA determined that NAC's proposal
represented the best value to the government. HRSA awarded a
contract to NAC on September 27, 2002. The contract had a 1-year
base period of performance with an estimated value of $854,681,
and included 2 option years for a total potential value of
$2,673,784. According to a HRSA official, this cost-reimbursement
contract did not include a fee. All of the prospective contractors
were made aware that a debriefing to explain the selection
decision and contract award would be provided at their request.
One prospective contractor requested and received a debriefing
from HRSA. No protests were filed with the agency challenging the
award of the contract to NAC. There were no bid protests filed
with GAO.^9
HRSA Identified No Actual or Potential Organizational Conflicts
of Interest
HRSA officials told us that they did not identify any actual or
potential organizational conflicts of interest during the
acquisition process. As defined in the Federal Acquisition
Regulation (FAR), an organizational conflict of interest arises
where
o because of other activities or relationships, a person is unable
or potentially unable to provide impartial assistance or advice to
the government; or
o the person's objectivity in performing the contract work is or
might be otherwise impaired; or
o a person has an unfair competitive advantage.^10
An organizational conflict of interest may result when factors
create an actual or potential conflict of interest during
performance of a contract, or when the nature of the work to be
performed under one contract creates an actual or potential
conflict of interest involving a future acquisition.^11 Under the
FAR, contracting officers are required to analyze planned
acquisitions to identify and evaluate potential organizational
conflicts of interest as early in the acquisition process as
possible, and to take steps to avoid, neutralize, or mitigate
significant potential conflicts of interest before a contract is
awarded.^12
According to HRSA's contracting officer, HRSA did not identify any
actual or potential organizational conflicts of interest. In
reaching this conclusion, the contracting officer told us that he
reviewed the statement of work, including the background and
objectives of the proposed contract, the stated purpose of the
contact, the criteria established to evaluate the proposals, the
past performance of the competitors, and NAC's proposal. HRSA's
contracting officer also told us that he did not formally document
his assessment of organizational conflict of interest.^13
Appendix II: Methodology for Identifying and Reviewing Research
Studies
To identify research studies that examine the effectiveness of
abstinence-until-marriage education programs among adolescents and
young adults, we searched two reference database systems, PubMed
and ProQuest. We used the following keywords to search for
research studies that were published from January 1, 1998, through
May 22, 2006: "virginity," "abstinence education," "abstinence and
curriculum," "abstinence only," "teen pregnancy and prevention,"
and "abstinence until marriage."^1 We reviewed the research
article titles that were generated from the PubMed and ProQuest
searches and identified articles that appeared to focus on the
evaluation of the effectiveness of abstinence-until-marriage
education programs. In cases where we could not determine, based
on the title, whether a study appeared to focus on an
abstinence-until-marriage education program evaluation, we
reviewed a summary of the article to obtain more information about
the research study. We also examined previous summaries of the
literature to identify additional research studies.
We then selected research studies for inclusion in our literature
review if they met three criteria. First, the study evaluated a
group-based, abstinence-until-marriage education program. We did
not select studies that evaluated one-on-one interactions, such as
education programs focused exclusively on parent-child
interactions, or that evaluated media campaigns. We reviewed the
description of each education program and curriculum, as described
in the study, to determine whether an abstinence-until-marriage
education program was being evaluated. Education programs that
were described as including detailed contraceptive information in
their curricula, for example, were not classified as
abstinence-until-marriage programs. Second, the study targeted
adolescents and young adults in the United States, for example, by
indicating that participants in the evaluation were high school or
middle school students. Third, the study was a quantitative rather
than a qualitative evaluation of an abstinence-until-marriage
education program. We selected 13 research studies for inclusion
in our literature review.
We reviewed the selected research studies to obtain detailed
information about their methodologies and outcome variables. For
example, we determined whether each study used an experimental or
quasi-experimental design and whether the outcome measures
included attitudes, behavioral intentions, behaviors such as
initiation of sexual intercourse, or a combination of these.
Appendix III: Comments from the Department of Health and Human
Services
Appendix IV: GAO Contact and Staff Acknowledgments
GAO Contact
Marcia Crosse, (202) 512-7119 or [email protected]
Acknowledgments
In addition to the contact named above, Kristi Peterson, Assistant
Director; Kelly DeMots; Pam Dooley; Krister Friday; Julian
Klazkin; and Amy Shefrin made key contributions to this report.
GAO�s Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
^56See Kirby. This panel included experts from NIH, the Medical Institute
for Sexual Health, the Alan Guttmacher Institute, the Institute for
Research and Evaluation, and various universities.
^57See Satcher. This panel included experts from a variety of
organizations, including the Medical Institute for Sexual Health, the Alan
Guttmacher Institute, and the American Academy of Pediatrics.
Most Efforts to Assess the Effectiveness of Abstinence-until-Marriage
Education Programs Have Not Used an Experimental or Quasi-Experimental Design
^58For example, experts have reported that the use of randomly assigned
intervention and control groups is particularly important when assessing
the effectiveness of abstinence-until-marriage programs because
adolescents who voluntarily participate in such programs may be
self-selecting--that is, they may be more willing to accept the principles
of--and be influenced by--such programs when compared with other
adolescents. Mathematica's interim report on the evaluation of the State
Program noted that selection bias can "seriously undermine the
credibility" of study results.
^59For example, one expert reported that studies assessing program
effectiveness should obtain information on participants for at least 3
months after the conclusion of a program when they are measuring behaviors
that can change quickly, such as frequency of sex. For behaviors or
outcomes that change less quickly, such as initiation of sex or pregnancy
rates, information on participants should be collected for at least 6
months after the conclusion of a program.
^60The panel of experts convened by the National Campaign to Prevent Teen
Pregnancy agreed that large sample sizes are necessary to determine the
magnitude of any discernable program effect and to ensure that results of
any study of effectiveness are statistically valid.
^61In addition, according to ACF and CDC officials, it is difficult to
draw conclusions from national data sets about the effectiveness of
abstinence-until-marriage education programs because the national survey
questions used to produce these data often do not identify the specific
type of program or intervention survey respondents may have participated
in or received.
^62See Goodson et al.
^63See, for example, S. M. Fitzgerald et al., "Effectiveness of the
Responsible Social Values Program for 6th Grade Students in One Rural
School District," Psychological Reports, vol. 91 (2002), and J. E. Barnett
and C. S. Hurst, "Abstinence Education for Rural Youth: An Evaluation of
the Life's Walk Program, " The Journal of School Health, vol. 73, no. 7
(2003).
^64See, for example, D. A. Zanis, "Use of a Sexual Abstinence Only
Curriculum with Sexually Active Youths," Children & Schools, vol. 27, no.
1 (2005), and G. Denny et al., "An Evaluation of An Abstinence Education
Curriculum Series: Sex Can Wait," American Journal of Health Behavior,
vol. 26, no. 5 (2002).
Most Efforts to Assess the Effectiveness of Abstinence-until-Marriage
Education Programs Have Not Measured Behavioral or Biological Outcomes
^65See, for example, L. Sather and K. Zinn, "Effects of Abstinence-Only
Education on Adolescent Attitudes and Values Concerning Premarital Sexual
Intercourse, " Family and Community Health, vol. 25, no. 2 (2002), and G.
Denny, M. Young, and C. E. Spear, "An Evaluation of the Sex Can Wait
Abstinence Education Curriculum Series," American Journal of Health
Behavior, vol. 23, no. 2 (1999).
Results of Some Scientifically Valid Assessment Efforts Have Varied, and Other
Key Studies Have Not Been Completed
^66See Borawski et al. See also T. L. St. Pierre et al., "A 27-Month
Evaluation of a Sexual Activity Prevention Program in Boys & Girls Clubs
Across the Nation," Family Relations, vol. 44, no. 1 (1995).
^67See Borawski et al.
^68See N.G. Harrington et al., "Evaluation of the All Stars Character
Education and Problem Behavior Prevention Program: Effects on Mediator and
Outcome Variables for Middle School Students," Health Education &
Behavior, vol. 28, no. 5 (2001). See also Jemmott, Jemmott, and Fong.
^69See Harrington et al.
^70See Jemmott, Jemmott, and Fong and J. B. Jemmott III, L. S. Jemmott,
and G. T. Fong, reply to letter to editor, Journal of the American Medical
Association, vol. 281, no. 16 (1999), 1487. This study found that an
abstinence-until-marriage program delayed the initiation of sexual
intercourse at the 3-month follow-up period but not at 6 and 12 months.
^71According to several scientific experts, Mathematica's study is an
important one, in part because of its sound design: the study randomly
assigns and compares control groups with groups receiving
abstinence-until-marriage education and uses surveys to follow up with
program participants for several months after their completion of a
program.
Conclusions
Recommendation for Executive Action
Agency Comments and Our Evaluation
Appendix I: HRSA's Technical Assistance Contract for Abstinence Education
Appendix I: HRSA's Technical Assistance Contract for Abstinence Education
^1The administration of this contract was transferred from HRSA to the
Administration of Children and Families (ACF) in May 2005.
^2The contract resulted in the development of criteria for reviewing
abstinence-until-marriage educational materials, and ACF included these
criteria in the fiscal year 2006 program announcement for the
Community-Based Program. According to the announcement, ACF will evaluate
grant applicants' proposed educational materials to ensure compliance with
the criteria. Medical accuracy training and technical support were
provided to grantees as a result of the contract. According to an ACF
official, a directory of approved abstinence-until-marriage educational
materials was not completed.
HRSA Used Full and Open Competition Procedures to Award the Contract to NAC
^3Section 510(b)(2) of the Social Security Act defines abstinence
education as an educational or motivational program that: A. has as its
exclusive purpose, teaching the social, psychological, and health gains to
be realized by abstaining from sexual activity; B. teaches abstinence from
sexual activity outside marriage as the expected standard for all school
age children; C. teaches that abstinence from sexual activity is the only
certain way to avoid out-of-wedlock pregnancy, sexually transmitted
diseases, and other associated health problems; D. teaches that a mutually
faithful monogamous relationship in context of marriage is the expected
standard of human sexual activity; E. teaches that sexual activity outside
of the context of marriage is likely to have harmful psychological and
physical effects; F. teaches that bearing children out-of-wedlock is
likely to have harmful consequences for the child, the child's parents,
and society; G. teaches young people how to reject sexual advances and how
alcohol and drug use increases vulnerability to sexual advances; and H.
teaches the importance of attaining self-sufficiency before engaging in
sexual activity.
^4The current FedBizOpps Web site address is [48]http://www.fbo.gov /.
Prior to October 1, 2005, the Web site address was
[49]http://www.eps.gov/. The solicitation number was 240-MCHB-012(02)-abg.
^5The Federal Acquisition Regulation (FAR) requires the contracting
officer to ensure that prospective contractors are responsible. FAR S
9.103. A responsible source refers to a prospective contractor that has,
among other things, adequate financial resources, the necessary experience
and technical skills to perform the work of the contract, a satisfactory
performance record, and the ability to meet the delivery schedule. FAR S
9.104-1.
^6Cost-reimbursement contracts are used only when uncertainties involved
in contract performance do not permit costs to be estimated with
sufficient accuracy to use any type of fixed-price contract. FAR S
16.301-2.
HRSA Identified No Actual or Potential Organizational Conflicts of Interest
^7Based on the ratings of each proposal against all evaluation criteria,
the contracting officer establishes a competitive range consisting of all
of the most highly rated proposals. FAR S 15.306(c)(1).
^8HRSA officials, including an auditor, reviewed the cost proposals in the
competitive range.
^9GAO's Office of General Counsel resolves disputes concerning awards of
federal contracts, which are known as bid protests.
^10FAR S 2.101.
^11FAR S 9.502(c).
^12The FAR requires contracting officers to exercise common sense, good
judgment, and sound discretion in determining whether a significant
potential conflict of interest exists. FAR S 9.505.
^13The FAR requires HRSA's contracting officers to formally document their
assessment only when a substantive issue concerning a potential
organizational conflict of interest exists. FAR S 9.504(d). HHS
acquisition regulations do not explicitly address the assessment of
organizational conflict of interest. Therefore, FAR subpart 9.5 is the
controlling regulation when HHS encounters an issue related to an
organizational conflict of interest.
Appendix II: Methodology for Identifying and Reviewing Research Studies
Appendix II: Methodology for Identifying and Reviewing Research Studies
^1ACF first awarded grants under the State Program in fiscal year 1998.
Appendix III: Comments from the Department of Health and Human Services
Appendix III: Comments from the Department of Health and Human Services
A Appendix IV: GAO Contact and Staff Acknowledgments
GAO Contact
Acknowledgments
(290491)
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone
To Report Fraud, Waste, and Abuse in Federal Programs
Congressional Relations
Public Affairs
www.gao.gov/cgi-bin/getrpt? [50]GAO-07-87 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Marcia Crosse at (202) 512-7119 or
[email protected].
Highlights of [51]GAO-07-87 , a report to congressional requesters
October 2006
ABSTINENCE EDUCATION
Efforts to Assess the Accuracy and Effectiveness of Federally Funded
Programs
Efforts by HHS and states to assess the scientific accuracy of materials
used in abstinence-until-marriage education programs have been limited.
This is because HHS's ACF--which awards grants to two programs that
account for the largest portion of federal spending on
abstinence-until-marriage education--does not review its grantees'
education materials for scientific accuracy and does not require grantees
of either program to review their own materials for scientific accuracy.
In contrast, OPA does review the scientific accuracy of grantees' proposed
educational materials. In addition, not all states that receive funding
from ACF have chosen to review their program materials for scientific
accuracy. In particular, 5 of the 10 states that GAO contacted conduct
such reviews. Officials from these states reported using a variety of
approaches in their reviews. While the extent to which federally funded
abstinence-until-marriage education materials are inaccurate is not known,
in the course of their reviews OPA and some states reported that they have
found inaccuracies in abstinence-until-marriage education materials. For
example, one state official described an instance in which
abstinence-until-marriage materials incorrectly suggested that HIV can
pass through condoms because the latex used in condoms is porous.
HHS, states, and researchers have made a variety of efforts to assess the
effectiveness of abstinence-until-marriage education programs; however, a
number of factors limit the conclusions that can be drawn about the
effectiveness of abstinence-until-marriage education programs. ACF and OPA
have required their grantees to report on various outcomes that the
agencies use to measure the effectiveness of grantees'
abstinence-until-marriage education programs. In addition, 6 of the 10
states in GAO's review have worked with third-party evaluators to assess
the effectiveness of abstinence-until-marriage education programs in their
states. Several factors, however, limit the conclusions that can be drawn
about the effectiveness of abstinence-until-marriage education programs.
Most of the efforts to evaluate the effectiveness of
abstinence-until-marriage education programs included in GAO's review have
not met certain minimum scientific criteria--such as random assignment of
participants and sufficient follow-up periods and sample sizes--that
experts have concluded are necessary in order for assessments of program
effectiveness to be scientifically valid, in part because such designs can
be expensive and time-consuming to carry out. In addition, the results of
efforts that meet the criteria of a scientifically valid assessment have
varied and two key studies funded by HHS that meet these criteria have not
yet been completed. When completed, these HHS-funded studies may add
substantively to the body of research on the effectiveness of
abstinence-until-marriage education programs.
Reducing the incidence of sexually transmitted diseases and unintended
pregnancies is one objective of the Department of Health and Human
Services (HHS). HHS provides funding to states and organizations that
provide abstinence-until-marriage education as one approach to address
this objective.
GAO was asked to describe the oversight of federally funded
abstinence-until-marriage education programs. GAO is reporting on (1)
efforts by HHS and states to assess the scientific accuracy of materials
used in these programs and (2) efforts by HHS, states, and researchers to
assess the effectiveness of these programs. GAO reviewed documents and
interviewed HHS officials in the Administration for Children and Families
(ACF) and the Office of Population Affairs (OPA) that award grants for
these programs.
[52]What GAO Recommends
To address concerns about the scientific accuracy of materials used in
abstinence-until-marriage education programs, GAO recommends that the
Secretary of HHS develop procedures to help assure the accuracy of such
materials used in programs administered by ACF. HHS agreed to consider
this recommendation. HHS also provided information on steps it takes to
assure accuracy, which we have incorporated into the report, as
appropriate.
References
Visible links
41. http://www.gao.gov/cgi-bin/getrpt?GAO-05-739SP
48. http://www.fbo.gov /
49. http://www.eps.gov/
50. http://www.gao.gov/cgi-bin/getrpt?GAO-07-87
51. http://www.gao.gov/cgi-bin/getrpt?GAO-07-87
*** End of document. ***