Abstinence Education: Efforts to Assess the Accuracy and	 
Effectiveness of Federally Funded Programs (03-OCT-06,		 
GAO-07-87).							 
                                                                 
Reducing the incidence of sexually transmitted diseases and	 
unintended pregnancies is one objective of the Department of	 
Health and Human Services (HHS). HHS provides funding to states  
and organizations that provide abstinence-until-marriage	 
education as one approach to address this objective. GAO was	 
asked to describe the oversight of federally funded		 
abstinence-until-marriage education programs. GAO is reporting on
(1) efforts by HHS and states to assess the scientific accuracy  
of materials used in these programs and (2) efforts by HHS,	 
states, and researchers to assess the effectiveness of these	 
programs. GAO reviewed documents and interviewed HHS officials in
the Administration for Children and Families (ACF) and the Office
of Population Affairs (OPA) that award grants for these programs.
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-87						        
    ACCNO:   A61784						        
  TITLE:     Abstinence Education: Efforts to Assess the Accuracy and 
Effectiveness of Federally Funded Programs			 
     DATE:   10/03/2006 
  SUBJECT:   Adolescent health					 
	     Education program evaluation			 
	     Educational grants 				 
	     Educational research				 
	     Evaluation criteria				 
	     Federal funds					 
	     Federal grants					 
	     Locally administered programs			 
	     Policy evaluation					 
	     Sexually transmitted diseases			 
	     Teenage pregnancy					 
	     Assessments					 
	     Community and supportive services			 
	     programs						 
                                                                 
	     Education programs 				 
	     Government agency oversight			 
	     HHS Abstinence Education Program			 
	     HHS Adolescent Family Life Program 		 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-87

   

     * [1]Results in Brief
     * [2]Background

          * [3]The State Program
          * [4]The Community-Based Program
          * [5]The AFL Program
          * [6]Funding for Abstinence-until-Marriage Education Has Increase
          * [7]Federal Agency Responsibilities Related to Abstinence-until-

               * [8]ACF
               * [9]OPA
               * [10]CDC
               * [11]ASPE
               * [12]NIH

     * [13]Federal and State Efforts to Assess the Scientific Accuracy

          * [14]ACF Does Not Review Program Materials for Scientific Accurac
          * [15]OPA Reviews Materials Used by AFL Program Grantees for Scien

     * [16]A Variety of Efforts Have Been Made to Assess the Effectiven

          * [17]HHS, States, and Researchers Have Made a Variety of Efforts

               * [18]ACF and OPA Have Required Grantee Reporting of Data on
                 Outco
               * [19]ACF Uses National Data on Adolescent Behavior as a
                 Measure o
               * [20]Some States Have Made Additional Efforts to Assess the
                 Effec
               * [21]ASPE, CDC, and NIH Are Funding Studies Designed to
                 Assess th
               * [22]Other Researchers Have Also Made Efforts to Assess the
                 Effec

          * [23]Several Factors Limit the Conclusions That Can Be Drawn abou

               * [24]Experts Have Developed Criteria to Evaluate Efforts to
                 Asses
               * [25]Most Efforts to Assess the Effectiveness of
                 Abstinence-until
               * [26]Most Efforts to Assess the Effectiveness of
                 Abstinence-until
               * [27]Results of Some Scientifically Valid Assessment Efforts
                 Have

     * [28]Conclusions
     * [29]Recommendation for Executive Action
     * [30]Agency Comments and Our Evaluation

          * [31]HRSA Used Full and Open Competition Procedures to Award the
          * [32]HRSA Identified No Actual or Potential Organizational Confli

     * [33]GAO Contact
     * [34]Acknowledgments
     * [35]GAO's Mission
     * [36]Obtaining Copies of GAO Reports and Testimony

          * [37]Order by Mail or Phone

     * [38]To Report Fraud, Waste, and Abuse in Federal Programs
     * [39]Congressional Relations
     * [40]Public Affairs

Report to Congressional Requesters

United States Government Accountability Office

GAO

October 2006

ABSTINENCE EDUCATION

Efforts to Assess the Accuracy and Effectiveness of Federally Funded
Programs

GAO-07-87

Contents

Letter 1

Results in Brief 5
Background 7
Federal and State Efforts to Assess the Scientific Accuracy of Materials
Used in Abstinence-until-Marriage Education Programs Have Been Limited 14
A Variety of Efforts Have Been Made to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs, but a Number of Factors
Limit the Conclusions That Can Be Drawn 19
Conclusions 39
Recommendation for Executive Action 40
Agency Comments and Our Evaluation 40
Appendix I HRSA's Technical Assistance Contract for Abstinence Education
44
Appendix II Methodology for Identifying and Reviewing Research Studies 49
Appendix III Comments from the Department of Health and Human Services 51
Appendix IV GAO Contact and Staff Acknowledgments 57

Tables

Table 1: Definition of Abstinence Education 9
Table 2: Funding Provided by HHS for the Three Main
Abstinence-until-Marriage Education Programs 12
Table 3: ACF's Reporting Requirements for the State Program and the
Community-Based Program, Fiscal Year 2006 21

Abbreviations

ACF Administration for Children and Families AFL Adolescent Family Life
ASPE Office of the Assistant Secretary for Planning and Evaluation CDC
Centers for Disease Control and Prevention FAR Federal Acquisition
Regulation HHS Department of Health and Human Services HIV human
immunodeficiency virus HPV human papillomavirus HRSA Health Resources and
Services Administration NAC National Abstinence Clearinghouse NIH National
Institutes of Health OMB Office of Management and Budget OPA Office of
Population Affairs RFP request for proposal STD sexually transmitted
disease TANF Temporary Assistance for Needy Families

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

October 3, 2006

Congressional Requesters

Preventing sexually transmitted diseases (STD) and unintended pregnancies
among adolescents is an important public health challenge. Although
pregnancy and birth rates among female adolescents in the United States
have been declining since the early 1990s, the rates continue to be high
when compared with those in other industrialized nations. The Centers for
Disease Control and Prevention (CDC) reports that about 141,000 children
were born to girls 17 years old and younger in the United States in 2003.
CDC also reports that STDs disproportionately affect adolescents, with
adolescents and young adults ages 15 to 24 acquiring almost half of the
estimated 19 million new infections each year.

Reducing the incidence of STDs and unintended pregnancies among
adolescents is an important objective for the Department of Health and
Human Services (HHS), which identifies as one of its goals the need to
reduce major threats to the health and well-being of Americans. Among the
efforts it supports to reduce the incidence of STDs and unintended
pregnancies among adolescents, HHS funds abstinence-until-marriage
education programs. Abstinence-until-marriage education programs, also
referred to as abstinence-only education programs, teach adolescents to
abstain from sexual activity until marriage in order to avoid risks of
unintended pregnancy, STDs, and related health problems.^1 The content of
federally funded abstinence-until-marriage programs is required to be
consistent with several principles, such as teaching that a mutually
faithful monogamous relationship in the context of marriage is the
expected standard of human sexual activity, and that abstinence from
sexual activity is the only certain way to avoid STDs.
Abstinence-until-marriage education programs are delivered by a variety of
entities, including schools, human service agencies, faith-based
organizations, youth development groups, and pregnancy crisis centers.
Instructors can incorporate a variety of educational materials into their
abstinence-until-marriage education programs, including textbooks, student
manuals, brochures, slide presentations, and videos.

^1Abstinence-until-marriage education programs also support HHS's
objective to promote family formation and healthy marriages.

The three main federally funded abstinence-until-marriage programs are the
Abstinence Education Program (State Program), which is administered by
HHS's Administration for Children and Families (ACF); the Community-Based
Abstinence Education Program (Community-Based Program), which is also
administered by ACF; and the Adolescent Family Life (AFL) Program, which
is administered by HHS's Office of Population Affairs (OPA) within the
Office of Public Health and Science. Funding provided by HHS for the three
abstinence-until-marriage programs increased from about $73 million in
fiscal year 2001 to about $158 million in fiscal year 2005.

Recent studies have raised concerns about the accuracy of educational
materials used in abstinence-until-marriage education programs and about
the effectiveness of these programs. These studies have reported that some
of the materials used in abstinence-until-marriage education programs
contain, for example, scientifically inaccurate information about anatomy
and physiology as they relate to reproductive health as well as misleading
information about contraceptive failure rates and STDs.^2 State and
federal agencies have also documented inaccuracies in
abstinence-until-marriage educational materials. Further, studies
examining the effectiveness of these programs have reported varied
results. For example, some researchers have reported that
abstinence-until-marriage education programs have resulted in adolescents
reporting less frequent sexual intercourse or fewer sexual partners, while
other researchers have reported that these types of programs did not
affect the frequency of sexual intercourse or were ineffective in delaying
the initiation of sexual intercourse.^3

You asked us to describe certain aspects of the oversight of federally
funded abstinence-until-marriage education programs. Our objectives were
to report on (1) efforts by HHS and states to assess the scientific
accuracy of materials used in abstinence-until-marriage education programs
and (2) efforts by HHS, states, and researchers to assess the
effectiveness of abstinence-until-marriage education programs. You also
asked us to describe how HHS selected a contractor for the
abstinence-until-marriage technical assistance contract that was awarded
in September 2002. This information is provided in appendix I.

^2See K. L. Wilson et al., "A Review of 21 Curricula for
Abstinence-Only-Until-Marriage Programs," The Journal of School Health,
vol. 75, no. 3 (2005), and The Content of Federally Funded Abstinence-Only
Education Programs, United States House of Representatives, Committee on
Government Reform-Minority Staff, Special Investigations Division (2004).

^3See, for example, E. A. Borawski et al., "Effectiveness of
Abstinence-only Intervention in Middle School Teens," American Journal of
Health Behavior, vol. 29, no. 5 (2005), and J. B. Jemmott III, L. S.
Jemmott, and G. T. Fong, "Abstinence and Safer Sex HIV Risk-Reduction
Interventions for African American Adolescents: A Randomized Controlled
Trial," Journal of the American Medical Association, vol. 279, no. 19
(1998).

To describe the efforts by HHS and states to assess the scientific
accuracy of materials used in abstinence-until-marriage education
programs, we reviewed published reports, program announcements, Federal
Register notices, agency Web sites, and other documents related to
abstinence-until-marriage education. We focused our review on efforts
related to the three main federally funded abstinence-until-marriage
education programs administered by HHS, as well as efforts to review the
accuracy of scientific facts included in abstinence-until-marriage
education materials. We did not assess the criteria used to determine the
scientific accuracy of education materials or the quality of the reviews.
We interviewed officials from ACF, the Health Resources and Services
Administration (HRSA), OPA, and CDC. We also interviewed officials from
the 10 states that received the largest share of federal funding (together
accounting for 51 percent of the total funding in fiscal year 2005)
through the State Program for abstinence-until-marriage education.^4

To describe efforts by HHS, states, and researchers to assess the
effectiveness of abstinence-until-marriage education programs, we focused
on efforts that examined the extent to which these programs achieved their
program goals. In general, these goals include teaching adolescents to
abstain from sexual activity until marriage in order to avoid unintended
pregnancies, STDs, and related health problems. As part of our review, we
compared these efforts to the design characteristics that experts have
identified as important for a scientifically valid study of program
effectiveness.^5 We reviewed journal articles and other published reports,
agency budget submissions, program announcements, agency and grantee
performance reports, Federal Register notices, agency Web sites, and other
documents related to abstinence-until-marriage education. (For a more
detailed description of our literature review methodology, see app. II).
We also interviewed officials from ACF, HRSA, OPA, CDC, the National
Institutes of Health (NIH), the Office of the Assistant Secretary for
Planning and Evaluation (ASPE), and 10 states that received the largest
share of federal funding for abstinence-only education through the State
Program in fiscal year 2005. We also interviewed individuals from the
National Campaign to Prevent Teen Pregnancy, The Brookings Institution,
ETR Associates, The Heritage Foundation, and Advocates for Youth, and
researchers from Case Western Reserve University and Columbia University
to obtain general information regarding the state of the research on
abstinence-until-marriage education. We focused our review on efforts to
assess the scientific accuracy of materials and the effectiveness of the
programs during fiscal year 2006, and also reviewed the administration of
the programs back to fiscal year 2001. We also attended conferences
organized by ACF and OPA to learn about training that is provided to
grantees on scientific accuracy and program evaluations.

^4The 10 states that received the largest share of funding in fiscal year
2005 through the State Program were Arizona, Florida, Georgia, Illinois,
Louisiana, Michigan, New York, North Carolina, Ohio, and Texas.

To describe how HHS selected a contractor for the
abstinence-until-marriage technical assistance contract that was awarded
in September 2002, we reviewed the Request for Proposals and other related
contract documents. We also interviewed officials at HRSA, ACF, and the
National Abstinence Clearinghouse about the technical assistance contract.
We performed our work from October 2005 through September 2006 in
accordance with generally accepted government auditing standards.

^5See Douglas Kirby, Emerging Answers: Research Findings on Programs to
Reduce Teen Pregnancy (Washington, D.C.: National Campaign to Prevent Teen
Pregnancy, 2001). The experts identifying the design characteristics of a
scientifically valid study for the National Campaign to Prevent Teen
Pregnancy were drawn from institutions that include the National
Institutes of Health, the Medical Institute for Sexual Health, the Alan
Guttmacher Institute, the Institute for Research and Evaluation, and
various universities. See David Satcher, The National Consensus Process on
Sexual Health and Responsible Sexual Behavior: Interim Report (Atlanta:
Morehouse School of Medicine, 2006). The panel convened by David Satcher
included experts from a variety of organizations, including the Medical
Institute for Sexual Health, the Alan Guttmacher Institute, and the
American Academy of Pediatrics. In addition, characteristics of a
scientifically valid study have been identified by other experts in the
field of evaluation research. For example, see Carol H. Weiss, Evaluation
(Upper Saddle River: Prentice Hall, 1998).

Results in Brief

Efforts by HHS and states to assess the scientific accuracy of materials
used in abstinence-until-marriage education programs have been limited.
This is because ACF--which awards grants through two programs that account
for the largest portion of federal spending on abstinence-until-marriage
education--does not review its grantees' education materials for
scientific accuracy and does not require grantees of either program to
review their own materials for scientific accuracy. In addition, not all
states that receive funding through ACF's State Program have chosen to
review their program materials for scientific accuracy. In particular, 5
of the 10 states in our review conduct such reviews. Officials from these
states reported using a variety of approaches in their reviews. In
contrast, OPA does review the scientific accuracy of AFL grantees'
proposed educational materials and any inaccuracies found must be
corrected before the materials can be used. While the extent to which
federally funded abstinence-until-marriage education materials are
inaccurate is not known, in the course of their reviews OPA and some
states reported that they have found some inaccuracies in
abstinence-until-marriage education materials. For example, OPA has
required that a grantee correct several statements in a true/false
quiz--including statements about STDs and condom use--in order for the
quiz to be approved for use in its curriculum. In addition, one state
official described an instance in which abstinence-until-marriage
materials incorrectly suggested that HIV can pass through condoms because
the latex used in condoms is porous.

HHS, states, and researchers have made a variety of efforts to assess the
effectiveness of abstinence-until-marriage education programs; however, a
number of factors limit the conclusions that can be drawn about the
effectiveness of abstinence-until-marriage education programs. To assess
the effectiveness of their abstinence-until-marriage education programs,
ACF and OPA have required their grantees to report on various outcomes
that the agencies use to measure the effectiveness of grantees'
abstinence-until-marriage education programs. For example, as of fiscal
year 2006, states that receive funding through the State Program are
required to report annually on four measures of the prevalence of
adolescent sexual behavior in their state, such as the rate of pregnancy
among adolescents aged 15 to 17 years. To assess the effectiveness of both
its State and Community-Based Programs, ACF also analyzes trends in
adolescent behavior, as reflected in national data on birth rates among
teens and the proportion of surveyed high school students reporting that
they have had sexual intercourse. OPA requires grantees of the AFL Program
to develop and report on outcome measures that demonstrate the extent to
which grantees' programs are having an effect on program participants. In
addition, other HHS agencies and offices--ASPE, CDC and NIH--are making
efforts to assess the effectiveness of abstinence-until-marriage education
programs. Further, 6 of the 10 states in our review that receive funding
through the State Program have worked with third-party evaluators to
assess the effectiveness of abstinence-until-marriage education programs
in their states. Several factors, however, limit the conclusions that can
be drawn about the effectiveness of abstinence-until-marriage education
programs. Most of the efforts to evaluate the effectiveness of
abstinence-until-marriage education programs that we describe in our
review have not met certain minimum criteria--such as random assignment of
participants and sufficient follow-up periods and sample sizes--that
experts have concluded are necessary in order for assessments of program
effectiveness to be scientifically valid, in part because such designs can
be expensive and time-consuming to carry out. In addition, the results of
efforts that meet the criteria of a scientifically valid assessment have
varied and two key studies funded by HHS that meet these criteria have not
yet been completed. When completed, these HHS-funded studies may add
substantively to the body of research on the effectiveness of
abstinence-until-marriage education programs.

To address concerns about the scientific accuracy of materials used in
abstinence-until-marriage education programs, we recommend that the
Secretary of HHS develop procedures to help assure the accuracy of such
materials used in the State and Community-Based Programs. To help provide
such assurance, the Secretary could consider alternatives such as (1)
extending the approach currently used by OPA to review the scientific
accuracy of the factual statements included in abstinence-until-marriage
education to materials used by grantees of ACF's Community-Based Program
and requiring grantees of ACF's State Program to conduct such reviews or
(2) requiring grantees of both programs to sign written assurances in
their grant applications that the materials they propose using are
accurate.

In commenting on a draft of this report, HHS agreed to consider requiring
grantees of both ACF programs to sign written assurances in grant
applications that the materials they use are accurate. In addition, HHS
noted that all federal grant applicants attest on a standard form that
information in their applications is correct. However, it is not clear
that this serves the purpose of assuring the scientific accuracy of the
educational materials. Further, the curricula to be used are not required
to be included with states' applications. HHS's written comments also
stated that ACF requires that the Community-Based Program curricula
conform to standards that are grounded in scientific literature by
requiring certain types of information. However, the inclusion of certain
types of information does not necessarily ensure the accuracy of the
scientific facts included in the abstinence-until-marriage materials. In
addition, HHS noted in its written comments that we did not define the
term scientific accuracy and stated that it disagreed with certain
findings of the report because it was difficult to precisely determine the
criteria employed by GAO in making the recommendation as to scientific
accuracy. However, the objective of our work was to focus on efforts by
HHS and states to review the accuracy of scientific facts included in
abstinence-until-marriage education materials and not to perform an
independent assessment of the criteria used or the quality of the reviews.
With regard to effectiveness, HHS agreed that it may be too soon to draw
conclusions about the effectiveness of ACF's and OPA's programs.

Background

Recent statistics from CDC show that many high school students engage in
sexual behavior that places them at risk for unintended pregnancy and
STDs. In 2005, 46.8 percent of high school students reported that they
have ever had sexual intercourse, with 14.3 percent of students reporting
that they had had sexual intercourse with four or more persons. The
likelihood of ever having sexual intercourse varied by grade, with the
highest rate among 12th grade students (63.1 percent) and the lowest rate
among 9th grade students (34.3 percent). CDC also has reported that the
prevalence of certain STDs--including the rate of chlamydia infection, the
most frequently reported STD in the United States--peaks in adolescence
and young adulthood. According to CDC, in 2004 the chlamydia rates among
adolescents 15 to 19 years old (1,579 cases per 100,000 adolescents) and
young adults 20 to 24 years old (1,660 cases per 100,000) were each more
than twice the rates among all other age groups.

HHS's current strategic plan includes the objectives to reduce the
incidence of STDs and unintended pregnancies and to promote family
formation and healthy marriages. These two objectives support HHS's goals
to reduce the major threats to the health and well-being of Americans and
to improve the stability and healthy development of American children and
youth, respectively. Abstinence-until-marriage education programs are one
of several types of programs that support these objectives.^6

The three main federal abstinence-until-marriage education programs--the
State Program, the Community-Based Program, and the AFL Program--provide
grants to support the recipients' own efforts to provide
abstinence-until-marriage education at the local level.^7 These programs
must comply with the statutory definition of abstinence education (see
table 1).^8

6HHS's Family Planning Program, for example, also supports the objective
to reduce the incidence of STDs and unintended pregnancies by providing
access to contraceptive supplies and family planning information,
especially for low-income persons, at community health clinics. This
program is authorized under Title X of the Public Health Service Act.

^7There are other federal sources of funding that are used for abstinence
education, such as the Temporary Assistance for Needy Families (TANF)
Program that is administered by ACF. Some states have allocated some of
their TANF funding for abstinence education programs. For example, Florida
has used TANF funds to provide community-based and faith-based
organizations with contracts to carry out abstinence education. Other
sources of funding that are used for abstinence education include ACF's
Compassion Capital Fund and CDC's Division of Adolescent and School Health
grants.

^842 U.S.C. S 710(b)(2). This definition is also referred to as the A-H
definition. This statutory provision defines abstinence education for
purposes of the State Program. Annual appropriations acts and program
announcements have extended this definition to the Community-Based and AFL
Programs. See, e.g , Departments of Labor, Health and Human Services, and
Education, and Related Agencies Appropriations Act, 2006,Pub. L. No.
109-149,119 Stat. 2833, 2855-56.

Table 1: Definition of Abstinence Education

Abstinence education refers to an educational or motivational program      
that:                                                                      
A. has as its exclusive purpose, teaching the social, psychological, and   
health gains to be realized by abstaining from sexual activity;            
B. teaches abstinence from sexual activity outside marriage as the         
expected standardfor all school age children;                              
C. teaches that abstinence from sexual activity is the only certain way to 
avoid out-of-wedlock pregnancy, sexually transmitted diseases, and other   
associated health problems;                                                
D. teaches that a mutually faithful monogamous relationship in context of  
marriage is the expected standard of human sexual activity;                
E. teaches that sexual activity outside of the context of marriage is      
likely to have harmful psychological and physical effects;                 
F. teaches that bearing children out-of-wedlock is likely to have harmful  
consequences for the child, the child's parents, and society;              
G. teaches young people how to reject sexual advances and how alcohol and  
drug use increases vulnerability to sexual advances; and                   
H. teaches the importance of attaining self-sufficiency before engaging in 
sexual activity.                                                           

Source: Social Security Act, S 510(b)(2) (codified at 42 U.S.C. S
710(b)(2)).

The State Program

The State Program, administered by ACF, provides funding to its
grantees--states--for the provision of abstinence-until-marriage education
to those most likely to have children outside of marriage.^9 States that
receive grants through the State Program have discretion in how they use
their funding to provide abstinence-until-marriage education. Some require
that organizations apply for funds and use them to administer
abstinence-until-marriage education programs. Others may directly
administer such programs. At their discretion, states may also provide
mentoring, counseling, and adult supervision to adolescents to promote
abstinence from sexual activity until marriage.

The Personal Responsibility and Work Opportunity Reconciliation Act of
1996 established the State Program, and states were awarded grants
beginning in fiscal year 1998.^10 Funds are allotted to each state that
submits the required annual application based on the ratio of the number
of low-income children in the state to the total number of low-income
children in all states. States are required to match every $4 they receive
in federal money with $3 of nonfederal money and are required to report
annually on the performance of the abstinence-until-marriage education
programs that they support or administer. In fiscal year 2005, 47 states,
the District of Columbia, and 3 insular areas were awarded funding.^11,12

^9Through the State Program funds are also provided to insular areas and
the District of Columbia.

^10Pub. L. No. 104-193, S 912; 110 Stat. 2353-54 (codified at 42 U.S.C. S
710).

The Community-Based Program

The Community-Based Program, which is also administered by ACF, is focused
on funding public and private entities that provide
abstinence-until-marriage education for adolescents from 12 to 18 years
old, with the purpose of creating an environment within communities that
supports adolescent decisions to postpone sexual activity until marriage.
The Community-Based Program provides grants for school-based programs,
adult and peer mentoring, and parent education groups. The Community-Based
Program first awarded grants in fiscal year 2001.^13 Grantees of the
Community-Based Program are selected through a competitive process and are
evaluated according to several criteria, such as the extent to which they
have demonstrated that a need exists for abstinence-until-marriage
education for a targeted population or in a specific geographic location.
Grantees are required to report to ACF, on a semiannual basis, on the
performance of their programs. For fiscal year 2005, 63 grants were
awarded to organizations and other entities.^14

11In this report, we refer to U.S. territories and commonwealths as
"insular areas."

^12Some states and insular areas have not applied for funding under the
State Program. California, Maine, and Pennsylvania did not apply for
funding under the State Program in fiscal year 2005. In this report, when
we refer to "states," we are referring to all grantees of the State
Program--including states, insular areas, and the District of Columbia.

^13The Community-Based Program is conducted under section 1110 of the
Social Security Act. See 42 U.S.C. S 1310.

^14In addition to the 63 grants awarded in fiscal year 2005, ACF is also
responsible for other grants that the agency awarded before 2005.

The AFL Program

The AFL Program supports programs that provide abstinence-until-marriage
education.^15 The primary purpose of these programs is to find effective
means of reaching preadolescents and adolescents before they become
sexually active in order to encourage them to abstain from sexual activity
and other risky behaviors. Under the AFL Program, OPA awards competitive
grants to public or private nonprofit organizations or agencies, including
community-based and faith-based organizations, to facilitate
abstinence-until-marriage education in a variety of settings, including
schools and community centers. Established in 1981, the AFL Program began
awarding grants in fiscal year 1982. AFL Program grantees include school
districts, youth development groups, and medical centers. Grant applicants
are evaluated based on several criteria, such as the extent to which they
provide a clear statement of mission, goals, measurable objectives, and a
reasonable method for achieving their objectives. Grantees are required to
conduct evaluations of certain aspects of their programs and report
annually on their performance. As of August 2006, OPA funded 58
abstinence-until-marriage education programs, and most of these were
focused on reaching young adolescents from the ages of 9 to 14.

Funding for Abstinence-until-Marriage Education Has Increased

Funding provided by HHS for abstinence-until-marriage education programs
has increased steadily since 2001 (see table 2). For the three main
programs combined--the State Program, the Community-Based Program, and the
AFL Program--the amount of agency funding increased from about $73 million
in fiscal year 2001 to about $158 million in fiscal year 2005. Nearly all
of this increase was for the Community-Based program; funding under this
program increased by about $84 million from fiscal years 2001 through
2005. In fiscal year 2005, agency funding for the Community-Based Program
constituted the largest share of the total funding (about 66 percent) for
the three main programs combined.

^15See 42 U.S.C. S 300z et seq. The AFL Program also supports other
projects for pregnant and parenting adolescents, their infants, male
partners, and family members. The purpose of these projects is to improve
the outcomes of early childbearing for teen parents, their infants, and
their families. However, in this report, when we use the term "AFL
Program," we are referring only to the abstinence-until-marriage component
of the AFL Program.

Table 2: Funding Provided by HHS for the Three Main
Abstinence-until-Marriage Education Programs

Dollars in millions                                               
Fiscal year State Program^a,b Community-Based Program AFL Program 
2001                      $43                     $20         $10 
2002                       43                      40          12 
2003                       44                      55          13 
2004                       41                      75          13 
2005                       41                     104          13 

Sources: ACF, OPA, and HRSA.

Notes: Figures are rounded to nearest $1,000,000. Funding levels represent
the total amount of grants awarded and funding for program-related
support, such as technical assistance and evaluation.

aStates that receive funding are required to match every $4 they receive
of federal funds with $3 of nonfederal money.

bThe amount of funding provided by HHS for the State Program has generally
varied by year because the states that have applied for funding each year
have varied.

Within each of the three main abstinence-until-marriage education
programs, the amount of individual grants varied.^16 In fiscal year 2005,
the State Program's annual grants ranged from $57,057 to $4,777,916 and
the median annual grant amount was $569,675. That same year, the
Community-Based Program's annual grants ranged from $213,276 to $800,000
and the median grant amount was $642,250. In fiscal year 2006, the AFL
Program's annual grants ranged from $95,676 to $300,000 and the median
grant amount was $225,000.

Federal Agency Responsibilities Related to Abstinence-until-Marriage Education

Five organizational units located within HHS--ACF, OPA, CDC, ASPE, and
NIH--have responsibilities related to abstinence-until-marriage education.
ACF and OPA administer the three main federal abstinence-until-marriage
education programs. CDC supports abstinence-until-marriage education at
the national, state, and local levels. CDC, ASPE, and NIH are sponsoring
research on the effectiveness of abstinence-until-marriage programs.

^16ACF awards formula grants under the State Program each year, and states
have 2 years to spend the funds they are awarded. In the Community-Based
Program and AFL Program, grantees develop multiyear projects--up to 5
years--for which the first year of funding is provided through competitive
grants; for subsequent years, grantees may obtain funding through
noncompetitive continuation grants.

  ACF

ACF is responsible for federal programs that promote the economic and
social well-being of families, children, individuals, and communities. ACF
administers and provides oversight of both the State Program and the
Community-Based Program by, among other things, awarding grants, providing
training and technical assistance to grantees, and requiring annual
performance reporting from grantees. ACF has been responsible for the
State Program since June 2004 and the Community-Based Program since
October 2005. HRSA previously administered these programs.

  OPA

OPA has responsibility for advising the Secretary of HHS on a wide range
of reproductive health topics, including adolescent pregnancy and family
planning. The office is also responsible for administering programs that
provide services for pregnant and parenting teens and prevention programs,
such as abstinence-until-marriage education programs. OPA administers and
provides oversight of the AFL Program by awarding grants, providing
training and technical assistance to grantees, and requiring annual
performance reporting from grantees.

  CDC

CDC is primarily responsible for the prevention and control of infectious
and chronic diseases, including STDs. CDC provides funding to state and
local education agencies in their efforts to support comprehensive school
health education and HIV/STD prevention education programs, and CDC
officials told us that some of these are focused on abstinence. CDC also
provides funding to several state education agencies to implement various
abstinence projects, such as collaboration-building among agencies to
increase the impact of their efforts to encourage abstinence. Further, CDC
develops tools to assist state and local education agencies with their
health education programs. CDC provides funding to several national
organizations to build the capacity of abstinence-until-marriage education
providers. Organizations' activities include, but are not limited to, the
development and distribution of educational materials. CDC is also
sponsoring research on the effectiveness of an abstinence-until-marriage
education program.

  ASPE

ASPE advises the Secretary of HHS in several areas, including policy
development in health, human services, data, and science. ASPE is
responsible for the development of policy analyses and it conducts
research and evaluation studies in several areas, including the health of
children and adolescents. ASPE is currently sponsoring research on the
effectiveness of abstinence-until-marriage education programs.

  NIH

NIH is the primary federal agency that conducts and supports medical and
behavioral research among various populations, including children and
adolescents. NIH is currently sponsoring research on the effectiveness of
abstinence-until-marriage education programs.

Federal and State Efforts to Assess the Scientific Accuracy of Materials Used in
Abstinence-until-Marriage Education Programs Have Been Limited

Efforts by HHS and states to assess the scientific accuracy of materials
used in abstinence-until-marriage education programs have been limited.
ACF--which awards grants to two programs that account for the largest
portion of federal spending on abstinence-until-marriage education--does
not review its grantees' education materials for scientific accuracy and
does not require grantees of either program to review their own materials
for scientific accuracy. In addition, not all states funded through the
State Program have chosen to review their program materials for scientific
accuracy. In contrast to ACF, OPA has reviewed the scientific accuracy of
grantees' proposed educational materials and corrected inaccuracies in
these materials.

ACF Does Not Review Program Materials for Scientific Accuracy and Does Not
Require Grantees to Do So, though Some State Grantees Have Conducted Such
Reviews

There have been limited efforts to review the scientific accuracy of
educational materials used in ACF's State and Community-Based
Programs--the two programs that account for the largest portion of federal
spending on abstinence education. ACF does not review materials for
scientific accuracy in either reviewing grant applications^17 or in
overseeing grantees' performance. Prior to fiscal year 2006, State Program
and Community-Based Program applicants were not required to submit copies
of their proposed educational materials with their applications. While ACF
required grantees of the Community-Based Program--but not the State
Program--to submit their educational materials with their fiscal year 2006
applications, ACF officials told us that grantee applications and
materials are only reviewed to ensure that they address all aspects of the
scope of the Community-Based Program, such as the A-H definition of
abstinence education.^18 Further, documents provided to us by ACF indicate
that the agency does not review grantees' educational materials for
scientific accuracy as a routine part of its oversight activities. In
addition, ACF also does not require its grantees to review their own
materials for scientific accuracy. Similarly, when HRSA was responsible
for the State and Community-Based Programs, the agency did not review
materials used by grantees for scientific accuracy or require grantees to
review their own materials.

^17In reviewing grantees applications, ACF does examine several issues,
including applicants' stated program goals and need for assistance, their
compliance with the A-H definition of abstinence education, their intended
approach in carrying out their objectives, and their budget plan.

^18HHS officials told us that if ACF finds inaccurate statements during
this more general review process or if inaccuracies are brought to their
attention at any time during the grant period, ACF officials work with the
grantees to take corrective action.

Not all grantees of the State Program have chosen to review the scientific
accuracy of their educational materials. Officials from 5 of the 10 states
in our review reported that their states have chosen to conduct such
reviews.^19,20 Officials in these states identified a variety of reasons
why their states reviewed abstinence-until-marriage educational materials,
including program requirements, state education laws and guidelines, and
past lawsuits, to ensure that materials used in abstinence-until-marriage
programs were accurate. For example, Michigan's Revised School Code states
that materials and instruction in the sex education curricula, including
information on abstinence, "shall not be medically inaccurate,"^21 and
Ohio's fiscal year 2007 abstinence-until-marriage education program
guidance states that abstinence-until-marriage educational materials
"should be medically accurate in all assertions."^22

The five states we contacted that review abstinence-until-marriage
educational materials for scientific accuracy have used a variety of
approaches in their reviews. Some states contracted with medical
professionals--such as nurses, gynecologists, and pediatricians--to serve
as medical advisors who review program materials and use their expertise
to determine what is and is not scientifically accurate. Some states have
created checklists or worksheets to guide their staff conducting the
review and document findings of inaccuracy or verification of a statement.
All five states use medical professionals in conducting these reviews. One
of the states requires that all statistics or scientific statements cited
in a program's materials are sourced to CDC or a peer-reviewed medical
journal. Officials from this state told us that if statements in these
materials cannot be attributed to these sources, the statements are
required to be removed until citations are provided and materials are
approved. Officials from this state told us they have also supplemented
their review of program materials with on-site classroom observations to
assess the scientific accuracy of the information presented to students.

^19In addition to reviewing materials for accuracy, one state requires
abstinence-until-marriage providers to sign a written assurance that their
materials are scientifically accurate. Officials from this state also
reported providing abstinence-until-marriage education programs with
public health consultants to provide technical assistance and training to
help ensure the accuracy of their educational materials.

^20In addition, some state officials we interviewed told us that review
committees for local school districts may review the scientific accuracy
of educational materials that include information about HIV and other
STDs, including abstinence-until-marriage education materials.

^21Mich. Comp. Laws Ann. S 380.1507b(2)(West 2004).

^22See Ohio Department of Health, "Abstinence Education Program Request
for Proposals for Fiscal Year 2007," (program announcement, 2005).

Officials from two of the five states reported that they have found
inaccuracies as a result of their reviews. For example, one state official
stated that because information is constantly evolving, state officials
have had to correct out-of-date scientific information. In addition, this
official cited an instance where materials incorrectly suggested that HIV
can pass through condoms because the latex used in condoms is porous. In
addition, this official provided documentation that the state has had to
correct a statement indicating that when a person is infected with the
human papillomavirus,^23 the virus is "present for life" because, in
almost all cases, this is not true. State officials who have identified
inaccuracies told us that they informed their grantees of inaccuracies so
that they could make corrections in their individual programs. One state
official added that she contacted the authors of the materials to report
an inaccuracy.

Some of the educational materials that states have reviewed are materials
that are commonly used in the Community-Based Program. Officials from four
of the five states that review materials for scientific accuracy told us
that they have each reviewed at least one of the five curricula most
commonly used in the Community-Based Program because programs in their
state were using them: Choosing the Best, WAIT Training, Sex Can Wait,
A.C. Green's Game Plan Abstinence Program, and Worth the Wait. Based on
ACF documents, we found that there were 58 different curricula used by
grantees of the Community-Based Program in fiscal year 2005. However, more
than half of the grantees of the Community-Based Program reported using at
least one of these five curricula.^24

While there has been limited review of materials used in the State and
Community-Based Programs, grantees of these programs have received some
technical assistance designed to improve the scientific accuracy of their
materials. For example, ACF officials reported that the agency provided a
conference for grantees of the Community-Based Program in February 2006
that included a presentation focused on medical accuracy, including a
discussion of state legislative proposals that would require medical
accuracy in abstinence-until-marriage education, and how to identify
reliable data. In addition, in 2002, HRSA awarded a contract to the
National Abstinence Clearinghouse requiring, among other things, that the
contractor develop and implement a program to provide medically accurate
information and training to grantees of the State and Community-Based
Programs.^25 (See app. I for a description of HRSA's process for awarding
this contract). The portion of the contract that focused on providing
medically accurate information to grantees was subcontracted to the
Medical Institute for Sexual Health (Medical Institute),^26 which has
conducted presentations at regional educational conferences to provide
grantees with medical and scientific information, such as updated
information on condoms and STD transmission. The Medical Institute has
also provided consultative services to grantees by responding to medical
and scientific questions.

^23The human papillomavirus (HPV) causes an STD called genital HPV
infection.

^24Some grantees of the Community-Based Program reported using more than
one of these curricula in fiscal year 2005.

OPA Reviews Materials Used by AFL Program Grantees for Scientific Accuracy

In contrast to ACF, OPA reviews for scientific accuracy the educational
materials used by AFL Program grantees. Specifically, OPA reviews its
grantees' proposed educational materials for scientific accuracy before
they are used. Agency officials stated that they began to review these
materials while litigation concerning the AFL Program was ongoing. OPA
continued to review these materials as part of a 1993 settlement to this
lawsuit.^27 The settlement agreement expired in 1998, though the agency
has continued to review grantees' proposed educational materials for
accuracy as a matter of policy. OPA officials told us that grant
applicants submit summaries of materials they propose to use, though the
materials are not reviewed for scientific accuracy until after grantees
have been selected. OPA officials said that after grants are awarded, a
medical education specialist (in consultation with several part-time
medical experts) reviews the grantees' printed materials and other
educational media, such as videos. OPA officials explained that the
medical education specialist must approve all materials before they are
used. On many occasions, OPA grantees have proposed using--and therefore
OPA has reviewed--materials commonly used in the Community-Based Program.
For example, an OPA official told us that the agency had reviewed three of
the Community-Based Program's commonly used curricula--Choosing the Best,
Sex Can Wait, and A.C. Green's Game Plan Abstinence Program--and is also
currently reviewing another curriculum commonly used by Community-Based
Program grantees, WAIT Training.^28

25The administration of this contract was transferred to ACF in May 2005.

^26The Medical Institute is a nonprofit organization that provides
educational resources, conferences, and seminars to educators, health
professionals, pregnancy care centers, and faith-based groups about
behaviors to decrease STDs and out-of-wedlock pregnancies, including
abstinence.

^27See  generally Bowen v. Kendrick, 487 U.S. 589 (1988), 657 F. Supp 1547
(D.D.C. 1987).

OPA officials stated that the medical education specialist has
occasionally found and addressed inaccuracies in grantees' proposed
educational materials. OPA officials stated that these inaccuracies are
often the result of information being out of date because, for example,
medical and statistical information on STDs changes frequently. OPA has
addressed these inaccuracies by either not approving the materials in
which they appeared or correcting the materials through discussions with
the grantees and, in some cases, the authors of the materials. In fiscal
year 2005, OPA disapproved of a grantee using a specific pamphlet about
STDs because the pamphlet contained statements about STD prevention and
HIV transmission that were considered incomplete or inaccurate. For
example, the pamphlet stated that there was no cure for hepatitis B, but
the medical education specialist required the grantee to add that there
was a preventive vaccine for hepatitis B. In addition, OPA required that a
grantee correct several statements in a true/false quiz--including
statements about STDs and condom use--in order for the quiz to be approved
for use. For example, the medical education specialist changed a sentence
from "The only 100% effective way of avoiding STDs or unwanted pregnancies
is to not have sexual intercourse." to "The only 100% effective way of
avoiding STDs or unwanted pregnancies is to not have sexual intercourse
and engage in other risky behaviors."

^28In addition, a CDC official told us that some of its grantees are
producing educational materials with CDC funds to be used by
abstinence-until-marriage education programs, which are likely to include
State and Community-Based Program grantees. These materials are required
to be reviewed for scientific accuracy. CDC officials told us that they
have made corrections to some of these materials. Materials used in
school-based HIV prevention education programs that are supported with CDC
funds are also reviewed for scientific accuracy. A CDC official told us
that some of these programs are abstinence-until-marriage education
programs.

While OPA and some states have reviewed their grantees'
abstinence-until-marriage education materials for scientific accuracy,^29
these types of reviews have the potential to affect
abstinence-until-marriage education providers more broadly. Such efforts
may create an incentive for authors of abstinence-until-marriage education
materials to ensure they are accurate. Thus, some authors of
abstinence-until-marriage education materials have recently updated
materials in their curricula following reports that questioned their
accuracy. For example, one of the most widely used curricula used by
grantees of the Community-Based Program--WAIT Training--has been recently
updated and provides the updated information on its Web site. A
representative from WAIT Training stated that the company recently revised
its curriculum, in part, in response to a congressional review that found
inaccuracies in its abstinence-until-marriage education materials.

A Variety of Efforts Have Been Made to Assess the Effectiveness of
Abstinence-until-Marriage Education Programs, but a Number of Factors Limit the
Conclusions That Can Be Drawn

HHS, states, and researchers have made a variety of efforts to assess the
effectiveness of abstinence-until-marriage education programs; however, a
number of factors limit the conclusions that can be drawn about the
effectiveness of these programs. ACF and OPA have required their grantees
to report on various outcomes used to measure the effectiveness of
grantees' abstinence-until-marriage education programs,^30 though the
reporting requirements for each of the three abstinence-until-marriage
programs differ. In addition, to assess the effectiveness of the State and
Community-Based Programs, ACF has analyzed national data on adolescent
birth rates and the proportion of adolescents who report having had sexual
intercourse. Other organizational units within HHS--ASPE, CDC, and
NIH--are funding studies designed to assess the effectiveness of
abstinence-until-marriage education programs in delaying sexual
initiation, reducing pregnancy and STD rates, and reducing the frequency
of sexual activity. Despite these efforts, several factors limit the
conclusions that can be drawn about the effectiveness of
abstinence-until-marriage education programs. Most of the efforts to
evaluate the effectiveness of abstinence-until-marriage education programs
that we describe in our review have not met certain minimum criteria that
experts have concluded are necessary in order for assessments of program
effectiveness to be scientifically valid, in part because such designs can
be expensive and time-consuming to carry out. In addition, the results of
some efforts that meet the criteria of a scientifically valid assessment
have varied, and two key studies that meet these criteria have not yet
been completed.

^29In addition to OPA and some states, others have also reviewed the
scientific accuracy of abstinence-until-marriage education materials. See,
for example, Wilson et al.

^30This reporting is a part of ACF's efforts to collect evaluative
information about these programs. These efforts include both performance
measurement--the ongoing monitoring and reporting of program
accomplishments toward preestablished goals--and program
evaluation--individual systematic studies to assess how well a program is
working. Both types of assessments aim to support decisions to improve
service delivery and program effectiveness. See GAO, Performance
Measurement and Evaluation: Definitions and Relationships,
[41]GAO-05-739SP (Washington, D.C.: May 2005), for more information on
types of assessments.

HHS, States, and Researchers Have Made a Variety of Efforts to Assess the
Effectiveness of Abstinence-until-Marriage Education Programs

Efforts of HHS, states, and researchers to assess the effectiveness of
abstinence-until-marriage education programs have included ACF and OPA
requiring grantees to report data on outcomes of their
abstinence-until-marriage education programs; ACF analyzing national data
on adolescent behavior and birth rates; and other HHS agencies, states,
and researchers funding or conducting studies to assess the effectiveness
of abstinence-until-marriage education programs.

  ACF and OPA Have Required Grantee Reporting of Data on Outcomes

ACF has made efforts to assess the effectiveness of
abstinence-until-marriage education programs funded by the State Program
and the Community-Based Program. One of ACF's efforts has been to require
grantees of both programs to report data on outcomes, though the two
programs have different requirements for the outcomes grantees must
report. For the State Program, as of fiscal year 2006, grantees must
report annually on four measures of the prevalence of adolescent sexual
behavior in their states, such as the rate of pregnancy among adolescents
aged 15 to 17 years, and compare these data to program targets over 5
years. To report on these four measures, states may choose the data
sources they will use.^31 States must also develop and report on two
additional performance measures that are related to the goals of their
programs.^32 (See table 3 for a list of ACF's fiscal year 2006 reporting
requirements for the State Program.)

^31Previously, to report on the four measures, states have relied on
either state or national data sources, such as CDC's Youth Risk Behavior
Surveillance System.

^32For example, in fiscal year 2002, state grantees developed such
measures as the percentage of teens surveyed who show an increase in
participating in structured activities after school hours; the percentage
of live births to women younger than 18, fathered by men age 20 and older;
the percentage of program participants proficient in refusal skills; the
percentage of high school students who reported using drugs or alcohol
before intercourse; and the percentage of high school students who had
sexual intercourse for the first time before age 13.

State Program                 Community-Based Program                      
      o Report on four              o Report on program goals that are        
      performance measures: (1)     developed by grantees with a third-party  
      rate of pregnancy among       evaluator. Such outcomes could include    
      female teenagers aged 15      changes in knowledge about abstinence or  
      to 17, (2) proportion of      declared behavior among participants of   
      adolescents who have          abstinence-until-marriage programs.       
      engaged in sexual                                                       
      intercourse, (3) incidence                                              
      of youths 15 to 19 years                                                
      old who have contracted                                                 
      selected STDs, and (4)                                                  
      rate of births among                                                    
      female teenagers aged 15                                                
      to 17.                                                                  
      o Develop and report on       o Report on program "outputs": the number 
      two additional performance    of youth served, the hours of service     
      measures that are related     provided to each youth, and the number of 
      to individual program         youths who complete the program. Grantees 
      goals. Past examples of       choose additional outputs that allow for  
      these additional measures     effective monitoring and management of    
      have included the             the project. The additional outputs may   
      percentage of high school     include tracking the number of staff      
      students who reported         trained to provide services, the number   
      using drugs or alcohol        of events hosted, number of marketing     
      before intercourse and the    materials distributed, and so forth.      
      percentage of high school                                               
      students who had sexual                                                 
      intercourse for the first                                               
      time before age 13.                                                     

Table 3: ACF's Reporting Requirements for the State Program and the
Community-Based Program, Fiscal Year 2006

Sources: State and Community-Based Programs' announcements, fiscal year
2006.

As of fiscal year 2006, ACF requires Community-Based Program grantees to
develop and report on outcome measures designed to demonstrate the extent
to which grantees' community-based abstinence education programs are
accomplishing their program goals.^33 ACF requires grantees of the
Community-Based Program to contract with third-party evaluators, who are
responsible for both helping grantees develop the outcome measures and
monitoring grantee performance against the measures,^34 but because this
is a new requirement established for fiscal year 2006 grantees, ACF has
not yet received the results of these evaluations. In addition to outcome
reporting, ACF requires grantees of the Community-Based Program to report
on program "outputs," which measure the quantity of program activities and
other deliverables, such as the number of participants who are served by
the abstinence-until-marriage education programs. According to ACF
officials, the agency requires grantees of both the State Program and the
Community-Based Program to report on program outcomes in order to monitor
grantees' performance, target training, and technical assistance, and help
grantees improve service delivery. (See table 3 for a list of ACF's fiscal
year 2006 reporting requirements for the Community-Based Program.)

^33The fiscal year 2006 program announcement for the Community-Based
Program provides examples of outcome measures that grantees could use,
including increased knowledge of the benefits of abstinence, the number of
youths who commit to abstaining from premarital sexual activity, and
increased knowledge of how to avoid high-risk situations and risk
behaviors.

^34Fiscal year 2006 Community-Based Program grantees are required to
devote a minimum of 15 percent of their requested budgets to performance
monitoring by third-party contractors.

ACF's fiscal year 2006 reporting requirements for grantees of the State
Program are the same as HRSA's when it administered the State Program. In
contrast, ACF's fiscal year 2006 reporting requirements for the
Community-Based Program differ from HRSA's reporting requirements for the
program. For example, for Community-Based Program grants awarded in fiscal
year 2001, HRSA required^35 grantees to report on the effectiveness of
their programs, as measured by program participation as well as behavioral
and biological outcomes.^36 These performance measures were modified for
fiscal year 2002, in part HHS officials explained, because of concerns
expressed by members of the abstinence-education community that the
original performance measures did not accurately reflect the efforts of
the grantees of the Community-Based Program. For grants awarded from
fiscal years 2002 through 2004, HRSA required grantees of the
Community-Based Program to report on a combination of program outputs,
such as the proportion of adolescents who completed an
abstinence-until-marriage education program, and measures of adolescent
intentions, such as the proportion of adolescents who committed to
abstaining from sexual activity until marriage.^37 For grants awarded in
fiscal year 2005, when ACF assumed responsibility for the Community-Based
Program from HRSA, grantees were not required to report on any specific
performance measures.

^35Some grantees of the Community-Based Program may have to meet reporting
requirements established by HRSA. Grants under this program are awarded
for projects that may extend over a period of several years. Grantees that
were awarded grants when HRSA administered the program and have since
received noncompetitive continuation grants for these projects are
required to meet the reporting requirements in place at the time they
first received the competitively awarded grants.

^36In fiscal year 2001, HRSA required grantees of the Community-Based
Program to report on the following four performance measures: the
proportion of program participants who successfully complete or remain
enrolled in an abstinence-only education program; the proportion of
program participants who have engaged in sexual intercourse; the
proportion of program participants who report a reduction in risk
behaviors, such as tobacco, alcohol, and drug use; and the rate of births
to female program participants.

OPA has also made efforts to assess the effectiveness of the AFL Program.
Specifically, OPA requires grantees of the AFL Program to develop and
report on outcome measures that are used to help demonstrate the extent to
which grantees' programs are having an effect on program
participants.^38,39 According to OPA officials, the agency recommends that
grantees report on outcome measures, such as participants' knowledge of
the benefits of abstinence and their reported intentions to abstain from
sexually activity, reported beliefs in their ability to remain abstinent,
and reported parental involvement in their lives. To collect data on these
outcome measures and any others, OPA requires all grantees funded in
fiscal year 2004 and beyond to administer, at a minimum, a standardized
questionnaire--developed by OPA--to their program participants, both when
participants begin an abstinence-only education program and after the
program's completion.^40 The standardized questionnaire includes questions
intended to obtain information on participants' reported involvement in
extracurricular activities, behaviors linked to health risks, attitudes
and intentions about abstinence, and opinions about the consequences of
premarital sexual activity. Like ACF, OPA requires its grantees to
contract with independent evaluators, such as colleges or universities,
which are responsible for evaluating the effectiveness of grantees'
individual abstinence-until-marriage education programs.^41 In addition to
evaluating the extent to which grantees are meeting their goals, OPA
officials stated that the independent evaluators may also provide input to
grantees of the AFL Program on other aspects of the program to improve
their service delivery. Unlike ACF, OPA requires that the third-party
evaluations incorporate specific methodological characteristics, such as
control groups or comparison groups^42 and sufficient sample sizes.^43 In
addition, OPA requires that the evaluations for grantees funded in fiscal
year 2004 and beyond account for baseline and follow-up data obtained from
the standardized questionnaires.

^37Specifically, HRSA required grantees of the Community-Based Program to
report annually on the following six performance measures: the proportion
of program participants who successfully completed or remained enrolled in
an abstinence-only education program; the proportion of adolescents who
understood that abstinence from sexual activity is the only certain way to
avoid out-of-wedlock pregnancy and STDs; the proportion of adolescents who
indicated an understanding of the social, psychological, and health gains
to be realized by abstaining from premarital sexual activity; the
proportion of participants who reported that they had the skills necessary
to resist sexual urges and advances; the proportion of youth who committed
to abstaining from sexual activity until marriage; and the proportion of
participants who intended to avoid situations and risk behaviors, such as
drug use and alcohol consumption, which make them more vulnerable to
sexual advances and urges.

^38In addition to these outcomes, grantees of the AFL Program are required
to report on program outputs, such as the number of program participants,
the average number of participants per session, and the average number of
sessions attended by participants. Agency officials stated that OPA has
implemented a new format for its grantees' reports, which is intended to
standardize their reporting on these outputs.

^39OPA's grantees are required to perform evaluations of their programs
that are directly tied to their program objectives. For these evaluations,
OPA requires grantees to develop research hypotheses that reflect the
outcomes the grantees intend to achieve. This type of evaluation is
generally considered to be an outcome evaluation--which assesses the
extent to which a program achieves its outcome-oriented objectives. These
evaluations focus on outputs and outcomes to judge program effectiveness
but may also assess program process to understand how outcomes are
produced. In addition, grantees of the AFL Program are required to perform
implementation evaluations.

OPA's requirement that grantees use a standardized set of questionnaires,
with data from these questionnaires used in evaluations, differs from
OPA's previous requirements. Previously, grantees of the AFL Program were
not required to use a standardized method for collecting data that could
be used to assess the effectiveness of their programs; instead, grantees
chose their own data collection instruments. As a result, an OPA official
explained, the collected data varied from one project to another. OPA
officials said that the agency developed the standardized questionnaire to
ensure uniformity in the data collected and allow the agency to more
effectively aggregate the data reported in evaluations of individual
abstinence-until-marriage education programs.

^40OPA officials stated that grantees may also supplement the standardized
questionnaire with additional data collection instruments to obtain
information on the effectiveness of their abstinence-until-marriage
education programs.

^41OPA has required that its grantees perform independent evaluations of
their programs since the program first awarded grants in 1982, and
requires that grantees devote from 1 percent to 5 percent of grant funds
to the evaluation of their programs. In cases where a more rigorous or
comprehensive evaluation is proposed, OPA may allow these grantees to use
up to 25 percent of their grant funds.

^42A control group is a group of individuals or communities in a study
that is compared to an intervention group--a group in a study that is
receiving or participating in the program being studied. A control group
is a randomly assigned group that does not receive the program. A
comparison group is not randomly assigned like a control group. However,
individuals or communities in well-matched comparison groups should have
similar characteristics.

^43Specifically, OPA requires that third-party evaluations of grantees of
the AFL Program compare, when possible, randomized control or matched
comparison groups with groups receiving abstinence-until-marriage
education. In addition, OPA requires that these evaluations include a
sufficient sample size to ensure that any observed differences between the
groups are statistically valid and that the evaluations include a
follow-up assessment of program participants at least 6 months after the
abstinence-until-marriage intervention has been tested.

OPA officials told us that they plan to aggregate information from certain
questions in the standardized set of questionnaires in order to report on
certain performance measures as part of the agency's annual performance
reports.^44 The measures include the extent of parental involvement in
adolescents' lives and the extent to which adolescents understand the
benefits of abstinence. An agency official stated that the agency expects
to begin receiving data from grantees that are using these questionnaires
in January 2007. OPA did not previously have long-term measures of the
performance of the AFL Program. Its current measures were developed in
collaboration with the Office of Management and Budget (OMB) in response
to an OMB review in 2004 that found that the AFL Program did not have any
annual performance measures for measuring progress toward long-term goals.

In addition to requiring their grantees to report on outcomes used to
assess program effectiveness, both ACF and OPA have provided technical
assistance and training to their grantees in order to support grantees'
own program evaluation efforts. For example, in November 2005 the two
agencies sponsored an evaluation conference for abstinence-until-marriage
grantees that included presentations about evaluations and their
methodology. Similarly, ACF's Office of Planning, Research, and Evaluation
sponsors annual evaluation conferences, and an ACF official told us that a
recent conference placed "a significant emphasis" on the evaluation of
abstinence-until-marriage education programs. In addition, HHS officials
told us that ACF, along with ASPE, is funding a multiyear project that is
designed to identify gaps in abstinence education evaluation and technical
assistance needs, develop materials on abstinence education evaluation,
deliver technical assistance and capacity-building activities related to
program evaluation, and develop research reports related to abstinence
education. OPA officials also told us that they attempt to help ensure
grantees' progress and effectiveness by offering various technical
assistance workshops and conferences. For example, in May 2006 OPA
provided a 2-day training conference to its grantees on the importance of
program evaluations and administering evaluation instruments. In addition,
OPA officials stated that the agency contracts with evaluation
consultants, who review grantees' evaluation tools and activities. OPA
officials explained that these consultants provide in-depth technical
assistance to grantees on how to improve grantees' evaluations.

^44OPA prepares annual performance reports as a part of HHS's
responsibilities under the Government Performance and Results Act, which
include program performance measures to help link funding decisions with
performance and review of related outcome measures.

  ACF Uses National Data on Adolescent Behavior as a Measure of the Overall
  Effectiveness of Its Two Programs

Requiring outcome reporting from state and community-based grantees is not
ACF's only effort to assess the effectiveness of its two programs. ACF
also analyzes trends in adolescent behavior, as reflected in national data
on birth rates among teens and the proportion of surveyed high school
students reporting that they have had sexual intercourse.^45 ACF uses
these national data as a measure of the overall effectiveness of its State
and Community-Based Programs, comparing the national data to program
targets. In its annual performance reports, the agency summarizes the
progress being made toward lowering the rate of births to unmarried
teenage girls and the proportion of students (grades 9-12) who report
having ever had sexual intercourse.

ACF's use of national data to assess the effectiveness of the State and
Community-Based Programs represents a change from how HRSA assessed the
overall effectiveness of these programs. Whereas ACF compares national
data on adolescent behavior to program targets, HRSA aggregated data from
its state and community-based grantees. HRSA's state grantees were allowed
to select the data sources used to gauge their progress against certain
performance measures. For example, in its annual performance reports on
the State Program, HRSA reported information on the percentage of its
state grantees meeting target rates for reducing the proportion of
adolescents who have engaged in sexual intercourse, the incidence of
youths aged 15 to 19 who have contracted selected STDs, and the rate of
births among youths aged 15 to 17. To determine their progress in meeting
their target rates, some state grantees, for example, reported national
data from the Youth Risk Behavior Surveillance System, while other
grantees reported state-collected data. After ACF assumed responsibility
for the State and Community-Based Programs from HRSA, ACF began using
national data on adolescent behavior as a measure of the programs'
effectiveness. According to ACF officials, the agency changed how it
assessed its programs out of concern over the quality of the data state
grantees were using in their performance reporting and because the agency
wanted to use parallel measures of effectiveness for both programs. For
example, according to state performance reports for fiscal year 2001 that
we reviewed, two reports did not include adolescent pregnancy rates that
year because the states did not collect data on abortions among this
population.^46 In addition, ACF officials told us that they decided not to
use national data on STDs as a measure of program effectiveness because
the goal of reducing STD rates is not as central to the State and
Community-Based Programs as reducing sexual activity and birth rates among
teens. However, one official stated that reducing STDs is an important
"by-product" of the programs.

^45Data on teen birth rates and adolescents' reported sexual behavior are
contained in the National Vital Statistics System and the Youth Risk
Behavior Surveillance System, respectively. The former is a national data
set of public health statistics reported by states to CDC, and the latter
is a national data set based on nationwide surveys administered to high
school students by CDC.

  Some States Have Made Additional Efforts to Assess the Effectiveness of
  Abstinence-until-Marriage Education Programs

Some states have made additional efforts to assess the effectiveness of
abstinence-until-marriage education programs, although they are not
required by ACF.^47 Specifically, we found that 6 of the 10 states in our
review that receive funding through ACF's State Program have made efforts
to conduct evaluations of selected abstinence-until-marriage programs in
their state. All 6 of the states worked with third-party evaluators, such
as university researchers or private research firms, to perform the
evaluations, which in general measure self-reported changes in program
participants' behavior and attitudes related to sex and abstinence as
indicators of program effectiveness. To obtain this information, the
third-party evaluators have typically relied on surveys administered to
program participants at the start of a program, its conclusion, and during
a follow-up period anywhere from 3 months to almost 3 years after the
conclusion. The third-party evaluations for 4 of the 6 states in our
review have been completed as of February 2006, and the results of these
studies have varied.^48

46In order to estimate pregnancy rates among adolescents, states use data
on both birth rates and abortions among adolescents.

^47ACF does require grantees of the State Program to describe, in their
grant applications, their plans for "tracking activities and measuring
achievement" of their program goals and objectives.

Among the 4 states that have completed third-party evaluations, 3 states
require the abstinence programs in their state to measure reported changes
in participants' behavior as an indicator of program effectiveness--both
at the start of the program and after its completion. The 3 states require
their programs to track participants' reported incidence of sexual
intercourse. In addition, 2 states require their programs to track
biological outcomes, such as pregnancies, births, or STDs. In addition, 6
of the 10 states in our review require their programs to track
participants' attitudes about abstinence and sex, such as the number of
participants who make pledges to remain abstinent.

Some states also provide technical assistance to the
abstinence-until-marriage programs they support in their state. This
assistance is designed to help programs evaluate and improve their
effectiveness. Officials from 5 of the 10 states in our review either told
us or provided documentation that they provide technical assistance on
evaluations to abstinence programs in their state. One state official said
that the abstinence-until-marriage programs supported by the state were
found to be ill-prepared to conduct evaluations themselves, and that she
now requires these programs to dedicate a portion of their grants to
contract with a third-party or state evaluator to assist them in
program-level evaluations. Officials from another state told us that they
contract with a private organization of public health professionals in
order to provide evaluation consultation and technical assistance for the
abstinence-until-marriage programs the state supports.

^48See, for example, LeCroy & Milligan Associates, Inc., Abstinence Only
Education Program: Fifth Year Evaluation Report, a report prepared for the
Arizona Department of Health Services, 2003; Patricia Goodson et al.,
Abstinence Education Evaluation: Phase 6, a report prepared for the Texas
Department of State Health Services, 2005; MGT of America, Evaluation of
Georgia Abstinence Education Programs Funded Under Title V, Section 510, a
report prepared for the Georgia Department of Human Resources, 2005;
Thomas E. Smith, It's Great to Wait: An Interim Evaluation, a report
prepared for the Florida Department of Health, 2001.

  ASPE, CDC, and NIH Are Funding Studies Designed to Assess the Effectiveness of
  Abstinence-until-Marriage Education Programs

In addition to ACF and OPA, other organizational units within HHS have
made efforts to assess the effectiveness of abstinence-until-marriage
education programs. ASPE is currently sponsoring a study of the
Community-Based Program and a study of the State Program.^49 For the
former program, ASPE has contracted with Abt Associates to help design the
study, and an ASPE official told us that once the agency selects an
appropriate design, it will competitively award a contract to conduct the
study.^50 For the latter program, ASPE has contracted with Mathematica
Policy Research, Inc. (Mathematica), which is in the process of examining
the impact of five programs funded through the State Program on
participants' attitudes and behaviors related to abstinence and sex.^51 As
of August 2006, Mathematica has published two reports on findings from its
study--an interim report documenting the experiences of schools and
communities that receive abstinence-until-marriage education funding, and
a report on the first-year impacts of selected state
abstinence-until-marriage education programs.^52,53 Mathematica's final
report, which has not been completed, will examine the impact of the State
Program on behavioral outcomes, including abstinence, sexual activity,
risk of STDs, risk of pregnancy, and drug and alcohol use.^54 An ASPE
official told us that the agency expects a final report to be published in
2007.

^49According to the House Appropriations Committee report accompanying the
fiscal year 2005 appropriation act for Labor, HHS, Education, and related
agencies, the effectiveness of abstinence education programs should be
determined by measures that include the prevention and reduction of
adolescent pregnancies and STD infections, age at first sexual activity
and intercourse, frequency of sexual activity and intercourse, and numbers
who postpone sexual activity or intercourse through adolescence. See H.R.
Rep. No. 108-636, at 139-140 (2004).

^50According to ASPE officials, one factor that has contributed to delays
in the initiation of this study is the difficulty in recruiting schools to
participate.

^51The five abstinence-until-marriage education programs being studied are
My Choice, My Future! in Powhatan, Virginia; ReCapturing the Vision in
Miami, Florida; Teens in Control in Clarksdale, Mississippi; Families
United to Prevent Teen Pregnancy in Milwaukee, Wisconsin; and Heritage
Keepers in Edgefield, South Carolina.

^52See B. Devaney et al., The Evaluation of Abstinence Education Programs
Funded Under Title V Section 510: Interim Report, a report prepared for
ASPE, 2002.

^53See R. Maynard et al., First-Year Impacts of Four Title V, Section 510
Abstinence Education Programs, a report prepared for ASPE, 2005.
Mathematica's report on the first-year impacts of selected state
abstinence-until-marriage education programs focused on intermediate
outcomes, including attitudes about abstinence, teen sex, and marriage;
perceived consequences of teen and nonmarital sex; and expectations to
abstain from sexual intercourse.

Like ASPE, CDC has made its own effort to assess the effectiveness of
abstinence-until-marriage education. CDC is sponsoring a study to evaluate
the effectiveness of two middle school curricula--one that complies with
abstinence education program requirements and one that teaches a
combination of abstinence and contraceptive information and skills.^55 In
CDC's study, five middle schools chosen at random will receive a program
consisting of abstinence-until-marriage education exclusively; five
schools will receive comprehensive sex education, which also includes
information on contraception; and five schools will be assigned to a
control group. The study will examine the relative effectiveness of the
programs on behavioral outcomes such as reported sexual risk behaviors and
changes in attitudes related to abstinence and sex. CDC plans to recruit
approximately 1,500 seventh grade students into its study and will follow
them over a 2-year period. The agency expects to complete the study in
2009.

NIH has funded studies comparing the effectiveness of education programs
that focus only on abstinence with the effectiveness of sex education
programs that teach both abstinence and information about contraception.
As of August 2006, NIH is funding five studies, which in general are
comparing the effects of these two types of programs on the sexual
behavior and related attitudes among groups of either middle school or
high school students. For example, in one NIH study, researchers are using
groups of seventh and eighth grade adolescents to assess the impact of a
variety of programs on, among other issues, adolescents' reported sexual
activities, knowledge, and beliefs. For this study, researchers are
comparing these outcomes among students who received
abstinence-until-marriage education; students who received a combination
of abstinence and contraceptive education; and students who participated
in a general health class, who serve as a comparison group. NIH expects
both this study and its other four studies to be competed in 2006.

^54An impact evaluation assesses the net effect of a program by comparing
program outcomes with an estimate of what would have happened in the
absence of the program. This form of evaluation is employed when external
factors are known to influence the program's outcomes, in order to isolate
the program's contribution to achievement of its objectives.

^55HHS officials told us that the two curricula being tested are intended
to be comparable in length, intensity, and other characteristics.

  Other Researchers Have Also Made Efforts to Assess the Effectiveness of
  Abstinence-until-Marriage Education Programs

In addition to the efforts of researchers working on behalf of HHS and
states, other researchers--such as those affiliated with universities and
various advocacy groups--have made efforts to study the effectiveness of
abstinence-until-marriage education programs. This work includes studies
of the outcomes of individual programs and reviews of other studies on the
effectiveness of individual abstinence-until-marriage education programs.
In general, research studies on the effectiveness of individual
abstinence-until-marriage education programs have examined the extent to
which they changed participants' demonstrated knowledge, declared
intentions, and reported behavior related to sexual activity and
abstinence. For example, some studies examined the impact of
abstinence-until-marriage education programs on participants' knowledge of
concepts taught in the programs, as well as participants' declared
attitudes about abstinence and teen sex. Some studies examined the impact
of these programs on such outcomes as participants' declared commitment to
abstain from sex until marriage, participants' understanding of the
potential consequences of having intercourse, and participants' reported
ability to resist pressures to engage in sexual activity. Some of the
studies we reviewed examined the impact of abstinence-until-marriage
programs on participants' sexual behavior, as measured, for example, by
the proportion of participants who reported having had sexual intercourse
and the frequency of sexual intercourse reported by participants. In
general, the efforts to study and build a body of research on the
effectiveness of most abstinence education programs have been under way
for only a few years, in part because grants under the two programs that
account for the largest portion of federal spending on abstinence
education--the State Program and the Community-Based Program--were not
awarded until 1998 and 2001, respectively.

Several Factors Limit the Conclusions That Can Be Drawn about the Effectiveness
of Abstinence-until-Marriage Education Programs

Most of the efforts of HHS, states, and other researchers to evaluate the
effectiveness of abstinence-until-marriage education programs included in
our review have not met certain minimum criteria that experts have
concluded are necessary in order for assessments of program effectiveness
to be scientifically valid. For example, most of the efforts included in
our review did not include experimental or quasi-experimental designs, nor
did they measure behavioral or biological outcomes. In addition, the
results of some assessment efforts that meet the criteria of a
scientifically valid assessment have varied, and two key studies that meet
these criteria have not yet been completed.

  Experts Have Developed Criteria to Evaluate Efforts to Assess
  Abstinence-until-Marriage Education Programs

In an effort to better assess the merits of the studies that have been
conducted on the effectiveness of sexual health programs--including
abstinence-until-marriage education programs--scientific experts have
developed criteria that can be used to gauge the scientific rigor of these
evaluations. For example, in 2001, the National Campaign to Prevent Teen
Pregnancy--an organization focused on reducing teen pregnancy--published a
report by a panel of scientific experts that assessed the evidence
reported on abstinence-until-marriage education programs in peer-reviewed
journals and other literature.^56 The panel developed criteria that an
evaluation of a program's effectiveness must meet in order for the
program's results to be considered scientifically valid. In addition, in
2004, former U.S. Surgeon General David Satcher convened a panel of
experts to discuss, among other things, best practices for evaluating the
effectiveness of sexual health education programs--including
abstinence-until-marriage education programs.^57 This panel published a
report in 2006 that describes similar scientific criteria that assessments
of program effectiveness need to meet in order for their results to be
scientifically valid. Further, experts we interviewed agreed that these
criteria are important for ensuring that the results of a study support
valid conclusions. In general, these panels, as well as the experts we
interviewed, agreed that scientifically valid studies of a program's
effectiveness should include the following characteristics:

           o An experimental design that randomly assigns individuals or
           schools to either an intervention group or control group, or a
           quasi-experimental design that uses nonrandomly assigned but
           well-matched comparison groups. According to the panel of
           scientific experts convened by the National Campaign to Prevent
           Teen Pregnancy, experimental designs or quasi-experimental designs
           with well-matched comparison groups have at least three important
           strengths that are typically not found in other studies, such as
           those that use aggregated data: they evaluate specific programs
           with known characteristics, they can clearly distinguish between
           participants who did and did not receive an intervention, and they
           control for other factors that may affect study outcomes.
           Therefore, experimental and quasi-experimental study designs have
           a greater ability to assess the causal impact of specific programs
           than other types of studies.^58 According to scientific experts,
           studies that include experimental or quasi-experimental designs
           should also collect follow-up data for a minimum number of months
           after subjects receive an intervention.^59 Experts reported that
           follow-up periods are important in order to identify the effects
           of a program that are not immediately apparent or to determine
           whether these effects diminish over time. In addition, experts
           have reported that studies should have a sample size of at least
           100 individuals for study results to be considered scientifically
           valid.^60

           o Studies should assess or measure changes in biological outcomes
           or reported behaviors instead of attitudes or intentions.
           According to scientific experts, biological outcomes--such as
           pregnancy rates, birth rates, and STD rates--and reported
           behaviors--such as reported initiation and frequency of sexual
           activity--are better measures of the effectiveness of
           abstinence-until-marriage programs, because adolescent attitudes
           and intentions may or may not be indicative of actual behavior.
           For example, adolescents may report that they intend to abstain
           from sexual intercourse but may not actually do so.
			  
			  Most Efforts to Assess the Effectiveness of Abstinence-until-
			  Marriage Education Programs Have Not Used an Experimental or
			  Quasi-Experimental Design

           Many of the efforts by HHS, states, and other researchers that we
           identified in our review lack at least one of the characteristics
           of a scientifically valid study of program effectiveness. That is,
           most of the efforts to assess the effectiveness of these programs
           have not used experimental or quasi-experimental designs with
           sufficient follow-up periods and sample sizes to make their
           conclusions scientifically valid. For example, ACF--and before it,
           HRSA--used, according to ACF officials, grantee reporting on
           outcomes in order to monitor grantees' performance, target
           training and technical assistance, and help grantees improve
           service delivery. However, because the outcomes reported by
           grantees have not been produced through experimentally or
           quasi-experimentally designed studies, such information cannot be
           causally attributed to any particular abstinence-until-marriage
           education program. While ACF requires its fiscal year 2006
           grantees of the Community-Based Program to contract with
           third-party evaluators to select and monitor outcomes for their
           programs, ACF is not specifically requiring these grantees to use
           experimental or quasi-experimental designs. Therefore, it is not
           clear whether these evaluations will include such designs.
           Similarly, ACF's use of national data on adolescent behavior and
           birth rates to assess its State and Community-Based Programs is of
           limited value because these data do not distinguish between those
           who participated in abstinence-until-marriage education programs
           and those who did not. Consequently, these national data sets,
           which represent state-reported vital statistics and a nationwide
           survey of high school students, cannot be used to causally link
           declines in birth rates and adolescent sexual activity to the
           effects of specific abstinence-until-marriage education
           programs.^61

           Similarly, the efforts we identified by states and researchers to
           assess the effectiveness of abstinence-until-marriage education
           programs often did not include experimental or quasi-experimental
           designs. None of the state evaluations we reviewed that have been
           completed included randomly assigned control groups. For instance,
           one state evaluation that we reviewed only included students who
           volunteered to participate in the study. This evaluation report
           stated that the absence of a randomly assigned control group in
           the evaluation did not allow the evaluators to determine whether
           observed changes in participants' reported sexual behavior--as
           indicated through surveys administered at the beginning and end of
           a program--could be attributed to the abstinence-until-marriage
           education program.^62 Similarly, some of the journal articles that
           we reviewed described studies to assess the effectiveness of
           abstinence-until-marriage programs that did not include
           experimental or quasi-experimental designs needed to support
           scientifically valid conclusions about the programs'
           effectiveness. In these studies, researchers administered
           questionnaires to study participants before and after they
           completed an abstinence-until-marriage education program and
           assessed the extent to which the responses of participants
           changed.^63 These studies did not compare the responses of study
           participants with a group that did not participate in an
           abstinence-until-marriage education program. In addition, some of
           the studies used insufficient follow-up periods, thereby limiting
           the conclusions that can be drawn about the effectiveness of the
           abstinence-until-marriage education programs being studied. For
           example, two journal articles that we reviewed described studies
           that measured the effectiveness of abstinence-until-marriage
           programs in delaying the initiation of sexual activity from 1 to 2
           months after completion of the program.^64 Scientific experts
           consider this follow-up period too short to assess whether the
           programs had a valid effect.

           According to scientific experts, HHS, states, and other
           researchers face a number of challenges in designing experimental
           or quasi-experimental studies of program effectiveness. According
           to these experts, experimental or quasi-experimental studies can
           be expensive and time-consuming to carry out, and many grantees of
           abstinence-until-marriage education programs have insufficient
           time and funding to support these types of studies. Moreover, it
           can be difficult for researchers assessing
           abstinence-until-marriage education programs to convince school
           districts to participate in randomized intervention and control
           groups, in part because of sensitivities to surveying attitudes,
           intentions, and behaviors related to abstinence and sex. For
           example, in a third-party evaluation of its program, one grantee
           of the State Program originally planned to administer follow-up
           surveys 1 year after participants finished their abstinence
           education program, but the evaluators decided not to conduct this
           follow-up because of confidentiality concerns and the difficulty
           of locating students. In addition, the contractors hired to design
           ASPE's study of the effectiveness of the Community-Based Program
           have reported difficulties finding school districts that are
           willing to participate in randomly assigned intervention and
           control groups receiving either abstinence-until-marriage
           education or comprehensive sex education. An ASPE official told us
           that although a "randomized approach" is the best design for
           assessing the effectiveness of a program, the approach is also the
           most difficult to conduct.
			  
			  Most Efforts to Assess the Effectiveness of Abstinence-until-
			  Marriage Education Programs Have Not Measured Behavioral or
			  Biological Outcomes

           Another factor that limits the conclusions that can be drawn about
           the effectiveness of abstinence-until-marriage education programs
           is the fact that most efforts in our review to study the
           effectiveness of these programs did not measure changes in
           behavioral or biological outcomes among participants. Instead,
           most of the efforts we identify in our review used reported
           intentions and attitudes in order to assess the effectiveness of
           abstinence-until-marriage programs. For example, neither ACF's
           community-based grantees nor OPA's AFL grantees are required to
           report on behavioral or biological outcomes, such as rates of
           intercourse or pregnancy. Similarly, the journal articles we
           reviewed were more likely to use reported attitudes and
           intentions--such as study participants' reported attitudes about
           premarital sexual activity or their reported intentions to remain
           abstinent until marriage--rather than their reported behaviors or
           biological outcomes to assess the effectiveness of
           abstinence-until-marriage programs. For example, in one journal
           article we reviewed, participants were asked to rate the
           likelihood that they would have sexual intercourse as unmarried
           teenagers; another journal article described a study in which
           participants rated the likelihood that they would have sexual
           intercourse in the next year, before finishing high school, and
           before marriage.^65

           Experts, as well as state and HHS officials, have reported that it
           can be difficult to obtain scientifically valid information on
           biological outcomes and sexual behaviors. Specifically, experts
           have reported that when measuring an abstinence-until-marriage
           education program's affect on biological outcomes--such as
           reducing pregnancy or birth rates--it is necessary to have large
           sample sizes in order to determine whether a small change in
           biological outcomes is the result of the abstinence-until-marriage
           education program. In addition, state and federal officials told
           us that they have experienced difficulties obtaining information
           on sexual behaviors because of the sensitive nature of the
           information they were trying to collect. For example, one state
           official told us that her state's effort to evaluate
           abstinence-until-marriage education programs was only able to
           measure changes in participants' reported attitudes, instead of
           behaviors, because the evaluators needed to obtain consent from
           the parents of the program participants in order to ask them about
           their sexual behavior. The state official explained that the
           requirement to obtain consent from parents raised issues of
           self-selection, and therefore state officials decided to
           ultimately halt the study and only report on the attitudes that
           they had measured. In another example, ACF's fiscal year 2006
           budget justification reports that ACF has had some difficulty in
           obtaining reliable data from state grantees, in part because
           questions about teenage sexual behavior are sensitive. OPA
           officials also acknowledged that many communities will not allow
           grantees to ask program participants questions about their sexual
           behavior because the communities believe such questions are too
           intrusive. One OPA official said that such restrictions affect the
           agency's ability to measure behavioral outcomes, explaining that
           OPA cannot measure what it cannot ask about.
			  
			  Results of Some Scientifically Valid Assessment Efforts Have
			  Varied, and Other Key Studies Have Not Been Completed

           Among the assessment efforts we identified are some studies that
           meet the criteria of a scientifically valid effectiveness study.
           However, results of these studies have varied, and this limits the
           conclusions that can be drawn about the effectiveness of
           abstinence-until-marriage education programs. Some researchers
           have reported that abstinence-until-marriage education programs
           have resulted in adolescents reporting having less frequent sexual
           intercourse or fewer sexual partners.^66 For example, in one study
           of middle school students, participants in an
           abstinence-until-marriage education program who had sexual
           intercourse during the follow-up period were 50 percent less
           likely to report having two or more sexual partners when compared
           with their nonparticipant peers.^67 In contrast, other studies
           have reported that abstinence-until-marriage education programs
           did not affect the reported frequency of sexual intercourse or
           number of sexual partners.^68 For example, one study of middle
           school students found that participants of an
           abstinence-until-marriage program were not less likely than
           nonparticipants at the 1 year follow-up to report less frequent
           sexual intercourse or fewer sexual partners.^69 In addition to
           these varied findings, one study found that an
           abstinence-until-marriage program was effective in delaying the
           initiation of sexual intercourse in the short term but not long
           term.^70 Experts with whom we spoke emphasized that there are
           still too few scientifically valid studies completed to date that
           can be used to determine conclusively which, if any,
           abstinence-until-marriage programs are effective.

           Additionally, among the assessment efforts we identified are some
           studies that experts anticipate will meet the criteria of a
           scientifically valid effectiveness study but are not yet
           completed. One of these key studies is the final Mathematica
           report, contracted by ASPE, on the State Program.^71 The final
           report was originally slated for publication in 2005, but an ASPE
           official stated that the final report has been delayed until 2007
           so that researchers can extend the follow-up period to improve
           their response rate and the reliability of the information they
           collect. Another key study is CDC's research on middle school
           programs, which is not expected to be completed until 2009.

           Experts and federal officials we interviewed stated that they
           expect the results of these two federally funded studies to add
           substantively to the body of research on the effectiveness of
           abstinence-until-marriage education programs. One expert with whom
           we spoke said that she expects the final Mathematica report on
           participants' behaviors to provide the groundwork for the field.
           Another expert we interviewed stated that the CDC study was very
           well-designed and she expects the results to contribute to the
           development of effective abstinence-until-marriage education
           curricula.
			  
			  Conclusions

           There have been various efforts--by HHS, states, and others--to
           assess the scientific accuracy of educational materials used in
           abstinence-until-marriage education programs and the effectiveness
           of these programs. However, efforts to evaluate both the accuracy
           and effectiveness of abstinence-until-marriage education programs
           have been, in various ways, limited. ACF, which administers the
           two programs that account for the largest portion of federal
           spending on abstinence-until-marriage education, does not review
           or require its grantees to review program materials for scientific
           accuracy. In addition, not all grantees of the State Program have
           chosen to review their materials. Because of these limitations,
           ACF cannot be assured that the materials used in its State and
           Community-Based Programs are accurate. Moreover, OPA, which
           reviews all grantees' proposed abstinence-until-marriage
           educational materials, and states that review educational
           materials have found inaccuracies in some educational materials
           used by abstinence-until-marriage programs.

           Similarly, most of the efforts described in our review to assess
           the effectiveness of abstinence-until-marriage programs have not
           met minimum scientific criteria needed to draw valid conclusions
           about their effectiveness. Specifically, most efforts by agencies,
           states, and other researchers have not included experimental or
           quasi-experimental designs that can establish whether changes in
           behaviors or biological outcomes can be causally linked to
           specific abstinence-until-marriage education programs. While these
           types of studies are time-consuming and expensive, experts said
           that they are the only definitive way to draw valid conclusions
           about the effectiveness of these programs. In addition, among the
           assessment efforts we identified are some studies funded by HHS
           that experts anticipate will meet the criteria of a scientifically
           valid effectiveness study but are not yet completed. When
           completed, these HHS-funded studies may add substantively to the
           body of research on the effectiveness of abstinence-until-marriage
           education programs.
			  
			  Recommendation for Executive Action

           To address concerns about the scientific accuracy of materials
           used in abstinence-until-marriage education programs, we recommend
           that the Secretary of HHS develop procedures to help assure the
           accuracy of such materials used in the State and Community-Based
           Programs. To help provide such assurances, the Secretary could
           consider alternatives such as (1) extending the approach currently
           used by OPA to review the scientific accuracy of the factual
           statements included in abstinence-until-marriage education to
           materials used by grantees of ACF's Community-Based Program and
           requiring grantees of ACF's State Program to conduct such reviews
           or (2) requiring grantees of both programs to sign written
           assurances in their grant applications that the materials they
           propose using are accurate.
			  
			  Agency Comments and Our Evaluation

           HHS provided written comments on a draft of this report. (See app.
           III.)

           In its written comments, HHS stated that it will consider
           requiring grantees of both ACF programs to sign written assurances
           in grant applications that the materials they use are accurate.
           Regarding accuracy, HHS's written comments also noted that all
           applicants for federal assistance attest on the application
           form--Standard Form 424--that all data in their applications are
           "true and correct," and that in the view of HHS, this applies to
           information presented in curricula funded by federal grants.
           However, as we stated in the draft report, grantees of the State
           Program are not required to submit curricula as a part of their
           applications; therefore, the attestation in Standard Form 424
           would not apply to curricula used by those grantees. In addition,
           as stated in the draft report, some states have reviewed materials
           used in abstinence-until-marriage education programs, but these
           reviews occurred after they received funding from ACF. Further,
           while grantees of the Community-Based Program were required to
           submit copies of their curricula and a Standard Form 424 in fiscal
           year 2006 as part of their applications, none of the materials
           specifically require an assurance of scientific accuracy. Further,
           OPA and states have found inaccuracies in educational materials
           used by abstinence-until-marriage programs.

           HHS's written comments also stated that ACF requires that
           curricula conform to HHS's standards grounded in scientific
           literature. HHS's comments refer to the curriculum standards for
           this program that detail what types of information must be
           included in abstinence-until-marriage curricula, and the comments
           stated that the curricula must provide supporting references for
           this information. Further, HHS's comments stated that ACF staff
           review the curricula to ensure compliance with these standards.
           The draft report stated this. However, a requirement that
           curricula include certain types of information does not
           necessarily ensure the accuracy of the scientific facts included
           in the abstinence-until-marriage materials. For example, while
           education materials may include information on failure rates
           associated with contraceptives or STD infections, this information
           may be outdated or otherwise inaccurate or incomplete. HHS's
           written comments also stated that if it finds inaccurate
           statements during the review process or at any time during the
           grant period, ACF works with grantees to take corrective action.
           To ensure completeness, we have added this statement to the
           report. Further, HHS stated that 2 inaccuracies cited in the draft
           report had been corrected before our work began. We believe HHS is
           referring to inaccuracies identified by OPA during its review of
           materials for scientific accuracy and this reinforces the need for
           review of materials used by ACF's grantees.

           As HHS noted in its written comments, we did not define the term
           scientific accuracy. HHS stated that it disagreed with certain
           findings of the report because it was difficult to precisely
           determine the criteria we employed in making the recommendation as
           to scientific accuracy. As we stated in the scope and methodology
           section of the draft report, the objective of our work was to
           focus on efforts by HHS and states to review the accuracy of
           scientific facts included in abstinence-until-marriage education
           materials. Performing an independent assessment of the criteria
           used by these entities to determine the scientific accuracy of
           education materials or the quality of the reviews was beyond the
           scope of the work.

           Regarding effectiveness, HHS's written comments also described a
           number of actions it is taking to determine program effectiveness
           and improve the quality of programs and research. Specifically,
           HHS's comments described (1) studies undertaken or funded by ASPE,
           CDC, and NIH; (2) technical assistance provided by OPA and ACF;
           (3) grantee evaluation requirements; and (4) ACF and OPA
           requirements for the amount of grant funds to be spent on
           evaluations. All of this information was included in our draft
           report. HHS's comments also described a new effort funded by ACF
           and ASPE that is designed to build capacity for quality research
           in the field of abstinence education. We added information on this
           effort to the report. HHS's written comments also describe
           evaluations that resulted from an Abstinence Education Evaluation
           Conference sponsored by ACF and OPA. While this conference was
           described in the draft report, we added more detail regarding the
           content of the conference. HHS's written comments also describe
           OPA's efforts to assess the effectiveness of the AFL Program. We
           had included this information in the draft report.

           HHS's written comments stated that it may be too soon to draw
           conclusions about the effectiveness of ACF's and OPA's programs,
           in part, because key studies have not been completed. We agree and
           discussed this in the draft report. As we noted in the draft
           report, key studies funded by HHS that experts anticipate will
           meet the criteria of a scientifically valid effectiveness study
           are not yet completed, but when completed these HHS funded studies
           may add substantively to the body of research on the effectiveness
           of abstinence-until-marriage education programs.

           In addition, the comments stated that having an inadequate amount
           of scientifically valid and conclusive evaluation studies is not
           unique to abstinence-until-marriage education programs, and a
           recent ASPE review of comprehensive sex education programs found
           mixed results on their effectiveness. However, the scope of our
           report was focused on abstinence-until-marriage education
           programs, and we did not review comprehensive sex education
           programs or make any comparisons between the two types of
           programs.

           HHS also provided technical comments, which we incorporated into
           the report as appropriate.

           As agreed with your office, unless you publicly announce its
           contents earlier, we plan no further distribution of this report
           until 30 days after its date. We will then send copies of this
           report to the Secretary of HHS and to other interested parties. In
           addition, this report is available at no charge on GAO's Web site
           at http://www.gao.gov. We will also make copies available to
           others upon request.

           If you or your staff have any questions about this report, please
           call me at (202) 512-3407 or [email protected]. Contact points for
           our Offices of Congressional Relations and Public Affairs may be
           found on the last page of this report. GAO staff who made major
           contributions to this report are listed in appendix IV.

           Marcia Crosse
			  Director, Health Care

           List of Requesters

           The Honorable Henry A. Waxman
			  Ranking Minority Member Committee on Government Reform
			  House of Representatives

           The Honorable Pete Stark
			  Ranking Minority Member
			  Subcommittee on
           Health Committee on Ways and Means
			  House of Representatives

           The Honorable Sherrod Brown
			  Ranking Minority Member Subcommittee on Health Committee on Energy and Commerce
			  House of Representatives

           The Honorable Barbara Boxer
			  The Honorable Maria Cantwell
			  The
           Honorable Richard J. Durbin
			  The Honorable Russell D. Feingold
			  The Honorable Dianne Feinstein
			  The Honorable Tom Harkin
			  The Honorable James M. Jeffords
			  The Honorable Edward M. Kennedy
			  The Honorable Frank R. Lautenberg
			  The Honorable Patrick Leahy
			  The Honorable Patty Murray
			  
			  United States Senate
           The Honorable Howard L. Berman
			  The Honorable Lois Capps
			  The Honorable Jay Inslee
			  The Honorable Barbara Lee
			  The Honorable Nita M. Lowey
			  The Honorable Betty McCollum
			  House of Representatives
			  
			  Appendix I: HRSAï¿½s Technical Assistance Contract for
			  Abstinence Education

           The Health Resources and Services Administration (HRSA) awarded a
           contract to the National Abstinence Clearinghouse (NAC) in 2002 to
           provide assistance with its Community-Based Abstinence Education
           Program (Community-Based Program) and Abstinence Education Program
           (State Program).^1 NAC is a nonprofit educational organization
           whose mission is to promote the appreciation for and practice of
           sexual abstinence until marriage through the distribution of age
           appropriate, factual, and medically referenced materials. The
           purpose of the contract was (1) to develop national criteria for
           the review of abstinence-until-marriage educational materials and
           to create a directory of approved materials; (2) to provide
           medical accuracy training to grantees; and (3) to provide
           technical support to grantees, such as assistance with program
           evaluation.^2

           We are reporting on the steps that HRSA took to award the contract
           to NAC in response to concerns that have been raised by a
           congressional requester. In general, these concerns centered on
           the extent to which the selection process was competitive and
           whether HRSA identified the potential for an organizational
           conflict of interest.

           HRSA awarded the contract to address three concerns it had with
           the Community-Based Program during 2001, the first year of its
           implementation. First, HRSA officials needed guidance to determine
           whether abstinence-until-marriage education materials conformed to
           the definitional requirements of the Social Security Act.^3
           Second, many grantees lacked the medical background and training
           to ensure that they would provide medically accurate,
           science-based information in their programs. Third, grantees also
           lacked experience with the technical management of federal grants,
           including how to conduct evaluations of their programs.

           HRSA used full and open competition procedures to award the
           contract to NAC. In doing so, HRSA (1) publicly solicited
           proposals from potential contractors; (2) conducted technical
           evaluations of both the original proposals and the revised
           proposals for those considered to be in the competitive range; and
           (3) determined that NAC's proposal represented the best overall
           value to the government. This process, which took place from May
           2002 through September 2002, resulted in HRSA awarding NAC the
           contract with a potential value of nearly $2.7 million.
			  
			  HRSA Used Full and Open Competition Procedures to Award the
			  Contract to NAC

           HRSA issued a notice on May 20, 2002, on the FedBizOpps Web site,
           the government point of entry for notifying potential contractors
           of federal contract opportunities, indicating its intent to
           publicly request proposals from prospective contractors in June
           2002.^4 On June 20, 2002, HRSA posted the solicitation on the
           FedBizOpps Web site indicating that the abstinence contract would
           be awarded using full and open competition procedures, that is,
           all responsible prospective contractors would be provided the
           opportunity to compete.^5 The solicitation, which was a Request
           for Proposals (RFP), described the contract objectives, which
           included (1) the development of national criteria for the review
           of abstinence-until-marriage educational materials and the
           development of a directory of approved materials; (2) the
           provision of medical accuracy training to grantees; and (3) the
           provision of technical support to grantees, such as assistance
           with program evaluation. The RFP stated that HRSA intended to
           award a cost-reimbursement contract with fixed fee for a 1-year
           base period and 2 option years.^6 This was a best value
           procurement; that is, HRSA reserved the right in the RFP to select
           for award the proposal that HRSA determined offered the best value
           to the government, even if it did not offer the lowest cost.
           Further, the RFP stated that the technical evaluation of the
           prospective contractors' proposals would receive paramount
           consideration in the selection of the contractor. According to the
           RFP, this evaluation would include an assessment of the
           prospective contractor's technical approach, the organizational
           experience and expertise of the prospective contractor, the plans
           for personnel and management of the work, and the prospective
           contractor's statement and understanding of the project purpose.
           Other factors, such as the estimated cost, past performance under
           other contracts for similar services, and the subcontracting plan
           would also be considered in the selection process. Five
           prospective contractors submitted proposals to HRSA by July 31,
           2002, when proposals were due.

           HRSA established a review committee to conduct the technical
           evaluation of the five proposals. This committee included three
           voting members and a nonvoting chairperson. The Director of HRSA's
           Community-Based and State Programs and two analysts from other
           programs within the Department of Health and Human Services (HHS)
           served as the voting members, and the chairperson of the review
           committee was a project officer of HRSA's Community-Based Program.
           The committee members conducted the technical evaluation of the
           proposals, according to the criteria in the RFP, as described
           above. Three proposals with the highest technical scores were
           determined to be in the competitive range,^7 with NAC's proposal
           receiving the highest technical score. HRSA requested in writing
           that the competitive range offerors address certain technical and
           cost issues and submit revised proposals to HRSA by September 17,
           2002.^8 For example, HRSA requested that one of the prospective
           contractors other than NAC clearly describe its proposed
           management of day-to-day tasks of the contract and provide
           justification for several labor and travel expenditures. HRSA did
           not have oral discussions with the competitive range offerors.
           HRSA's review committee evaluated the revised proposals and again
           gave NAC's revised proposal the highest technical score.

           Although NAC's estimated cost was not the lowest among the
           proposals in the competitive range, HRSA determined that NAC had
           proposed a realistic cost estimate for the contract. Accordingly,
           and in light of the NAC proposal's high technical rating and the
           RFP's evaluation criteria giving paramount consideration to the
           technical evaluation, HRSA determined that NAC's proposal
           represented the best value to the government. HRSA awarded a
           contract to NAC on September 27, 2002. The contract had a 1-year
           base period of performance with an estimated value of $854,681,
           and included 2 option years for a total potential value of
           $2,673,784. According to a HRSA official, this cost-reimbursement
           contract did not include a fee. All of the prospective contractors
           were made aware that a debriefing to explain the selection
           decision and contract award would be provided at their request.
           One prospective contractor requested and received a debriefing
           from HRSA. No protests were filed with the agency challenging the
           award of the contract to NAC. There were no bid protests filed
           with GAO.^9
			  
			  HRSA Identified No Actual or Potential Organizational Conflicts
			  of Interest

           HRSA officials told us that they did not identify any actual or
           potential organizational conflicts of interest during the
           acquisition process. As defined in the Federal Acquisition
           Regulation (FAR), an organizational conflict of interest arises
           where

           o because of other activities or relationships, a person is unable
           or potentially unable to provide impartial assistance or advice to
           the government; or

           o the person's objectivity in performing the contract work is or
           might be otherwise impaired; or

           o a person has an unfair competitive advantage.^10

           An organizational conflict of interest may result when factors
           create an actual or potential conflict of interest during
           performance of a contract, or when the nature of the work to be
           performed under one contract creates an actual or potential
           conflict of interest involving a future acquisition.^11 Under the
           FAR, contracting officers are required to analyze planned
           acquisitions to identify and evaluate potential organizational
           conflicts of interest as early in the acquisition process as
           possible, and to take steps to avoid, neutralize, or mitigate
           significant potential conflicts of interest before a contract is
           awarded.^12

           According to HRSA's contracting officer, HRSA did not identify any
           actual or potential organizational conflicts of interest. In
           reaching this conclusion, the contracting officer told us that he
           reviewed the statement of work, including the background and
           objectives of the proposed contract, the stated purpose of the
           contact, the criteria established to evaluate the proposals, the
           past performance of the competitors, and NAC's proposal. HRSA's
           contracting officer also told us that he did not formally document
           his assessment of organizational conflict of interest.^13
			  
			  Appendix II: Methodology for Identifying and Reviewing Research
			  Studies

           To identify research studies that examine the effectiveness of
           abstinence-until-marriage education programs among adolescents and
           young adults, we searched two reference database systems, PubMed
           and ProQuest. We used the following keywords to search for
           research studies that were published from January 1, 1998, through
           May 22, 2006: "virginity," "abstinence education," "abstinence and
           curriculum," "abstinence only," "teen pregnancy and prevention,"
           and "abstinence until marriage."^1 We reviewed the research
           article titles that were generated from the PubMed and ProQuest
           searches and identified articles that appeared to focus on the
           evaluation of the effectiveness of abstinence-until-marriage
           education programs. In cases where we could not determine, based
           on the title, whether a study appeared to focus on an
           abstinence-until-marriage education program evaluation, we
           reviewed a summary of the article to obtain more information about
           the research study. We also examined previous summaries of the
           literature to identify additional research studies.

           We then selected research studies for inclusion in our literature
           review if they met three criteria. First, the study evaluated a
           group-based, abstinence-until-marriage education program. We did
           not select studies that evaluated one-on-one interactions, such as
           education programs focused exclusively on parent-child
           interactions, or that evaluated media campaigns. We reviewed the
           description of each education program and curriculum, as described
           in the study, to determine whether an abstinence-until-marriage
           education program was being evaluated. Education programs that
           were described as including detailed contraceptive information in
           their curricula, for example, were not classified as
           abstinence-until-marriage programs. Second, the study targeted
           adolescents and young adults in the United States, for example, by
           indicating that participants in the evaluation were high school or
           middle school students. Third, the study was a quantitative rather
           than a qualitative evaluation of an abstinence-until-marriage
           education program. We selected 13 research studies for inclusion
           in our literature review.

           We reviewed the selected research studies to obtain detailed
           information about their methodologies and outcome variables. For
           example, we determined whether each study used an experimental or
           quasi-experimental design and whether the outcome measures
           included attitudes, behavioral intentions, behaviors such as
           initiation of sexual intercourse, or a combination of these.
			  
			  Appendix III: Comments from the Department of Health and Human
			  Services
			  
			  Appendix IV: GAO Contact and Staff Acknowledgments
			  
			  GAO Contact

           Marcia Crosse, (202) 512-7119 or [email protected]
			  
			  Acknowledgments

           In addition to the contact named above, Kristi Peterson, Assistant
           Director; Kelly DeMots; Pam Dooley; Krister Friday; Julian
           Klazkin; and Amy Shefrin made key contributions to this report.
			  
			  GAOï¿½s Mission

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.
			  
			  Obtaining Copies of GAO Reports and Testimony

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each
           weekday, GAO posts newly released reports, testimony, and
           correspondence on its Web site. To have GAO e-mail you a list of
           newly posted products every afternoon, go to www.gao.gov and
           select "Subscribe to Updates."
			  
			  Order by Mail or Phone

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061
			  
			  To Report Fraud, Waste, and Abuse in Federal Programs

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470
			  
			  Congressional Relations

           Gloria Jarmon, Managing Director, [email protected] (202)
           512-4400 U.S. Government Accountability Office, 441 G Street NW,
           Room 7125 Washington, D.C. 20548
			  
			  Public Affairs

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

^56See Kirby. This panel included experts from NIH, the Medical Institute
for Sexual Health, the Alan Guttmacher Institute, the Institute for
Research and Evaluation, and various universities.

^57See Satcher. This panel included experts from a variety of
organizations, including the Medical Institute for Sexual Health, the Alan
Guttmacher Institute, and the American Academy of Pediatrics.

  Most Efforts to Assess the Effectiveness of Abstinence-until-Marriage
  Education Programs Have Not Used an Experimental or Quasi-Experimental Design

^58For example, experts have reported that the use of randomly assigned
intervention and control groups is particularly important when assessing
the effectiveness of abstinence-until-marriage programs because
adolescents who voluntarily participate in such programs may be
self-selecting--that is, they may be more willing to accept the principles
of--and be influenced by--such programs when compared with other
adolescents. Mathematica's interim report on the evaluation of the State
Program noted that selection bias can "seriously undermine the
credibility" of study results.

^59For example, one expert reported that studies assessing program
effectiveness should obtain information on participants for at least 3
months after the conclusion of a program when they are measuring behaviors
that can change quickly, such as frequency of sex. For behaviors or
outcomes that change less quickly, such as initiation of sex or pregnancy
rates, information on participants should be collected for at least 6
months after the conclusion of a program.

^60The panel of experts convened by the National Campaign to Prevent Teen
Pregnancy agreed that large sample sizes are necessary to determine the
magnitude of any discernable program effect and to ensure that results of
any study of effectiveness are statistically valid.

^61In addition, according to ACF and CDC officials, it is difficult to
draw conclusions from national data sets about the effectiveness of
abstinence-until-marriage education programs because the national survey
questions used to produce these data often do not identify the specific
type of program or intervention survey respondents may have participated
in or received.

^62See Goodson et al.

^63See, for example, S. M. Fitzgerald et al., "Effectiveness of the
Responsible Social Values Program for 6th Grade Students in One Rural
School District," Psychological Reports, vol. 91 (2002), and J. E. Barnett
and C. S. Hurst, "Abstinence Education for Rural Youth: An Evaluation of
the Life's Walk Program, " The Journal of School Health, vol. 73, no. 7
(2003).

^64See, for example, D. A. Zanis, "Use of a Sexual Abstinence Only
Curriculum with Sexually Active Youths," Children & Schools, vol. 27, no.
1 (2005), and G. Denny et al., "An Evaluation of An Abstinence Education
Curriculum Series: Sex Can Wait," American Journal of Health Behavior,
vol. 26, no. 5 (2002).

  Most Efforts to Assess the Effectiveness of Abstinence-until-Marriage
  Education Programs Have Not Measured Behavioral or Biological Outcomes

^65See, for example, L. Sather and K. Zinn, "Effects of Abstinence-Only
Education on Adolescent Attitudes and Values Concerning Premarital Sexual
Intercourse, " Family and Community Health, vol. 25, no. 2 (2002), and G.
Denny, M. Young, and C. E. Spear, "An Evaluation of the Sex Can Wait
Abstinence Education Curriculum Series," American Journal of Health
Behavior, vol. 23, no. 2 (1999).

  Results of Some Scientifically Valid Assessment Efforts Have Varied, and Other
  Key Studies Have Not Been Completed

^66See Borawski et al. See also T. L. St. Pierre et al., "A 27-Month
Evaluation of a Sexual Activity Prevention Program in Boys & Girls Clubs
Across the Nation," Family Relations, vol. 44, no. 1 (1995).

^67See Borawski et al.

^68See N.G. Harrington et al., "Evaluation of the All Stars Character
Education and Problem Behavior Prevention Program: Effects on Mediator and
Outcome Variables for Middle School Students," Health Education &
Behavior, vol. 28, no. 5 (2001). See also Jemmott, Jemmott, and Fong.

^69See Harrington et al.

^70See Jemmott, Jemmott, and Fong and J. B. Jemmott III, L. S. Jemmott,
and G. T. Fong, reply to letter to editor, Journal of the American Medical
Association, vol. 281, no. 16 (1999), 1487. This study found that an
abstinence-until-marriage program delayed the initiation of sexual
intercourse at the 3-month follow-up period but not at 6 and 12 months.

^71According to several scientific experts, Mathematica's study is an
important one, in part because of its sound design: the study randomly
assigns and compares control groups with groups receiving
abstinence-until-marriage education and uses surveys to follow up with
program participants for several months after their completion of a
program.

Conclusions

Recommendation for Executive Action

Agency Comments and Our Evaluation

Appendix I: HRSA's Technical Assistance Contract for Abstinence Education
Appendix I: HRSA's Technical Assistance Contract for Abstinence Education

^1The administration of this contract was transferred from HRSA to the
Administration of Children and Families (ACF) in May 2005.

^2The contract resulted in the development of criteria for reviewing
abstinence-until-marriage educational materials, and ACF included these
criteria in the fiscal year 2006 program announcement for the
Community-Based Program. According to the announcement, ACF will evaluate
grant applicants' proposed educational materials to ensure compliance with
the criteria. Medical accuracy training and technical support were
provided to grantees as a result of the contract. According to an ACF
official, a directory of approved abstinence-until-marriage educational
materials was not completed.

HRSA Used Full and Open Competition Procedures to Award the Contract to NAC

^3Section 510(b)(2) of the Social Security Act defines abstinence
education as an educational or motivational program that: A. has as its
exclusive purpose, teaching the social, psychological, and health gains to
be realized by abstaining from sexual activity; B. teaches abstinence from
sexual activity outside marriage as the expected standard for all school
age children; C. teaches that abstinence from sexual activity is the only
certain way to avoid out-of-wedlock pregnancy, sexually transmitted
diseases, and other associated health problems; D. teaches that a mutually
faithful monogamous relationship in context of marriage is the expected
standard of human sexual activity; E. teaches that sexual activity outside
of the context of marriage is likely to have harmful psychological and
physical effects; F. teaches that bearing children out-of-wedlock is
likely to have harmful consequences for the child, the child's parents,
and society; G. teaches young people how to reject sexual advances and how
alcohol and drug use increases vulnerability to sexual advances; and H.
teaches the importance of attaining self-sufficiency before engaging in
sexual activity.

^4The current FedBizOpps Web site address is [48]http://www.fbo.gov /. 
Prior to October 1, 2005, the Web site address was
[49]http://www.eps.gov/. The solicitation number was 240-MCHB-012(02)-abg.

^5The Federal Acquisition Regulation (FAR) requires the contracting
officer to ensure that prospective contractors are responsible. FAR S
9.103. A responsible source refers to a prospective contractor that has,
among other things, adequate financial resources, the necessary experience
and technical skills to perform the work of the contract, a satisfactory
performance record, and the ability to meet the delivery schedule. FAR S
9.104-1.

^6Cost-reimbursement contracts are used only when uncertainties involved
in contract performance do not permit costs to be estimated with
sufficient accuracy to use any type of fixed-price contract. FAR S
16.301-2.

HRSA Identified No Actual or Potential Organizational Conflicts of Interest

^7Based on the ratings of each proposal against all evaluation criteria,
the contracting officer establishes a competitive range consisting of all
of the most highly rated proposals. FAR S 15.306(c)(1).

^8HRSA officials, including an auditor, reviewed the cost proposals in the
competitive range.

^9GAO's Office of General Counsel resolves disputes concerning awards of
federal contracts, which are known as bid protests.

^10FAR S 2.101.

^11FAR S 9.502(c).

^12The FAR requires contracting officers to exercise common sense, good
judgment, and sound discretion in determining whether a significant
potential conflict of interest exists. FAR S 9.505.

^13The FAR requires HRSA's contracting officers to formally document their
assessment only when a substantive issue concerning a potential
organizational conflict of interest exists. FAR S 9.504(d). HHS
acquisition regulations do not explicitly address the assessment of
organizational conflict of interest. Therefore, FAR subpart 9.5 is the
controlling regulation when HHS encounters an issue related to an
organizational conflict of interest.

Appendix II: Methodology for Identifying and Reviewing Research Studies
Appendix II: Methodology for Identifying and Reviewing Research Studies

^1ACF first awarded grants under the State Program in fiscal year 1998.

Appendix III: Comments from the Department of Health and Human Services
Appendix III: Comments from the Department of Health and Human Services

A Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

(290491)

GAO's Mission

Obtaining Copies of GAO Reports and Testimony

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Congressional Relations

Public Affairs

www.gao.gov/cgi-bin/getrpt? [50]GAO-07-87 .

To view the full product, including the scope

and methodology, click on the link above.

For more information, contact Marcia Crosse at (202) 512-7119 or
[email protected].

Highlights of [51]GAO-07-87 , a report to congressional requesters

October 2006

ABSTINENCE EDUCATION

Efforts to Assess the Accuracy and Effectiveness of Federally Funded
Programs

Efforts by HHS and states to assess the scientific accuracy of materials
used in abstinence-until-marriage education programs have been limited.
This is because HHS's ACF--which awards grants to two programs that
account for the largest portion of federal spending on
abstinence-until-marriage education--does not review its grantees'
education materials for scientific accuracy and does not require grantees
of either program to review their own materials for scientific accuracy.
In contrast, OPA does review the scientific accuracy of grantees' proposed
educational materials. In addition, not all states that receive funding
from ACF have chosen to review their program materials for scientific
accuracy. In particular, 5 of the 10 states that GAO contacted conduct
such reviews. Officials from these states reported using a variety of
approaches in their reviews. While the extent to which federally funded
abstinence-until-marriage education materials are inaccurate is not known,
in the course of their reviews OPA and some states reported that they have
found inaccuracies in abstinence-until-marriage education materials. For
example, one state official described an instance in which
abstinence-until-marriage materials incorrectly suggested that HIV can
pass through condoms because the latex used in condoms is porous.

HHS, states, and researchers have made a variety of efforts to assess the
effectiveness of abstinence-until-marriage education programs; however, a
number of factors limit the conclusions that can be drawn about the
effectiveness of abstinence-until-marriage education programs. ACF and OPA
have required their grantees to report on various outcomes that the
agencies use to measure the effectiveness of grantees'
abstinence-until-marriage education programs. In addition, 6 of the 10
states in GAO's review have worked with third-party evaluators to assess
the effectiveness of abstinence-until-marriage education programs in their
states. Several factors, however, limit the conclusions that can be drawn
about the effectiveness of abstinence-until-marriage education programs.
Most of the efforts to evaluate the effectiveness of
abstinence-until-marriage education programs included in GAO's review have
not met certain minimum scientific criteria--such as random assignment of
participants and sufficient follow-up periods and sample sizes--that
experts have concluded are necessary in order for assessments of program
effectiveness to be scientifically valid, in part because such designs can
be expensive and time-consuming to carry out. In addition, the results of
efforts that meet the criteria of a scientifically valid assessment have
varied and two key studies funded by HHS that meet these criteria have not
yet been completed. When completed, these HHS-funded studies may add
substantively to the body of research on the effectiveness of
abstinence-until-marriage education programs.

Reducing the incidence of sexually transmitted diseases and unintended
pregnancies is one objective of the Department of Health and Human
Services (HHS). HHS provides funding to states and organizations that
provide abstinence-until-marriage education as one approach to address
this objective.

GAO was asked to describe the oversight of federally funded
abstinence-until-marriage education programs. GAO is reporting on (1)
efforts by HHS and states to assess the scientific accuracy of materials
used in these programs and (2) efforts by HHS, states, and researchers to
assess the effectiveness of these programs. GAO reviewed documents and
interviewed HHS officials in the Administration for Children and Families
(ACF) and the Office of Population Affairs (OPA) that award grants for
these programs.

[52]What GAO Recommends

To address concerns about the scientific accuracy of materials used in
abstinence-until-marriage education programs, GAO recommends that the
Secretary of HHS develop procedures to help assure the accuracy of such
materials used in programs administered by ACF. HHS agreed to consider
this recommendation. HHS also provided information on steps it takes to
assure accuracy, which we have incorporated into the report, as
appropriate.

References

Visible links

  41. http://www.gao.gov/cgi-bin/getrpt?GAO-05-739SP
  48. http://www.fbo.gov /
  49. http://www.eps.gov/
  50. http://www.gao.gov/cgi-bin/getrpt?GAO-07-87
  51. http://www.gao.gov/cgi-bin/getrpt?GAO-07-87
*** End of document. ***