Youthbuild Program: Analysis of Outcome Data Needed to Determine 
Long-Term Benefits (28-FEB-07, GAO-07-82).			 
                                                                 
Since 1993, the Department of Housing and Urban Development (HUD)
has provided funding for Youthbuild, a competitive grant program 
that trains and educates disadvantaged youth and helps build	 
low-income housing. In 2006, Youthbuild was transferred to the	 
Department of Labor (Labor) to better align the Youthbuild	 
program with existing youth workforce and training programs. In  
response to concerns about the overall quality of Youthbuild, a  
Senate report directed GAO to assess the program. GAO's 	 
objectives included (1) evaluating how HUD assessed and oversaw  
the program, (2) determining what results the program achieved,  
and (3) assessing how successful grantees were in obtaining	 
outside funding. GAO analyzed Youthbuild performance data,	 
visited Youthbuild sites, and interviewed agency officials.	 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-82						        
    ACCNO:   A66359						        
  TITLE:     Youthbuild Program: Analysis of Outcome Data Needed to   
Determine Long-Term Benefits					 
     DATE:   02/28/2007 
  SUBJECT:   Data collection					 
	     Disadvantaged persons				 
	     Federal grants					 
	     Government grants					 
	     Government information				 
	     Low income housing 				 
	     Performance measures				 
	     Program evaluation 				 
	     Reporting requirements				 
	     Youth employment programs				 
	     Executive agency oversight 			 
	     HUD Youthbuild Program				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-82

   

     * [1]Results in Brief
     * [2]Background
     * [3]HUD's Grant-Making Procedures Incorporated Legislative Crite
     * [4]HUD Oversight of Grantees' Performance Has Been Limited

          * [5]HUD Has Not Collected or Analyzed Data That Could Help It As
          * [6]Citing Resource Constraints, HUD Performed Limited Oversight

     * [7]Incomplete Data on Participants' Outcomes Precludes an Overa

          * [8]A Few Studies Document Individual Youthbuild Program Success
          * [9]YouthBuild USA WebSTA Data Have Limitations, Particularly fo
          * [10]Closeout Reports We Analyzed Provided Little Detail on Long-
          * [11]Limited Information We Developed from Site Visits Pointed to
          * [12]HUD Does Not Review or Disseminate Impediments and Best Prac
          * [13]Labor Officials Indicated They Will Consider Using Informati

     * [14]Youthbuild Grantees Have Experienced Varied Success in Obtai
     * [15]Conclusions
     * [16]Recommendations for Executive Action
     * [17]Agency Comments and Our Evaluation
     * [18]GAO Contact
     * [19]Staff Acknowledgments
     * [20]GAO's Mission
     * [21]Obtaining Copies of GAO Reports and Testimony

          * [22]Order by Mail or Phone

     * [23]To Report Fraud, Waste, and Abuse in Federal Programs
     * [24]Congressional Relations
     * [25]Public Affairs

Report to Congressional Committees

United States Government Accountability Office

GAO

February 2007

YOUTHBUILD PROGRAM

Analysis of Outcome Data Needed to Determine Long-Term Benefits

GAO-07-82

Contents

Letter 1

Results in Brief 4
Background 7
HUD's Grant-Making Procedures Incorporated Legislative Criteria 11
HUD Oversight of Grantees' Performance Has Been Limited 14
Incomplete Data on Participants' Outcomes Precludes an Overall Assessment
of Program's Success 19
Youthbuild Grantees Have Experienced Varied Success in Obtaining Outside
Funding Sources, and Many Cited Continued HUD Funding as Crucial 28
Conclusions 31
Recommendations for Executive Action 32
Agency Comments and Our Evaluation 33
Appendix I Comments from the U.S. Department of Labor 35
Appendix II GAO Contact and Staff Acknowledgments 37

Figures

Figure 1: Youthbuild Appropriations, Fiscal Years 1993-2006 8
Figure 2: Characteristics of Participants Entering YouthBuild
USA-Affiliated Programs, 1997-2005 10
Figure 3: NOFA Point-Scoring System for Fiscal Year 2005 Applications 12
Figure 4: HUD Awards by Funding Category for Fiscal Years 2001-2006 13
Figure 5: Youthbuild Grant Applications and the Number Funded by Fiscal
Year (1996-2006) 14
Figure 6: Based on Closeout Reports, Youthbuild Performance Measures
Indicate Varied Outputs 23
Figure 7: Outside Funding Amounts Varied Widely for Grants We Reviewed 29

Abbreviations

CPD Community Planning and Development
GED General Equivalency Diploma
GPRA Government Performance and Results Act
HUD Department of Housing and Urban Development
NOFA Notice of Funding Availability
OMB Office of Management and Budget
WIA Workforce Investment Act

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

February 28, 2007

The Honorable Patty Murray
Chairman
The Honorable Christopher Bond
Ranking Minority Member
Subcommittee on Transportation, Housing and Urban
Development, and Related Agencies
Committee on Appropriations
United States Senate

The Honorable John W. Olver
Chairman
The Honorable Joe Knollenberg
Ranking Minority Member
Subcommittee on Transportation, Housing and Urban
Development, and Related Agencies
Committee on Appropriation
House of Representatives

Since 1993, the Department of Housing and Urban Development (HUD) has
provided funding for Youthbuild, a "second chance" program that trains and
educates disadvantaged youth and helps build housing for low- and very
low-income families across the United States. As stated in the authorizing
legislation, the purposes of the Youthbuild program are (1) expanding the
supply of affordable housing by utilizing economically disadvantaged young
adults, (2) providing economically disadvantaged young adults with
opportunities for meaningful work and service to their communities, (3)
enabling economically disadvantaged young adults to obtain the education
and employment skills necessary to achieve economic self-sufficiency, and
(4) fostering the development of leadership skills and commitment to
community development among young adults in low-income communities.^1
Although a few studies have shown instances where Youthbuild graduates
credit the program for helping them overcome difficult circumstances, gain
critical job skills, and continue their education, concerns have been
expressed about whether HUD's program oversight has been sufficient to
adequately measure program outcomes.

^1Youthbuild was originally authorized by provisions contained in the
Housing and Community Development Act of 1992. The YouthBuild Transfer Act
repealed the original authorizing legislation, effective September 2006,
when the Youthbuild program was transferred to the Department of Labor
(Labor). The original authorizing legislation was in effect during the
time period covered by our report.

The Housing and Community Development Act of 1992 authorized the
Youthbuild program. The program was based on the Youth Action Program
developed in East Harlem, New York, in the late 1970s in which high school
dropouts received academic training and learned job skills by renovating
abandoned buildings. The YouthBuild Transfer Act shifted program oversight
of grants issued beginning in fiscal year 2007 to the Department of Labor
(Labor)--in part, to better align the program with existing federal
workforce development and youth training programs.^2 The Transfer Act also
expanded the program's scope to allow construction and rehabilitation of
community facilities in addition to housing. HUD will continue to oversee
Youthbuild grants funded through fiscal year 2006. That is, HUD's
oversight responsibility could continue for several more years since
grants can remain active for 3 years or more.

Noting concerns regarding the overall quality of the Youthbuild program,
the Senate report accompanying the fiscal year 2006 Transportation,
Treasury, Housing and Urban Development, the Judiciary, the District of
Columbia, and Independent Agencies Appropriations Act directed us to
comprehensively assess the Youthbuild program's overall success, including
its ability to attract local support and funding. Specifically, the
objectives of this report are to (1) describe how HUD makes Youthbuild
program grants; (2) evaluate how HUD assesses and oversees the Youthbuild
program; (3) determine how successful the Youthbuild program has been in
achieving desired results, such as providing the education and job
training needed to achieve economic self-sufficiency and supplying
affordable housing; and (4) assess how successful local Youthbuild
organizations have been in obtaining private funding and support.

We focused our analysis on reviewing HUD's grant making and oversight
processes and evaluating available data for measuring program success. To
determine how HUD makes annual program grants, we reviewed the application
and review process HUD uses to identify and select organizations for grant
awards.^3 To evaluate how HUD assesses and oversees the program, we
interviewed HUD officials at the national and field offices. We also
obtained information on the extent to which grantees are monitored and the
results of the monitoring. Further, we obtained information on the type of
performance data grantees submit to HUD and HUD's role in reviewing the
information. We also obtained funding and grant award information from
HUD. We assessed the reliability of these data by comparing the
information with other data sources to identify obvious problems with
completeness or accuracy, and interviewing knowledgeable agency officials
about the data. We determined that the data were sufficiently reliable for
the purposes of this report. To develop information on how successful the
Youthbuild program has been, we consulted relevant literature, including
three studies of the Youthbuild program's effect on participants' lives.^4
However, the studies were limited in scope and methodology and constrained
by the absence of a database containing performance information from all
the grantees in the Youthbuild program. We also interviewed officials and
received information from HUD's primary technical assistance contractor,
YouthBuild USA. Further, because HUD lacked a centralized database on
grantees' program performance results, we analyzed 245 grant closeout
reports submitted by grantees to HUD from April 2000 to August 2006.^5 We
also visited nine Youthbuild programs in California, Illinois,
Massachusetts, Pennsylvania, Rhode Island, and Washington, D.C.^6 The
sites were chosen in part because the programs varied in size, leadership
style, and teaching approach, including the charter school format. The
closeout reports we reviewed represented about 46 percent of the closed
grants awarded from 1997 through 2003. We were unable to review the
remainder of the reports either because HUD was not able to provide a copy
of the closeout reports, the grantee had not submitted a closeout report,
or the report did not have sufficient information for us to use. In
addition, we reviewed monitoring reports covering 16 grantees prepared by
HUD field office staff as part of their program oversight duties. To
assess how successful local Youthbuild organizations have been in
obtaining private funding and support, we analyzed the closeout reports to
determine the extent to which the individual grantees obtained additional
funding. From a list of 100 grantees from YouthBuild USA that closed their
operations, we obtained information on a nonprobability selection of 23
grantees by directly contacting them or from other knowledgeable sources
to determine how HUD funding decisions contributed to their closure. We
selected and contacted grantees primarily based on those that had received
more than one grant and which we were able to locate because they were
still providing other social services. We also reviewed literature and
reports on Labor's youth employment and training programs and met with
officials from Labor, where oversight of the Youthbuild program will be
transferred. We conducted our work in Boston and Somerville,
Massachusetts; Washington, D.C.; and various Youthbuild locations in the
states noted above from March 2006 through February 2007 in accordance
with generally accepted government auditing standards.

^2HUD spells Youthbuild with a lower case "b," as written in the Housing
and Community Development Act of 1992. As such, when referring to the HUD
program, we spell Youthbuild with a "b." The YouthBuild Transfer Act used
the upper case "B" spelling. To be consistent with the legislation, Labor
plans to use the capitalized spelling when it takes over program
administration.

^3Grantees include public or private nonprofit organizations, states or
units of local government, or any entity eligible to provide education and
employment training under other federal employment training programs.

^4Andrew Hahn, Thomas Leavitt, Erin McNamara Horvat, and James Earl Davis,
"Life after YouthBuild, 900 YouthBuild Graduates Reflect on Their Lives,
Dreams, and Experiences" (YouthBuild USA: June 2004); Maxine V. Mitchell,
CRE et al., "Evaluation of the Youthbuild Program (U.S. Department of
Housing and Urban Development, Office of Policy Development and Research:
August 2003); and Minnesota Department of Economic Security, "Minnesota
Youthbuild Program: A Measurement of Costs and Benefits to the State of
Minnesota" (Commissioned by the Minnesota Youthbuild Coalition, Prepared
by MDES staff: February 2003).

^5HUD identifies the reports as Final Performance Evaluation Reports.
However, in the report we use the term "closeout reports."

^6Youthbuild locations we visited included San Francisco, California;
Bloomington and Rockford, Illinois; Brockton, Lowell, and Springfield,
Massachusetts; Philadelphia, Pennsylvania; Providence, Rhode Island; and
Washington, D.C.

Results in Brief

Each year, HUD staff scored grant applications based on factors outlined
in the enacting legislation for serving economically distressed areas and
detailed in an annual Notice of Funding Availability (NOFA). The Housing
and Community Development Act of 1992 required HUD to establish selection
criteria such as the qualifications or potential capabilities of
applicants, the feasibility of the proposed Youthbuild program, the
potential for developing a successful Youthbuild program, and the
applicants' commitment to obtain outside resources. HUD incorporated these
criteria in an annual NOFA, which directed applicants and detailed the
scoring process for grant awards. The award process has been competitive,
with HUD having sufficient resources to fund less than one-third of the
applicants since 1996.

While HUD requires grantees to submit closeout reports with basic
performance data, HUD has not verified, aggregated, or analyzed the data
and conducts limited oversight of grantees. The data in the closeout
reports were intended for performance assessments; specifically, in its
instructions to grantees, HUD states that closeout reports will be used to
help evaluate the grantees' programs. Further, in accordance with federal
performance management standards, accurate and complete performance data
are needed to inform Congress and the public about the effectiveness of
programs and help agencies effectively manage and oversee programs.^7 HUD
officials said that they did not have staff available to verify,
aggregate, or analyze the closeout reports that they required Youthbuild
grantees to submit; instead, they contracted with YouthBuild USA to
collect a set of performance outputs on a quarterly basis, which provides
a more limited perspective on grantee performance because the data have
not been aggregated for the term of the grants. While HUD has recognized
the importance of performance information by planning for a computerized
data collection system to capture performance information on Youthbuild
and other competitive grant programs, the implementation has been delayed
since 2003 due to funding and technical problems. In addition, according
to HUD officials, a lack of resources also limited assessment and
oversight of Youthbuild grantees. For example, HUD field offices, which
are responsible for overseeing the Youthbuild program, typically were
limited to visiting and reviewing one or two grantees annually. HUD's
monitoring primarily focused on compliance with program requirements such
as documentation rather than on performance. As a result of a lack of
verified and aggregated data and analysis and limited oversight, HUD
largely was unable to tell how well the individual Youthbuild grantee
programs performed.

In addition, lack of programwide performance data and analysis limited any
overall assessment of the success of the Youthbuild program, and limited
available analyses cannot be used for programwide assessments. Since 2002,
a few studies have provided some evidence of the success of individual
Youthbuild programs. While limited in scope, the studies have shown that,
at a small number of grantees, Youthbuild has been successful in assisting
youth to enter higher education programs and obtain job skills. The value
of the data that YouthBuild USA collects in its Web-based system is
limited as well; the data are self-reported and contain little information
on participant outcomes, partly because grantees are not required to
follow participants after graduation. To augment limited existing data, we
analyzed the performance information available in 245 closeout reports
that HUD supplied to us, which covered 12,863 program participants. The
results cannot be generalized to the entire program. For each of the
grantees that reported information, we calculated percentages of
participants who completed or placed in an education program or in
employment. The median percentage of participants who obtained a general
equivalency or high school diploma across all the programs that reported
information was 28, compared with 36 percent who were placed in jobs, and
7 percent who went on to higher education. However, because the reports
lack some baseline information and grantees generally did not follow
participants after program completion, we were unable to determine
long-term outcomes such as percentages or number of participants receiving
degrees or retaining employment. Moreover, while closeout reports do
include information about impediments to program success and "best
practices," HUD did not systematically review this information or share it
with YouthBuild USA, which is the primary technical assistance contractor
for the program with responsibilities that include identifying new
strategies for operating successful programs. As a result, the lack of
follow-up data and programwide evaluations make it very difficult to
determine which programs and strategies have worked best and assess the
performance of Youthbuild over time. As Labor assumes program oversight
responsibilities, officials said they would consider that the closeout
reports include program participant data, such as degrees obtained prior
to program entry and details on participant outcomes such as the types of
construction jobs obtained and retained upon program completion. The
officials also told us that they believe impediments and best practices
should be evaluated and disseminated to grantees, but they had not yet
determined who would have primary responsibility for doing so.

^7Government Performance and Results Act of 1993 and S. Rep. No. 58, 103rd
Cong., 1st Sess. (1993); Office of Management and Budget Circular (OMB)
A-11, Part 6, "Preparation and Submission of Strategic Plans, Annual
Performance Plans, and Annual Program Performance Reports" (June 2006).

Grantees had varying success in obtaining funds from outside sources, but
grant officials told us that HUD funding was critical to attracting the
leveraged funds. In assessing applications, HUD gave consideration to
program applicants that were able to secure outside funding. Applicants
receive points based on the ratio of committed leveraged funds compared
with the amount of Youthbuild funds requested in the application. Data
from the closeout reports showed the median amount spent by HUD Youthbuild
grantees was $400,000 in HUD funds and an additional $430,000 from outside
sources. However, success in obtaining additional funds varied widely,
from 21 grants reporting no additional funding sources to 40 reporting
more than $1 million. While most grantees have been successful in
generating outside funding, several grant officials told us that many
existing and potential contributors would not be willing to continue
funding if they perceived that the program would not be able to continue
without the HUD grant. YouthBuild USA reported that most grantees have had
difficulty continuing operations without continued HUD funding. Further,
YouthBuild USA also noted that grantees with follow-on HUD funding
achieved better performance outcomes than grantees that did not receive
subsequent HUD funding.

This report contains three recommendations for the Secretaries of HUD and
Labor to analyze closeout reports and share the best practices identified,
develop post-program performance outcomes and share the data with
grantees, and consider whether multiyear funding could be useful in
helping Youthbuild grantees attract additional outside funding. We
provided a draft of this report to HUD and Labor for their review and
comments. Officials from HUD's Office of Community Planning and
Development (CPD) provided comments. They stated that they did not
disagree with the report's overall findings, but added that HUD's
oversight activity was limited by resource constraints. Labor's Assistant
Secretary for Employment and Training wrote that she agreed with our
recommendation to develop and monitor post program performance outcome
measures and share data with grantees. She also wrote that Labor agreed
that there could be value in multiyear funding and that the department was
considering such an approach for existing grantees.

Background

CPD manages the Youthbuild program. CPD programs generally fall into three
categories: (1) formula grants, of which the largest are the Community
Development Block Grant and the HOME Investment Partnership Program, which
account for the majority of CPD funding; (2) competitive homeless
programs; and (3) competitive nonhomeless programs, including Youthbuild.
While no staff are dedicated solely to Youthbuild oversight, CPD monitors
the program as part of its oversight responsibilities for 22 programs and
initiatives, which 829 staff (233 in headquarters and 596 in field
offices) administered as of the beginning of fiscal year 2007. In fiscal
year 2006, Congress appropriated $19.1 billion for the CPD programs and
initiatives, of which Youthbuild represents a small portion.^8 The first
program appropriation for Youthbuild was $40 million for fiscal year 1993.
Appropriations reached a high of $65 million in fiscal year 2002, but
dropped to $49.5 million in fiscal year 2006, the last year in which HUD
was authorized to award grants (see fig. 1).^9

8Appropriations for CPD programs and initiatives in fiscal year 2006 were
significantly higher than the $8 billion in prior fiscal years because of
the approximately $11.5 billion Congress added for disaster assistance to
five Gulf Coast states affected by hurricanes.

Figure 1: Youthbuild Appropriations, Fiscal Years 1993-2006

HUD has provided program funding for Youthbuild through an annual
competitive grant application process. Any applicant can receive up to a
$700,000 grant; however, new applicants and rural/underserved applicants
primarily receive grants up to $400,000. Grant awards can last up to 30
months (or longer if the grantee is given an extension), but some grantees
use the funds at a much quicker rate and usually reapply every year, while
others may take the full 30 months.

Prospective grantees are state, private, nonprofit, and local entities
that implement the Youthbuild program by serving low-income youth, ages 16
to 24, who demonstrate educational need--either by virtue of being
high-school dropouts or by justifying their need for inclusion in the
program. According to program regulations, at least 75 percent of the
participants must come from very low-income families and have dropped out
of high school.^10 The program allows up to 25 percent of the youth to
have a high-school diploma or have a slightly higher income; however, data
developed by YouthBuild USA show that most organizations report that less
than 25 percent of participants have high-school diplomas or General
Equivalency Diplomas (GED) upon entering their programs. Also, according
to data from YouthBuild USA, a number of the youths have been adjudicated
(judicially determined to be delinquent), were on welfare upon entering
the program, or lived in public housing (see fig. 2). Other studies we
reviewed identified similar participant characteristics. Program
regulations require participants to spend 50 percent of their program time
receiving job training on construction sites, building affordable housing,
and 50 percent receiving academic training. Recently, some of the
Youthbuild grantees have added nonconstruction programs to their
offerings, such as in the areas of computer repair and health care.
Funding for the additional programs has come from AmeriCorps.^11

9HUD completed the grant awards process in November 2006.

^10Very low-income families are those with incomes 50 percent or less of
area median income.

^11The Corporation for National and Community Service, a public-private
partnership, administers AmeriCorps. The program allows participants to
earn education awards to help pay for postsecondary education in exchange
for performing community service.

Figure 2: Characteristics of Participants Entering YouthBuild
USA-Affiliated Programs, 1997-2005

Note: Participants could have one or more of depicted characteristics.

In 1994, YouthBuild USA, located in Somerville, Massachusetts, became the
primary technical assistance contractor for the Youthbuild program.
YouthBuild USA is a national support center and intermediary for the
program. As such, it provides training, develops materials and handbooks
to help grantees replicate the program, conducts peer-to-peer seminars,
and disseminates materials on best practices. YouthBuild USA also has
established an affiliated network that grantees can choose to join. As of
September 2006, 107 of 173 Youthbuild grantees with active grants were
affiliates. Since 2002, YouthBuild USA has been using a Web-based system,
WebSTA, to collect participant characteristic and performance data from
its affiliates.

Among the purposes of transfering the YouthBuild program to Labor was
better alignment of existing federal workforce and youth training
programs. Labor's Employment and Training Administration offers programs
that assist disadvantaged youths, similar to the programs that Youthbuild
grantees offer. The Workforce Investment Act (WIA) of 1998 established a
variety of programs to serve low-income youths who face barriers to
employment. Labor allocates funds for the youth service programs to state
and local areas based on a distribution formula. The grants are made to
states, and the states in turn allocate funds to approximately 600 local
workforce investment boards. The amount of funding available to states is
determined by the distribution of unemployed individuals and disadvantaged
youths by state. From 2000 through 2005, Labor administered the Youth
Opportunity Grant program, which also was authorized by WIA. This grant
program focused on improving education and employment opportunities for
at-risk youths, ages 14 to 21, in high-poverty areas. Funding for the
program was eliminated beginning in fiscal year 2004. In addition, Labor
administers the Job Corps program, which began in 1964, and provides
education, training and support services to economically challenged
youths, ages 16 through 24, who face multiple barriers to employment.
Private companies--chosen through competitive contracting processes--and
other federal agencies operating under interagency agreements with Labor,
operate the Job Corps centers (located throughout the United States and
Puerto Rico) on behalf of Labor.

HUD's Grant-Making Procedures Incorporated Legislative Criteria

Each year, HUD staff rated and ranked grant applications for serving
economically distressed areas based on factors outlined in the enacting
legislation and detailed in an annual NOFA. The Housing and Community
Development Act of 1992 required HUD to establish selection criteria such
as the qualifications or potential capabilities of applicants, the
feasibility of the proposed Youthbuild program, the potential for
developing a successful Youthbuild program, and the applicants' commitment
to obtain outside resources.^12

Accordingly, HUD annually issued a NOFA that described the grant
application process, outlined criteria for grant awards, and included
scoring factors (see fig. 3). The major factors considered were the
capacity and relevant experience of the organization (performance), need
and extent of the problem, applicant's soundness of approach, leveraging
of nonhousing funds, and ability to show how it would achieve and measure
results. Under these factors, grant applicants had to consider area
poverty and high-school dropout rates, describe their management and
training capabilities, and offer proof of access to property and the
existence of partners to build low-income housing. HUD issued its last
NOFA for the Youthbuild program on March 8, 2006, and announced the
winning grantees in November 2006.

^12HUD program officials also complied with the fairness, transparency,
and nondisclosure requirements of sections 102 and 103 of the HUD Reform
Act of 1989.

Figure 3: NOFA Point-Scoring System for Fiscal Year 2005 Applications

Since fiscal year 2001, HUD has grouped applicants into three separate
scoring categories--new, general (that are typically awarded to
established Youthbuild organizations), and rural/underserved area
grantees. Figure 4 shows the awards by category. The total numbers of
grants as well as the grants by the individual categories have varied by
year.

Figure 4: HUD Awards by Funding Category for Fiscal Years 2001-2006

The awards process also has been competitive. HUD historically funded less
than one-third of all the applicants (see fig. 5). Since 1996, the number
of applicants funded ranged from about 7 to 44 percent.

Figure 5: Youthbuild Grant Applications and the Number Funded by Fiscal
Year (1996-2006)

HUD Oversight of Grantees' Performance Has Been Limited

Although HUD requires grantees to submit closeout reports, which are
intended to help evaluate the programs, it has not verified, aggregated,
or analyzed data from those reports. Further, the closeout reports do not
include other data that could help determine participant characteristics
upon entry or the types of jobs they received after completing the
program. YouthBuild USA, the technical services provider for HUD, collects
quarterly information, but it has not aggregated the information to obtain
performance information over the span of a grant. HUD also has encountered
significant delays in developing a database intended to gather additional
performance information from grantees. Finally, citing resource
constraints, HUD has conducted limited on-site oversight of grantees. As a
result of a lack of verified and aggregated data or analysis and limited
oversight, HUD largely was unable to tell how well the individual
Youthbuild grantees performed.

HUD Has Not Collected or Analyzed Data That Could Help It Assess Grantees

HUD requires grantees to submit performance reports, including closeout
reports, but HUD has not developed or fully utilized
information--particularly from the closeout reports--that could help it
assess Youthbuild grantee programs and has lacked resources to conduct
comprehensive oversight of grantees. Such information serves as the basis
for management and oversight of programs. For instance, HUD states in the
instructions for filling out the closeout reports that the reports will be
used to monitor and evaluate progress of grantees and programs. Further,
the Government Performance and Results Act of 1993 (GPRA) shifted the
focus of federal managers from the number of tasks completed or services
provided to a more direct consideration of the outcomes or results of the
program. GPRA provides a performance-based management framework for
agencies to set goals; measure progress toward those goals; deploy
strategies and resources to achieve them; and ultimately, use performance
information to make the programmatic decisions necessary to improve
performance. Accurate and complete performance data that document outcomes
are needed to inform Congress and the public about the effectiveness of
programs and help agencies conduct effective oversight. The combination of
baseline information and intermediate outputs or outcomes can be used to
show progress or contribution toward final outcomes, that is, the intended
results of carrying out the program.^13 For example, in Youthbuild,
baseline information such as the number of participants entering a program
with and without GEDs or high-school diplomas could be combined with
intermediate outputs and outcomes such as graduation rates and jobs
attained, which could then be linked to desired program outcomes such as
the number of Youthbuild participants attaining economic self-sufficiency.

However, HUD has not set up a system to verify, aggregate, or analyze the
data from closeout reports--information and analyses that would form the
basis for assessments of grantee performance. Citing resource constraints,
HUD filed the reports in the individual grant files in headquarters and
field offices and did not further review them. In addition, some basic
baseline data are missing from the closeout reports, including the number
of participants who enter the programs with a GED or high-school diploma.
Such information is necessary to get an accurate count of the number of
participants receiving a GED degree or diploma through Youthbuild programs
and also provides a basis for comparison with which performance can be
measured. Furthermore, while closeout reports require that some
performance outputs be collected, including the number of participants
receiving GEDs or high-school diplomas, placed in jobs, or entering higher
education programs, the reports do not collect other output information
such as how many participants were placed in construction-related
employment.

^13GPRA; S. Rep. No. 58, 103rd Cong., 1st Sess. (1993); and OMB Circular
A-11.

Citing resource constraints and a need to comply with the Office of
Management and Budget (OMB) performance reporting requirements, HUD
contracted with YouthBuild USA in 2004 to collect quarterly grantee
performance information for all Youthbuild grantees. Programs administered
by six federal agencies, including HUD and Labor, are subject to "common
measures" reporting which OMB established to provide agencies with the
ability to describe, in a similar manner, their core performance outputs
across different grantee programs.^14 The common measures include
employment or education placements, degrees or certificates attained, and
literacy and numeracy skills increased (increases in educational
functioning). Consequently, YouthBuild USA provides HUD with program
information including participant gender and ethnicity data, GED
attainment statistics, grade level improvements, job or school placements,
and the number of new and rehabilitated housing units that were completed.
HUD uses the reports to provide performance information to Congress and
meet OMB reporting requirements. However, because the information is
captured on a quarterly basis, it only represents the performance of a
program at a particular point in time and YouthBuild USA has not
aggregated the information to obtain data for the term of individual
grants. As a result, HUD has been largely unable to track program
outcomes, but its data will be needed for ongoing program assessments.

Moreover, HUD has encountered significant delays in developing a database
intended to provide additional performance outcomes on grantee programs.
Recognizing this deficiency, HUD officials said that they have been
planning for about 3 years to implement an electronic data collection
system to help HUD measure the success of Youthbuild programs, along with
its other nonhomeless competitive grant programs. However, the system's
implementation has been delayed because of funding and technical problems.
As a result of not systematically collecting or analyzing needed
information from closeout reports, relying predominately on limited
point-in-time program information, and not being able to utilize a planned
database to aggregate performance outcomes, HUD is limiting its ability to
assess grantees. Further, although the YouthBuild Transfer Act shifted
oversight of grants issued in fiscal year 2007 to Labor, HUD will be
required to continue monitoring existing Youthbuild grants for
approximately the next 3 years. Therefore, the data HUD has at its
disposal or could develop will continue to be valuable for ongoing
assessments of the program.

^14In addition to HUD and Labor, the Departments of Education, Health, and
Human Services, Veterans Affairs, and Interior are subject to the common
measures.

Citing Resource Constraints, HUD Performed Limited Oversight of Grantees

In addition to not having or utilizing available performance data, HUD has
exercised limited on-site review and oversight of grantee performance.
According to HUD's Office of Inspector General, on-site monitoring is an
essential tool for HUD (through CPD) to assess program performance and
identify and address potential program problems. Such monitoring is also
useful in motivating grantees to exercise sound judgment in carrying out
their grant activities. GAO's Standards for Internal Control in the
Federal Government note that controls generally should be in place to
ensure that ongoing monitoring occurs during normal operations. Monitoring
should be performed continually, be ingrained in an entity's operations,
be used to assess the quality of performance over time, and support the
prompt resolution of any identified problems.^15

CPD goals require on-site monitoring of only a small percentage of grants.
For fiscal year 2003, the plan called for the field offices to monitor 35
percent of formula grants; 20 percent of competitive homeless grants; and
7 percent of competitive nonhomeless grants, including Youthbuild. HUD has
since changed its baseline monitoring goals (requiring monitoring of 20
percent of grantees with active, open grants established since 2004) to
give its field offices more flexibility to meet monitoring priorities
based on local grantee knowledge. As a result, goals are no longer set by
program. Field offices decide on the number of formula versus competitive
grantees to be monitored based on risk analyses that evaluate each grantee
on factors such as the size of the grant, perceived project management
capacity, grantee responsiveness, and the extent of citizen complaints.
HUD officials cited a lack of resources as the primary reason for not
requiring the monitoring of a larger percentage of grantees. In 2006,
according to HUD officials, field offices typically had one person working
part time responsible for overseeing the individual Youthbuild grantees in
the state, visiting no more than one or two grantees per year. As a
result, field staff have not been able to comprehensively monitor grantees
and verify the accuracy of the output data submitted by the grantees.

^15GAO issued these standards as required by 31 U.S.C. S3512(c). Also see
GAO, Standards for Internal Control in the Federal Government,
[26]GAO/AIMD-00-21 .3.1 (Washington, D.C.: November 1999) and Internal
Control Management and Evaluation Tool, [27]GAO-01- 1008G (Washington,
D.C.: August 2001).

In general, we found that the monitoring reports that we reviewed (which
HUD field offices produced) on grantees primarily addressed various
compliance issues, rather than reporting on the overall performance of a
particular grant. For example, several reports identified grantees'
failure to receive prior approval from HUD for changing proposed job
training construction sites. Other reports identified inadequate record
keeping, insufficient documentation, and lack of sufficient financial
internal controls on the part of grantees. However, although a few reports
did comment on some individual performance outputs, such as job placements
or GEDs attained, most did not.

The lack of analysis of performance information of grantees also has
implications for assessing the Youthbuild program overall, particularly
because HUD has devoted few staff to overall program management and
oversight. Since 2000, in headquarters, Youthbuild had a program director
with one or two staff managing the program each year. During fiscal year
2006, with the expected program transfer to Labor, the HUD program manager
was assigned to other duties, and a part-time program manager was assigned
to oversee the 2006 grant process, in which HUD received 308 applications
and awarded 74 grants. Although they are aware of the limited oversight of
Youthbuild, CPD division officials said it has provided appropriate staff
resources for program oversight of Youthbuild grantees, commensurate with
the program's size--the annual appropriation for which represents less
than 1 percent of CPD's usual annual budget of about $8 billion. However,
without sufficient oversight HUD was unable to tell how the individual
Youthbuild grantee programs performed.

Incomplete Data on Participants' Outcomes Precludes an Overall Assessment of
Program's Success

HUD has not performed the programwide analysis needed to determine overall
program success, and weaknesses in other assessments of the programs and
data preclude an overall assessment of Youthbuild success. While limited
in scope, three studies suggest Youthbuild helped youths achieve economic
self-sufficiency. And, although YouthBuild USA tracks performance data,
grantees do not have the follow-up information needed to measure
performance outcomes. To augment limited existing data, we analyzed
information in 245 closeout reports, but the results cannot be generalized
to the entire program. We found a wide range of results in outputs such as
job placements; however, because grantees generally did not follow up on
participants, we were unable to determine long-term outcomes such as
percentages of participants retaining employment. Moreover, while closeout
reports include impediments to success and "best practices," HUD did not
systematically review this information or share it with YouthBuild USA. As
a result, the lack of follow-up data and programwide evaluations make it
very difficult to determine which programs and strategies have worked best
and assess the performance of Youthbuild over time. Labor officials said
they would consider that the closeout reports include outcomes such as
jobs obtained and retained and said that they believe impediments and best
practices should be evaluated and disseminated to grantees.

A Few Studies Document Individual Youthbuild Program Successes but Are Limited
in Scope

A few studies have provided some evidence of the success of individual
Youthbuild programs. Although subject to many limitations, the studies
suggest that identified programs have achieved certain Youthbuild
objectives. However, without comparable information for the entire
population of Youthbuild grantees, it is not possible to determine the
extent to which these outcomes and reported successes are reflective of
what was achieved by the grantees not included in the studies. More
specifically:

           o In June 2004, researchers from Brandeis and Temple Universities
           published a study entitled, "Life after YouthBuild - 900
           YouthBuild Graduates Reflect on Their Lives, Dreams, and
           Experiences," in collaboration with YouthBuild USA. The study was
           based on a survey of graduates from 11 Youthbuild grant sites and
           included in-depth interviews with 57 graduates from eight of these
           sites. The researchers determined that, for those 882 participants
           (for a 23 percent response rate) who responded to the survey, many
           are doing well and were helped considerably by the Youthbuild
           program. Many participants found jobs or furthered their education
           and have volunteered in community activities. In addition, a
           number of participants surveyed indicated they also obtained
           intangible benefits from the program, such as increased
           self-esteem and had developed positive attitudes about themselves
           and their community. A limited nonresponse analysis was conducted
           by YouthBuild USA to compare the 882 survey responses and the 57
           interviews with one site that had a 98 percent response rate.
           However, using one site that had a high response rate is not
           representative and cannot be generalized to the whole sample. As
           such, it is not known if the 77 percent who did not respond were
           doing more or less well than those who responded to the survey.
           o In August 2003, HUD issued a study entitled, "Evaluation of the
           Youthbuild Program," in which a HUD contractor compared program
           results and costs per participant from the Youthbuild Program and
           four other federal employment and training programs--Job Corps,
           JOBSTART, both designed to serve disadvantage youth, and the youth
           components of the Job Training Partnership Act and Supported Work
           Demonstration programs.^16 The researchers conducted their work at
           20 Youthbuild sites, selected in part because they had received
           funding for at least two grants, obtained a variety of
           organizational sponsorships, built new or rehabilitated housing,
           and covered a variety of funding category sizes. The study found
           that 36 percent of the Youthbuild participants at these sites
           obtained jobs after enrolling and 29 percent obtained either a GED
           or high-school diploma. In addition, 12 percent of the Youthbuild
           graduates in the 20 sites reportedly went onto higher education.
           However, results at the five federal programs were not directly
           comparable because researchers obtained data from Youthbuild
           program participants at the time they exited the program while
           data from the other programs' were obtained from tracking
           participants up to 3 years after leaving the program. In addition,
           the limited scope of this study precludes an overall assessment of
           the program's success.
           o In February 2003, the state of Minnesota issued a study
           entitled, "Minnesota Youthbuild Program: A Measurement of Costs
           and Benefits to the State of Minnesota." This study analyzed the
           costs and benefits of the state-funded Youthbuild program in
           Minnesota. The researchers measured post-program participant
           performance outcomes by tracking program year participants beyond
           graduation. They found that Youthbuild graduates earned more than
           double the minimum wage, whereas youth with similar
           characteristics who had not been through the program typically
           earned only the minimum wage. In addition, the graduates who had
           been adjudicated had a considerably lower rate of recidivism
           (rearrest, reconviction, or return to prison) than other youth
           involved with the correctional system.^17 Using state
           administrative data to estimate the amount of revenues and savings
           generated through increased tax revenues and reduced prison costs,
           the study estimated first-year benefits to the state of about $1.5
           million dollars versus costs of about $877,000. Over a 4- year
           period, the study estimated the total net benefit to the state at
           about $7.3 million (by 2006). The study was limited to costs and
           benefits directly related to state programs and did not include
           potential benefits such as reductions to welfare rolls,
           state-sponsored medical care, or other reductions in human
           services costs.
			  
			  YouthBuild USA WebSTA Data Have Limitations, Particularly for
			  Assessing Outcomes

           Since 2002, YouthBuild USA has been using a Web-based
           system--WebSTA--to collect participant characteristics and
           performance data from its affiliates. According to YouthBuild USA
           officials, as of December 2006, about 90 percent of its affiliates
           were entering data into WebSTA to some extent. However, the value
           of YouthBuild USA's WebSTA data is limited, largely because
           grantees self-report the data, not all grantees are meeting the
           requirement to send in information, and the information is not
           reported at the grant level. That is, if affiliates have more than
           one Youthbuild grant, the data for each grant are not reported
           separately. As a result, the data are not verified, comprehensive,
           or comparable. Also, because the grantees submit the information
           quarterly, the data provide only a point-in-time snapshot of the
           program's performance and cannot be summed to provide annual data.
           Recognizing these limitations, according to YouthBuild USA's
           President, upgrades are planned to be implemented in early 2007
           that will allow grantees to enter data in WebSTA by individual
           grant, which in turn will allow YouthBuild USA to analyze
           grant-level performance.

           We also found that the limited depth of the information collected
           restricts the value of WebSTA information for performance
           assessments. For example, WebSTA collects information on
           participant attendance, retention and graduation rates, and
           initial job and college placements and participant characteristics
           (such as welfare, public housing, and adjudication status) when
           participants enter Youthbuild programs. However, many grantees do
           not follow up on participants once they leave the programs. As a
           result, they often do not know if students placed in jobs were
           still working at those jobs weeks or months later. Also, several
           program directors we visited said that most of their program
           graduates who go on to higher education sign up for free
           remedial-level courses at nearby colleges. Although they have not
           had the funding to systematically follow the students' progress,
           some said they generally have found that the students have had
           varying success in advancing to college-level credit courses.

           Without grantee-level follow-up systems, WebSTA cannot track
           program outcomes that would reflect whether the graduates are
           achieving economic self-sufficiency--such as the percentage of
           program graduates who remove themselves from the welfare rolls,
           move out of public housing, or stay out of the court system. The
           system also does not collect follow-up information on the type of
           jobs attained (including construction), pay raises, benefits
           received, and college or trade degrees or certifications received.
           Analyzing such performance outcomes could provide, HUD, Labor, and
           other interested parties with the measures needed to evaluate
           overall program performance and success.
			  
			  Closeout Reports We Analyzed Provided Little Detail on Long-Term
			  Outcomes Because Grantees Are Not Required to Follow up on Graduates

           Because HUD had not aggregated or analyzed data for Youthbuild, we
           obtained and analyzed 245 closeout reports that grantees submitted
           to HUD. Figure 6 provides selected performance information that we
           summarized from the closeout reports. However, because HUD has not
           established objective performance goals for the program, which
           would serve as baseline criteria against which to judge
           performance, the information related to the experience of the
           12,863 participants who entered the program under the 245 grants
           gave us limited perspectives about how individual programs were
           working. In addition, because we had information on only 245 (or
           46 percent) of the grants, the results cannot be generalized to
           the entire program.

           As shown in figure 6 the percentages for participants graduating,
           receiving a GED, or being placed in employment varied widely. As
           previously discussed, HUD has not set benchmarks, and critical
           data elements (baseline data) are missing in the closeout reports.
           For instance, it is not possible to obtain an accurate number of
           diplomas and GEDs attained from the closeout reports because the
           reports do not contain data on the number of participants that
           enter the program with a high-school diploma or GED. As result, we
           could not determine how well the programs (except for those with
           the very highest percentages) performed individually or in
           comparison with each other.

^16The Job Training Partnership Act program was replaced by the Workforce
Investment Act, which was implemented in 2000. This evaluation included
data from 1996 through 1999 when the Job Training Partnership Act programs
were still in effect.

^17Minnesota Youthbuild program participants entered unsubsidized
employment with an average starting wage of $11.60 an hour. The minimum
wage of $5.15 an hour was used as a baseline for the comparison group.
Fourteen (5 percent) of the participants with one or more offenses prior
to enrollment were rearrested, reconvicted, or returned to a state
correctional facility within 2 years after enrollment in the Minnesota
Youthbuild program, compared with a 20 percent rate of return for
juveniles within 2 years of release from Minnesota correctional
facilities.

Figure 6: Based on Closeout Reports, Youthbuild Performance Measures
Indicate Varied Outputs

^aNot all grantees provided information for each category.

^bIncludes 1,239 participants that were reported as still active because
they were carried over to a subsequent grant or allowed to stay on the
books to receive continuing education and other support services. In
addition, the drop outs, graduates, and still active categories do not sum
to the number of participants category because all grantees did not report
information for each category.

^cRegulations allow up to 25 percent of the participants who enroll to
already have a GED or high-school diploma. As a result, this number may be
underrepresentative since the data do not specify the number of
participants that entered the program with a GED or high-school diploma.

^dThe same participants can be included in the employment and higher
education categories.

The closeout reports also do not provide consistent and reliable data on
the number of units rehabilitated or built by Youthbuild participants. A
basic purpose for which Youthbuild was authorized was to expand the supply
of affordable housing. According to the closeout reports, 158 grantees
reported that their participants built a total of 1,483 new units, while
139 grantees reported that their participants rehabilitated a total of
2,238 units.^18 Some of the Youthbuild program directors with whom we met
said that during a typical reporting period they would complete from one
to four new or gut-rehabilitated housing units. Other directors said that
when a program reports about 200 units rehabilitated, it generally means
the students are performing minor repetitive tasks such as changing locks
or painting a limited number of interior walls. As a result, the number of
units rehabilitated likely is overstated in the closeout reports. Although
construction training is a major part of the Youthbuild program, HUD's
closeout reports also do not track the number of graduates obtaining
construction-related jobs or specify the types of jobs obtained
(construction or otherwise).

In addition, HUD officials told us that grantees are not required to
follow up on participants after they leave the program. While some grantee
officials with whom we met recently had attempted to establish follow-up
systems, most grantees cited funding constraints as the reason for not
following up on graduates. Several officials further noted that it was
difficult to locate many participants after they had left the program.
However, as a result, it is not possible to obtain comprehensive
post-program results.

Finally, some program directors with whom we spoke cautioned that any
follow-up system should not be strictly focused on students obtaining
higher education degrees or obtaining high-paying jobs. They emphasized
that such goals were beyond the abilities of some of their participants,
but that moving from a fifth-grade to a ninth-grade reading level or
obtaining a service sector job sometimes was a real victory and should be
counted as a success.

Limited Information We Developed from Site Visits Pointed to Varied Results for
Programs, and Also Provided Little Detail on Long-Term Outcomes

The information we obtained from our visits also demonstrates varied
results for continued employment and academic success, and little detail
on long-term success. As with our closeout report analysis, the
information cannot be generalized to the universe of Youthbuild
grantees--although YouthBuild USA officials told us that they believe we
visited some of the more innovative grantees. For example, although not
required, some of the nine grantees we visited used other funding sources,
such as United Way or AmeriCorps funds, to establish a participant
follow-up system. As a result, some of the grantees were able to provide
us with varying amounts of information on the status of program graduates.
For example:

^18Some grantees reported that they used their program to both construct
new units as well as rehabilitated others under the same grant.

           o One grantee, which was able to track individuals who completed
           the program since 2001, provided us with data that showed 70 of
           the 82 program participants were either employed or pursuing
           higher education. Seven of the 70 participants were working and
           going to school. The hourly wages of those employed ranged from $7
           to $13.75. However, the grantee did not have information on the
           types of employment.
           o Another grantee that recently established a follow-up system
           showed that 11 of the 26 program graduates who started in 2004 had
           obtained employment. The positions ranged from pizza delivery
           person to union carpenter. However, only the union carpenter and a
           graduate hired by the program as a construction trainer were
           working in the construction field. The hourly wages of the
           graduates ranged from $5.50 to more than $16 per hour for a
           graduate hired by the grantee.
           o Another grantee used WEBSTA to track continuing education and
           work history, including each job held. The program director told
           us that such a follow-up system allows the program to spot trends,
           which in turn can be used to fine-tune the program. Of 60
           graduates from 2002 to 2005, the program continually has tracked
           53 and reported that 8 are receiving higher education, 24 are
           working in nonconstruction jobs, and 16 are working in
           construction jobs. The pay for the graduates ranged from $6.50 to
           $15 per hour. The grantee continues to keep in touch with the
           graduates by helping them with housing, legal aid, school advice,
           tutoring, and help toward achieving a GED.

           Grantees are not required to collect such post-program follow-up
           information. Yet, both grantees and program participants
           emphasized that the Youthbuild program created a very supportive
           atmosphere that many participants had never experienced prior to
           joining a program. During our visits, we met with students who
           credited the program with improving their lives by either helping
           get them off drugs, get out of the gang lifestyle, or disengage
           from other activities that likely could have resulted in
           incarceration or worse. The studies we mentioned previously had
           similar findings. However, it is difficult to gain insight into
           how many program participants remain drug free, crime free, in
           advanced degree programs, or working in construction or other
           trades without post-program performance data. It is also difficult
           to gain insight into which program strategies are most effective
           for serving these youth.
			  
			  HUD Does Not Review or Disseminate Impediments and Best Practices
			  Identified in Closeout Reports That Could Help Improve Program
			  Performance

           Citing resource constraints, HUD did not systematically review the
           impediments and best practices identified in closeout reports or
           share the information in them with YouthBuild USA. In particular,
           identifying best practices (or the converse, impediments) can
           provide a model for improvement for other organizations with
           similar missions. The closeout reports contain a section
           addressing what the grantees considered to be best practices. For
           example:

           o Several grantees set-up educational opportunities, such as GED
           preparation, at local colleges for participants. Two grantees
           noted that being on a college campus provided a positive
           experience for the participants. Two grantees noted that they made
           an arrangement with a local technical college to offer free
           enrollment to all participants that successfully completed the
           Youthbuild program. Another grantee noted that working through
           community college GED programs allowed them to lower their
           per-student cost and serve more students.
           o One grantee established an alumni council consisting of elected
           representatives from each of its seven graduating classes. All
           graduates receive quarterly mailings that include an alumni
           newsletter, new job and educational opportunities, and a quarterly
           calendar of events.
           o Another grantee has maintained a program serving graduates for
           up to 12 months to provide on-going employment support, including
           assistance in securing a second job in the event that the initial
           placement did not work out.
           o One grantee started a "drug free" club and found that students
           really wanted to participate and were anxious to become drug free
           and would publicly announce their success.
           o Another grantee worked extensively recruiting area college
           students to volunteer to help with tutoring and program community
           service activities.

           Closeout reports also contain information on impediments that
           grantees encountered in implementing and operating the programs.
           Impediments to program success varied in scope and severity. Some
           of the more significant impediments mentioned by numerous grantees
           included the following.

           o Turnover of key staff, especially on-site construction managers;
           o Negative and apathetic attitudes among participants;
           o Alcohol and drug abuse issues among participants;
           o Difficulties in dealing with municipal organizations to obtain
           building permits and contractual difficulties, which delayed
           construction training activities; and
           o Acceptance of students with very low educational levels, which
           made obtaining high GED levels very difficult.

           HUD's contract with YouthBuild USA specifically calls for the
           contractor to assist HUD in the management, supervision, and
           coordination of Youthbuild programs by strengthening Youthbuild
           program design and disseminating information on best program
           practices. Although analyzing the impediment and best practices
           falls within the scope of HUD's contract with YouthBuild USA, two
           CPD program officials said that HUD never considered sharing the
           reports with its technical services provider. YouthBuild USA's
           chief operating officer said that the organization would welcome
           access to the closeout reports. The officer said his staff already
           spent many hours each month trying to identify best practices from
           Youthbuild grantees and would be anxious to analyze the closeout
           reports for both the problems and successes identified and forward
           the information to interested grantees. YouthBuild USA's President
           told us that she not only agreed that reviewing the closeout
           reports for problems and "best practices" would be beneficial, but
           that she believes the closeout reports needed to "be analyzed to
           get to the bottom line because the total impact (outcomes) of a
           grant could not be determined from data provided on a rolling
           quarterly basis."^19 As a result of not sharing the closeout
           reports with YouthBuild USA, HUD may have missed opportunities to
           improve the program as a whole and help grantees improve
           performance. 
			  
			  Labor Officials Indicated They Will Consider Using Information
			  from Closeout Reports to Assess and Improve YouthBuild
			  Performance

           According to Labor officials, they will consider analyzing
           information from the closeout reports to assess and improve the
           performance of the YouthBuild program. Until 2006, Labor assessed
           past performance of its youth employment and training grantees
           through measures required under WIA. For example, under the WIA
           Youth and Job Corps programs, Labor tracked and analyzed 6-month
           retention and earnings wage change information. Labor officials
           told us that they would use OMB's common measures to track and
           analyze the performance of YouthBuild grantees but said that they
           have not finalized the information they would require YouthBuild
           grantees to collect. The officials indicated to us that they would
           consider developing post-program performance outcomes such as the
           types of employment graduates attained and retained (including
           construction), wage rates, and college or trade degrees or
           certifications received. Although Labor may incur some additional
           costs in developing such information, the ability to measure
           program success would improve. Labor officials also noted that the
           information currently contained in the closeout reports would be
           considered with other factors they deem as pertinent to assessing
           grantees' performance. However officials said they would also
           likely consider modifying the closeout reports to include other
           program participant data, such as degrees obtained prior to
           program entry and details on the types of construction jobs
           obtained upon program completion.
			  
^19As we previously discussed, the quarterly data cannot be summed to
provide an annualized report.

           Labor officials also acknowledged that information on program
           impediments and best practices should be evaluated and passed to
           grantees and that performance data needed to be aggregated,
           verified, and analyzed to determine which programs and strategies
           worked best. As of October 2006, Labor had not determined whether
           its Employment and Training Administration or an outside
           contractor would have primary responsibility for collecting,
           aggregating, and disseminating the information.
			  
			  Youthbuild Grantees Have Experienced Varied Success in Obtaining
			  Outside Funding Sources, and Many Cited Continued HUD Funding as
			  Crucial

           Our analysis of available closeout reports showed that the
           grantees have had varying success in obtaining additional funding
           sources. In assessing grant applications, HUD gave consideration
           to program applicants that were able to secure outside funding.
           Applicants received points for leveraging nonhousing resources
           compared with the amount of Youthbuild funds requested in the
           application. Overall, the median amount spent by a HUD Youthbuild
           grantee was $400,000 in HUD funds; the HUD grant represented about
           48 percent of the average total spending per grantee, which
           included additional funding sources of about $430,000. However,
           success in obtaining additional funds varied widely, from 21
           grants reporting no additional funding sources to 40 reporting
           more than $1 million (see fig. 7).

           Figure 7: Outside Funding Amounts Varied Widely for Grants We
           Reviewed

           According to information in the closeout reports and the officials
           from the grantees that we visited, the additional funding they
           obtained largely came from federal, state, and local agencies,
           businesses, and charitable organizations. According to a
           YouthBuild USA official, states have provided funding to local
           Youthbuild grantees over the past several years. For instance, the
           Massachusetts State Department of Education has provided funding
           in its budget for the state's Youthbuild program. The
           Massachusetts funds are provided to existing Youthbuild programs,
           based on a formula the state developed in conjunction with a state
           coalition of Youthbuild grantees. In Minnesota, 10 Youthbuild
           programs in 2005 shared $754,000 in state Youthbuild program
           funding, which the state supplemented with more than $3 million in
           matching funds from local partners. In Minnesota as elsewhere,
           urban grantees have had greater opportunities to obtain funding or
           in-kind support from local businesses and charitable organizations
           than rural grantees that typically have had access to fewer
           businesses and might not be a focus of charitable activity.

           Regardless of their success in obtaining outside funds, several
           grantees we visited stated that their Youthbuild program could not
           be sustained without the HUD grant because the HUD funding allowed
           them to attract the leveraged funds. Also, several grant officials
           said that many existing and potential contributors would not be
           willing to continue funding if they perceived that the program
           would not be able to continue without the HUD grant. One exception
           (which we did not visit) was in Minnesota where the state
           Youthbuild program director said that only 4 of the state's 10
           programs have had HUD funds in the past, including 2 programs
           which received HUD funding in 2004 through a HUD grant
           administered by the state. The program director added that the
           structure of the Minnesota state Youthbuild program is similar to
           that of HUD's, although the programs are typically smaller, having
           from 10 to 25 students.

           Information from YouthBuild USA generally agrees with grantees'
           contention about the need for continuous HUD funding. Overall,
           YouthBuild USA found that 60 percent of programs that HUD did not
           fund for 2 consecutive years ceased operations, and 90 percent of
           programs not funded for 3 consecutive years ceased operations.
           According to YouthBuild USA, 173 of 462 grantees funded since 1994
           currently are operating with active HUD grants. YouthBuild USA,
           also found that grantees with continuous HUD funding achieved
           higher outputs, such as GED attainments and job placements, than
           the ones that did not receive one or more follow-on grants.
           According to some grantee officials, that success is partly due to
           continuous funding that allows grantees to develop infrastructure
           and maintain experienced staff to establish a better program. In
           its fiscal year 2005 Performance and Accountability Report, HUD
           noted the benefits of making grants to previous awardees: "Having
           established programs and experience running a Youthbuild program,
           these grantees are more efficient in enrolling students, resulting
           in a greater number of youth trained." Furthermore, Youthbuild
           program directors with whom we met emphasized the intangible
           benefit that such established programs provide such as a family
           atmosphere that allows graduates to return for advice and support
           as needed--a benefit that goes away when programs close.

           HUD has not kept statistics to find out why some grantees did not
           reapply for additional grants or why grantees did not qualify for
           follow-up grants. Through discussions with a nonprobability
           selection of 23 of the 100 grantees that closed programs, we found
           that some did not reapply because they were no longer interested
           in dealing with at-risk youth and others closed due to poor
           management. But most of the closed grantees with whom we spoke
           said they closed because they lost HUD funding for one or more
           years and were unable to obtain sufficient private funds to make
           up for the lost funding. Of the sites we contacted, two were still
           in operation after ceasing to receive HUD funds--but they appeared
           to be the exception. Both were already operating youth technical
           training programs prior to their first HUD Youthbuild grant.

           In addition, officials from several grantees that also received
           funding from AmeriCorps stated that they preferred the AmeriCorps
           concept of providing dedicated funds for 3 years. These officials
           stated that knowing that funding levels were longer-term provided
           more continuity to their program, enabled better planning, and
           allowed them to build a cushion through leveraged funding in the
           event that they did not receive a HUD grant for a year or two.
           Finally, when the question of multiyear funding was raised at a
           Web-based listening session in December 2006, designed to help
           Labor identify Youthbuild program administration issues and
           concerns, most participating grantees spoke out in favor of
           multiyear funding.
			  
			  Conclusions

           HUD has not taken advantage of several opportunities to use
           existing information for purposes of program assessment and
           oversight. Such opportunities should not be ignored, particularly
           because HUD has limited resources with which to assess and oversee
           current grantees. Specifically, HUD has not verified, aggregated
           or analyzed closeout reports, which grantees are required to
           submit. HUD also has conducted limited oversight of grantees,
           citing resource constraints; the impending transfer of oversight
           to Labor bodes no change to this specific situation. However,
           given these conditions, analysis of existing closeout reports
           would help provide crucial information on the performance of the
           Youthbuild programs, particularly in the absence of a centralized
           HUD database that could be used to track program performance. At a
           minimum, the closeout reports form a logical basis for providing
           useful information that HUD, Labor, and other interested parties
           can use to judge the performance of Youthbuild. HUD still will
           maintain an active involvement for several more years, so
           coordination between Labor and HUD will be critical to producing
           reliable assessments of how successful the program has been and
           will be. HUD also does not share reported constraints and best
           practices with the technical services contractor it pays for
           performing such work and disseminating best practices. By not
           utilizing performance data and sharing information, HUD has missed
           opportunities to determine whether the overall Youthbuild program
           has achieved its intended results, among them enabling
           disadvantaged young adults to gain economic self-sufficiency.
           Therefore, Congress and the public also lack the information
           needed to make such determinations.

           Labor has experience in managing youth training and education
           programs and collecting performance information, and Congress
           transferred Youthbuild to Labor in part to better align the
           program with existing federal workforce development and youth
           training programs. In offering perspectives for Labor to consider,
           we stress the value that closeout reports have in the absence of
           more comprehensive and programwide data, although the existing
           reports lack certain elements that could help program managers
           determine and report on outcomes. The reports do contain valuable
           information on impediments and best practices, which could provide
           both program managers and grantees with much broader perspectives
           on how successful the program has been and suggestions for
           improvements. Labor has indicated that it would consider
           developing post-program performance outcomes such as the types of
           jobs graduates attained and retained (including construction),
           wage rates, and college or trade degrees or certifications
           received. As a result, while there may be some additional costs
           involved, the ability to measure program success would improve.
           And, because alternatives to the HUD data are limited, the
           grant-level assessment discussed above could act as the building
           blocks of an overall program assessment. Labor also has agreed
           that information on impediments and best practices would be
           valuable and ought to be passed on to grantees, but has not yet
           determined how this might be done.

           Finally, data on grantee success in attracting other funding,
           although limited, offer more perspectives for Labor to consider
           because outside funding helps leverage limited federal dollars.
           While many grantees have obtained extensive outside funding and
           support, YouthBuild USA has found that very few operated for more
           than a few years without continued HUD funding. Tellingly, their
           data show that 90 percent of programs not funded by HUD for 3
           consecutive years ceased operations. According to YouthBuild USA
           officials, their data indicate that grantees with repeat awards
           also produced better outcomes such as a higher level of job
           placements. Further, when HUD funding stopped, potential
           contributors became concerned that grantees would no longer stay
           in business and were reluctant to jeopardize their own funds.
           These data, while not definitive, do suggest that multiyear
           funding has the potential to produce dividends for the program.
           Many grantees expressed similar belief in the benefits of longer
           term funding at a Department of Labor YouthBuild program listening
           session.
			  
			  Recommendations for Executive Action

           To improve the reporting and assessment of performance for
           Youthbuild grantee programs and develop the bases for an overall
           assessment of the program, we are making the following three
           recommendations:

           o that the Secretary of HUD analyze closeout reports by grant and
           share information on identified problems and "best practices" with
           its technical services contractor and Labor;
           o that the Secretary of Labor develop and monitor post-program
           performance outcome measures for the YouthBuild program, such as
           the types of employment graduates attained and retained, wage
           rates, and degrees or certifications received, and share the data
           with the grantees; and
           o that the Secretary of Labor consider whether multiyear funding
           could be useful in helping YouthBuild grantees attract additional
           outside funding.
			  
			  Agency Comments and Our Evaluation

           We provided HUD and Labor with a draft of this report for review
           and comment. HUD's comments are summarized below. Labor's
           Assistant Secretary for Employment and Training provided written
           comments that are presented in appendix I.

           Officials from HUD's Office of Community and Planning and
           Development provided comments. They stated that they did not
           disagree with GAO's overall findings, but added that HUD's
           oversight responsibility was limited by resource constraints. They
           cited the fact that the Youthbuild program represents 0.63 percent
           of the Community Planning and Development Division's entire
           portfolio. Nevertheless, because HUD will have monitoring
           responsibilities for existing grants for 3 years or more, we
           believe it is important that HUD analyze the closeout reports in
           order to provide Labor with information about how current grantees
           are performing.

           Labor's Assistant Secretary for Employment and Training wrote that
           she believes that the report will be very useful to Labor as it
           assumes responsibility for administering the YouthBuild program.
           She noted that Labor agreed with our recommendation to develop and
           monitor post program performance outcome measures for the
           YouthBuild program and share these data with grantees. She added
           that Labor was building a Web-based Management Information System
           to collect and report on participant information. Finally, she
           wrote that Labor agreed that there could be value on multiyear
           funding and that the Department was considering such an approach
           for existing grantees.

           We are sending copies of this report to other interested
           congressional committees and the Secretaries of Housing and Urban
           Development and Labor. We will also make copies available to
           others upon request. The report will also be available at no
           charge on GAO's Web site at http://www.gao.gov .

           If you or your staff have questions regarding this report, please
           contact me at (202) 512-4325 or [email protected] . Contact
           points for our Offices of Public and Congressional Affairs may be
           found on the last page of this report. GAO staff who made major
           contributions to this report are listed in appendix II.

           William B. Shear
			  Director, Financial Markets and Community
           Investment
			  
			  Appendix I: Comments from the U.S. Department of Labor
			  
			  Appendix II: GAO Contact and Staff Acknowledgments
			  
			  GAO Contact

           William B. Shear, (202) 512-4325, [email protected]
			  
			  Staff Acknowledgments

           In addition to the individual named above, Andy Finkel, Assistant
           Director; Rich LaMore, John McGrail, Marc Molino, Luann Moy,
           Barbara Roesmann, and Tom Taydus made key contributions to this
           report.
			  
			  GAOï¿½s Mission

           The Government Accountability Office, the audit, evaluation and
           investigative arm of Congress, exists to support Congress in
           meeting its constitutional responsibilities and to help improve
           the performance and accountability of the federal government for
           the American people. GAO examines the use of public funds;
           evaluates federal programs and policies; and provides analyses,
           recommendations, and other assistance to help Congress make
           informed oversight, policy, and funding decisions. GAO's
           commitment to good government is reflected in its core values of
           accountability, integrity, and reliability.
			  
			  Obtaining Copies of GAO Reports and Testimony

           The fastest and easiest way to obtain copies of GAO documents at
           no cost is through GAO's Web site ( www.gao.gov ). Each
           weekday, GAO posts newly released reports, testimony, and
           correspondence on its Web site. To have GAO e-mail you a list of
           newly posted products every afternoon, go to www.gao.gov and
           select "Subscribe to Updates."
			  
			  Order by Mail or Phone

           The first copy of each printed report is free. Additional copies
           are $2 each. A check or money order should be made out to the
           Superintendent of Documents. GAO also accepts VISA and Mastercard.
           Orders for 100 or more copies mailed to a single address are
           discounted 25 percent. Orders should be sent to:

           U.S. Government Accountability Office 441 G Street NW, Room LM
           Washington, D.C. 20548

           To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
           (202) 512-6061
			  
			  To Report Fraud, Waste, and Abuse in Federal Programs

           Contact:

           Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
           [email protected] Automated answering system: (800) 424-5454 or
           (202) 512-7470
			  
			  Congressional Relations

           Gloria Jarmon, Managing Director, [email protected] (202)
           512-4400 U.S. Government Accountability Office, 441 G Street NW,
           Room 7125 Washington, D.C. 20548
			  
			  Public Affairs

           Paul Anderson, Managing Director, [email protected] (202)
           512-4800 U.S. Government Accountability Office, 441 G Street NW,
           Room 7149 Washington, D.C. 20548

(250279)

GAO's Mission

www.gao.gov/cgi-bin/getrpt?GAO-07-82 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact William B. Shear at (202) 512-4325 or
[email protected].

Highlights of [37]GAO-07-82 , a report to congressional committees

February 2007

YOUTHBUILD PROGRAM

Analysis of Outcome Data Needed to Determine Long-Term Benefits

Since 1993, the Department of Housing and Urban Development (HUD) has
provided funding for Youthbuild, a competitive grant program that trains
and educates disadvantaged youth and helps build low-income housing. In
2006, Youthbuild was transferred to the Department of Labor (Labor) to
better align the Youthbuild program with existing youth workforce and
training programs. In response to concerns about the overall quality of
Youthbuild, a Senate report directed GAO to assess the program. GAO's
objectives included (1) evaluating how HUD assessed and oversaw the
program, (2) determining what results the program achieved, and (3)
assessing how successful grantees were in obtaining outside funding. GAO
analyzed Youthbuild performance data, visited Youthbuild sites, and
interviewed agency officials.

[38]What GAO Recommends

GAO recommends that (1) HUD analyze closeout reports by grant and share
information on identified problems and best practices with its technical
assistance contractor and Labor, (2) Labor develop post-program
performance outcomes, and (3) Labor consider multiyear funding of grants
for the program. HUD and Labor provided comments on a draft of this
report. Labor generally agreed with the recommendations. HUD agreed but
cited resource constraints in implementing its recommendation.

While HUD requires grantees to report basic performance data, such as the
number of program participants and graduates and job placements, HUD has
not aggregated or analyzed the data and conducted limited oversight of
grantees. According to HUD officials, they did not have staff available to
analyze the closeout reports that grantees must submit, and a lack of
resources also limited oversight of grantees. The monitoring HUD did
primarily focused on compliance with program requirements such as
documentation rather than on performance. As a result, HUD largely was
unable to tell how the individual Youthbuild grantee programs performed.

Limited outcome data preclude any overall assessment of the performance of
the Youthbuild program; further, the few other analyses available such as
the one GAO did in this study to augment limited existing data cannot be
generalized programwide. GAO analyzed 245 closeout reports, representing
46 percent of the grantees or 12,863 participants. While GAO could
determine percentages of participants who received high school diplomas or
were placed in jobs, GAO could not determine outcomes over time, partly
because the reports lacked baseline information and grantees were not
required to and generally did not follow participants after graduation.
Further, while closeout reports include information about impediments to
program success and "best practices," HUD did not systematically review
this information or share it with its primary technical assistance
contractor. Consequently, the lack of programwide evaluations, follow-up
data, and dissemination of best practices make it very difficult to assess
the performance of Youthbuild over time and determine which programs and
strategies have worked best. Reporting on post-program performance
outcomes, such as the number of participants placed and retained in
construction-related employment, could increase the value of the closeout
reports and better measure program results. Labor officials indicated that
they would consider including such measures for program reporting.

Grantees had varying success in obtaining funds from outside sources, but
YouthBuild USA data suggest that continued (multiyear) HUD funding was
critical to sustaining grantee operations and attracting leveraged funds.
Grantees' success in obtaining additional funds varied widely, from 21
grants reporting no additional funding sources to 40 reporting more than
$1 million. While most grantees have generated outside funding, YouthBuild
USA reported that most grantees have had difficulty continuing operations
without continued HUD funding. Their data show that 90 percent of grantees
ceased operations if not funded for 3 consecutive years by HUD. Further,
YouthBuild USA also noted that grantees with follow-on HUD funding
achieved better performance outcomes, such as higher rates of job
placements, than grantees that did not receive subsequent HUD funding.

References

Visible links
  26. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1
  27. http://www.gao.gov/cgi-bin/getrpt?GAO-01-1008G
  37. http://www.gao.gov/cgi-bin/getrpt?GAO-07-82
*** End of document. ***