Youthbuild Program: Analysis of Outcome Data Needed to Determine
Long-Term Benefits (28-FEB-07, GAO-07-82).
Since 1993, the Department of Housing and Urban Development (HUD)
has provided funding for Youthbuild, a competitive grant program
that trains and educates disadvantaged youth and helps build
low-income housing. In 2006, Youthbuild was transferred to the
Department of Labor (Labor) to better align the Youthbuild
program with existing youth workforce and training programs. In
response to concerns about the overall quality of Youthbuild, a
Senate report directed GAO to assess the program. GAO's
objectives included (1) evaluating how HUD assessed and oversaw
the program, (2) determining what results the program achieved,
and (3) assessing how successful grantees were in obtaining
outside funding. GAO analyzed Youthbuild performance data,
visited Youthbuild sites, and interviewed agency officials.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-82
ACCNO: A66359
TITLE: Youthbuild Program: Analysis of Outcome Data Needed to
Determine Long-Term Benefits
DATE: 02/28/2007
SUBJECT: Data collection
Disadvantaged persons
Federal grants
Government grants
Government information
Low income housing
Performance measures
Program evaluation
Reporting requirements
Youth employment programs
Executive agency oversight
HUD Youthbuild Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-82
* [1]Results in Brief
* [2]Background
* [3]HUD's Grant-Making Procedures Incorporated Legislative Crite
* [4]HUD Oversight of Grantees' Performance Has Been Limited
* [5]HUD Has Not Collected or Analyzed Data That Could Help It As
* [6]Citing Resource Constraints, HUD Performed Limited Oversight
* [7]Incomplete Data on Participants' Outcomes Precludes an Overa
* [8]A Few Studies Document Individual Youthbuild Program Success
* [9]YouthBuild USA WebSTA Data Have Limitations, Particularly fo
* [10]Closeout Reports We Analyzed Provided Little Detail on Long-
* [11]Limited Information We Developed from Site Visits Pointed to
* [12]HUD Does Not Review or Disseminate Impediments and Best Prac
* [13]Labor Officials Indicated They Will Consider Using Informati
* [14]Youthbuild Grantees Have Experienced Varied Success in Obtai
* [15]Conclusions
* [16]Recommendations for Executive Action
* [17]Agency Comments and Our Evaluation
* [18]GAO Contact
* [19]Staff Acknowledgments
* [20]GAO's Mission
* [21]Obtaining Copies of GAO Reports and Testimony
* [22]Order by Mail or Phone
* [23]To Report Fraud, Waste, and Abuse in Federal Programs
* [24]Congressional Relations
* [25]Public Affairs
Report to Congressional Committees
United States Government Accountability Office
GAO
February 2007
YOUTHBUILD PROGRAM
Analysis of Outcome Data Needed to Determine Long-Term Benefits
GAO-07-82
Contents
Letter 1
Results in Brief 4
Background 7
HUD's Grant-Making Procedures Incorporated Legislative Criteria 11
HUD Oversight of Grantees' Performance Has Been Limited 14
Incomplete Data on Participants' Outcomes Precludes an Overall Assessment
of Program's Success 19
Youthbuild Grantees Have Experienced Varied Success in Obtaining Outside
Funding Sources, and Many Cited Continued HUD Funding as Crucial 28
Conclusions 31
Recommendations for Executive Action 32
Agency Comments and Our Evaluation 33
Appendix I Comments from the U.S. Department of Labor 35
Appendix II GAO Contact and Staff Acknowledgments 37
Figures
Figure 1: Youthbuild Appropriations, Fiscal Years 1993-2006 8
Figure 2: Characteristics of Participants Entering YouthBuild
USA-Affiliated Programs, 1997-2005 10
Figure 3: NOFA Point-Scoring System for Fiscal Year 2005 Applications 12
Figure 4: HUD Awards by Funding Category for Fiscal Years 2001-2006 13
Figure 5: Youthbuild Grant Applications and the Number Funded by Fiscal
Year (1996-2006) 14
Figure 6: Based on Closeout Reports, Youthbuild Performance Measures
Indicate Varied Outputs 23
Figure 7: Outside Funding Amounts Varied Widely for Grants We Reviewed 29
Abbreviations
CPD Community Planning and Development
GED General Equivalency Diploma
GPRA Government Performance and Results Act
HUD Department of Housing and Urban Development
NOFA Notice of Funding Availability
OMB Office of Management and Budget
WIA Workforce Investment Act
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
February 28, 2007
The Honorable Patty Murray
Chairman
The Honorable Christopher Bond
Ranking Minority Member
Subcommittee on Transportation, Housing and Urban
Development, and Related Agencies
Committee on Appropriations
United States Senate
The Honorable John W. Olver
Chairman
The Honorable Joe Knollenberg
Ranking Minority Member
Subcommittee on Transportation, Housing and Urban
Development, and Related Agencies
Committee on Appropriation
House of Representatives
Since 1993, the Department of Housing and Urban Development (HUD) has
provided funding for Youthbuild, a "second chance" program that trains and
educates disadvantaged youth and helps build housing for low- and very
low-income families across the United States. As stated in the authorizing
legislation, the purposes of the Youthbuild program are (1) expanding the
supply of affordable housing by utilizing economically disadvantaged young
adults, (2) providing economically disadvantaged young adults with
opportunities for meaningful work and service to their communities, (3)
enabling economically disadvantaged young adults to obtain the education
and employment skills necessary to achieve economic self-sufficiency, and
(4) fostering the development of leadership skills and commitment to
community development among young adults in low-income communities.^1
Although a few studies have shown instances where Youthbuild graduates
credit the program for helping them overcome difficult circumstances, gain
critical job skills, and continue their education, concerns have been
expressed about whether HUD's program oversight has been sufficient to
adequately measure program outcomes.
^1Youthbuild was originally authorized by provisions contained in the
Housing and Community Development Act of 1992. The YouthBuild Transfer Act
repealed the original authorizing legislation, effective September 2006,
when the Youthbuild program was transferred to the Department of Labor
(Labor). The original authorizing legislation was in effect during the
time period covered by our report.
The Housing and Community Development Act of 1992 authorized the
Youthbuild program. The program was based on the Youth Action Program
developed in East Harlem, New York, in the late 1970s in which high school
dropouts received academic training and learned job skills by renovating
abandoned buildings. The YouthBuild Transfer Act shifted program oversight
of grants issued beginning in fiscal year 2007 to the Department of Labor
(Labor)--in part, to better align the program with existing federal
workforce development and youth training programs.^2 The Transfer Act also
expanded the program's scope to allow construction and rehabilitation of
community facilities in addition to housing. HUD will continue to oversee
Youthbuild grants funded through fiscal year 2006. That is, HUD's
oversight responsibility could continue for several more years since
grants can remain active for 3 years or more.
Noting concerns regarding the overall quality of the Youthbuild program,
the Senate report accompanying the fiscal year 2006 Transportation,
Treasury, Housing and Urban Development, the Judiciary, the District of
Columbia, and Independent Agencies Appropriations Act directed us to
comprehensively assess the Youthbuild program's overall success, including
its ability to attract local support and funding. Specifically, the
objectives of this report are to (1) describe how HUD makes Youthbuild
program grants; (2) evaluate how HUD assesses and oversees the Youthbuild
program; (3) determine how successful the Youthbuild program has been in
achieving desired results, such as providing the education and job
training needed to achieve economic self-sufficiency and supplying
affordable housing; and (4) assess how successful local Youthbuild
organizations have been in obtaining private funding and support.
We focused our analysis on reviewing HUD's grant making and oversight
processes and evaluating available data for measuring program success. To
determine how HUD makes annual program grants, we reviewed the application
and review process HUD uses to identify and select organizations for grant
awards.^3 To evaluate how HUD assesses and oversees the program, we
interviewed HUD officials at the national and field offices. We also
obtained information on the extent to which grantees are monitored and the
results of the monitoring. Further, we obtained information on the type of
performance data grantees submit to HUD and HUD's role in reviewing the
information. We also obtained funding and grant award information from
HUD. We assessed the reliability of these data by comparing the
information with other data sources to identify obvious problems with
completeness or accuracy, and interviewing knowledgeable agency officials
about the data. We determined that the data were sufficiently reliable for
the purposes of this report. To develop information on how successful the
Youthbuild program has been, we consulted relevant literature, including
three studies of the Youthbuild program's effect on participants' lives.^4
However, the studies were limited in scope and methodology and constrained
by the absence of a database containing performance information from all
the grantees in the Youthbuild program. We also interviewed officials and
received information from HUD's primary technical assistance contractor,
YouthBuild USA. Further, because HUD lacked a centralized database on
grantees' program performance results, we analyzed 245 grant closeout
reports submitted by grantees to HUD from April 2000 to August 2006.^5 We
also visited nine Youthbuild programs in California, Illinois,
Massachusetts, Pennsylvania, Rhode Island, and Washington, D.C.^6 The
sites were chosen in part because the programs varied in size, leadership
style, and teaching approach, including the charter school format. The
closeout reports we reviewed represented about 46 percent of the closed
grants awarded from 1997 through 2003. We were unable to review the
remainder of the reports either because HUD was not able to provide a copy
of the closeout reports, the grantee had not submitted a closeout report,
or the report did not have sufficient information for us to use. In
addition, we reviewed monitoring reports covering 16 grantees prepared by
HUD field office staff as part of their program oversight duties. To
assess how successful local Youthbuild organizations have been in
obtaining private funding and support, we analyzed the closeout reports to
determine the extent to which the individual grantees obtained additional
funding. From a list of 100 grantees from YouthBuild USA that closed their
operations, we obtained information on a nonprobability selection of 23
grantees by directly contacting them or from other knowledgeable sources
to determine how HUD funding decisions contributed to their closure. We
selected and contacted grantees primarily based on those that had received
more than one grant and which we were able to locate because they were
still providing other social services. We also reviewed literature and
reports on Labor's youth employment and training programs and met with
officials from Labor, where oversight of the Youthbuild program will be
transferred. We conducted our work in Boston and Somerville,
Massachusetts; Washington, D.C.; and various Youthbuild locations in the
states noted above from March 2006 through February 2007 in accordance
with generally accepted government auditing standards.
^2HUD spells Youthbuild with a lower case "b," as written in the Housing
and Community Development Act of 1992. As such, when referring to the HUD
program, we spell Youthbuild with a "b." The YouthBuild Transfer Act used
the upper case "B" spelling. To be consistent with the legislation, Labor
plans to use the capitalized spelling when it takes over program
administration.
^3Grantees include public or private nonprofit organizations, states or
units of local government, or any entity eligible to provide education and
employment training under other federal employment training programs.
^4Andrew Hahn, Thomas Leavitt, Erin McNamara Horvat, and James Earl Davis,
"Life after YouthBuild, 900 YouthBuild Graduates Reflect on Their Lives,
Dreams, and Experiences" (YouthBuild USA: June 2004); Maxine V. Mitchell,
CRE et al., "Evaluation of the Youthbuild Program (U.S. Department of
Housing and Urban Development, Office of Policy Development and Research:
August 2003); and Minnesota Department of Economic Security, "Minnesota
Youthbuild Program: A Measurement of Costs and Benefits to the State of
Minnesota" (Commissioned by the Minnesota Youthbuild Coalition, Prepared
by MDES staff: February 2003).
^5HUD identifies the reports as Final Performance Evaluation Reports.
However, in the report we use the term "closeout reports."
^6Youthbuild locations we visited included San Francisco, California;
Bloomington and Rockford, Illinois; Brockton, Lowell, and Springfield,
Massachusetts; Philadelphia, Pennsylvania; Providence, Rhode Island; and
Washington, D.C.
Results in Brief
Each year, HUD staff scored grant applications based on factors outlined
in the enacting legislation for serving economically distressed areas and
detailed in an annual Notice of Funding Availability (NOFA). The Housing
and Community Development Act of 1992 required HUD to establish selection
criteria such as the qualifications or potential capabilities of
applicants, the feasibility of the proposed Youthbuild program, the
potential for developing a successful Youthbuild program, and the
applicants' commitment to obtain outside resources. HUD incorporated these
criteria in an annual NOFA, which directed applicants and detailed the
scoring process for grant awards. The award process has been competitive,
with HUD having sufficient resources to fund less than one-third of the
applicants since 1996.
While HUD requires grantees to submit closeout reports with basic
performance data, HUD has not verified, aggregated, or analyzed the data
and conducts limited oversight of grantees. The data in the closeout
reports were intended for performance assessments; specifically, in its
instructions to grantees, HUD states that closeout reports will be used to
help evaluate the grantees' programs. Further, in accordance with federal
performance management standards, accurate and complete performance data
are needed to inform Congress and the public about the effectiveness of
programs and help agencies effectively manage and oversee programs.^7 HUD
officials said that they did not have staff available to verify,
aggregate, or analyze the closeout reports that they required Youthbuild
grantees to submit; instead, they contracted with YouthBuild USA to
collect a set of performance outputs on a quarterly basis, which provides
a more limited perspective on grantee performance because the data have
not been aggregated for the term of the grants. While HUD has recognized
the importance of performance information by planning for a computerized
data collection system to capture performance information on Youthbuild
and other competitive grant programs, the implementation has been delayed
since 2003 due to funding and technical problems. In addition, according
to HUD officials, a lack of resources also limited assessment and
oversight of Youthbuild grantees. For example, HUD field offices, which
are responsible for overseeing the Youthbuild program, typically were
limited to visiting and reviewing one or two grantees annually. HUD's
monitoring primarily focused on compliance with program requirements such
as documentation rather than on performance. As a result of a lack of
verified and aggregated data and analysis and limited oversight, HUD
largely was unable to tell how well the individual Youthbuild grantee
programs performed.
In addition, lack of programwide performance data and analysis limited any
overall assessment of the success of the Youthbuild program, and limited
available analyses cannot be used for programwide assessments. Since 2002,
a few studies have provided some evidence of the success of individual
Youthbuild programs. While limited in scope, the studies have shown that,
at a small number of grantees, Youthbuild has been successful in assisting
youth to enter higher education programs and obtain job skills. The value
of the data that YouthBuild USA collects in its Web-based system is
limited as well; the data are self-reported and contain little information
on participant outcomes, partly because grantees are not required to
follow participants after graduation. To augment limited existing data, we
analyzed the performance information available in 245 closeout reports
that HUD supplied to us, which covered 12,863 program participants. The
results cannot be generalized to the entire program. For each of the
grantees that reported information, we calculated percentages of
participants who completed or placed in an education program or in
employment. The median percentage of participants who obtained a general
equivalency or high school diploma across all the programs that reported
information was 28, compared with 36 percent who were placed in jobs, and
7 percent who went on to higher education. However, because the reports
lack some baseline information and grantees generally did not follow
participants after program completion, we were unable to determine
long-term outcomes such as percentages or number of participants receiving
degrees or retaining employment. Moreover, while closeout reports do
include information about impediments to program success and "best
practices," HUD did not systematically review this information or share it
with YouthBuild USA, which is the primary technical assistance contractor
for the program with responsibilities that include identifying new
strategies for operating successful programs. As a result, the lack of
follow-up data and programwide evaluations make it very difficult to
determine which programs and strategies have worked best and assess the
performance of Youthbuild over time. As Labor assumes program oversight
responsibilities, officials said they would consider that the closeout
reports include program participant data, such as degrees obtained prior
to program entry and details on participant outcomes such as the types of
construction jobs obtained and retained upon program completion. The
officials also told us that they believe impediments and best practices
should be evaluated and disseminated to grantees, but they had not yet
determined who would have primary responsibility for doing so.
^7Government Performance and Results Act of 1993 and S. Rep. No. 58, 103rd
Cong., 1st Sess. (1993); Office of Management and Budget Circular (OMB)
A-11, Part 6, "Preparation and Submission of Strategic Plans, Annual
Performance Plans, and Annual Program Performance Reports" (June 2006).
Grantees had varying success in obtaining funds from outside sources, but
grant officials told us that HUD funding was critical to attracting the
leveraged funds. In assessing applications, HUD gave consideration to
program applicants that were able to secure outside funding. Applicants
receive points based on the ratio of committed leveraged funds compared
with the amount of Youthbuild funds requested in the application. Data
from the closeout reports showed the median amount spent by HUD Youthbuild
grantees was $400,000 in HUD funds and an additional $430,000 from outside
sources. However, success in obtaining additional funds varied widely,
from 21 grants reporting no additional funding sources to 40 reporting
more than $1 million. While most grantees have been successful in
generating outside funding, several grant officials told us that many
existing and potential contributors would not be willing to continue
funding if they perceived that the program would not be able to continue
without the HUD grant. YouthBuild USA reported that most grantees have had
difficulty continuing operations without continued HUD funding. Further,
YouthBuild USA also noted that grantees with follow-on HUD funding
achieved better performance outcomes than grantees that did not receive
subsequent HUD funding.
This report contains three recommendations for the Secretaries of HUD and
Labor to analyze closeout reports and share the best practices identified,
develop post-program performance outcomes and share the data with
grantees, and consider whether multiyear funding could be useful in
helping Youthbuild grantees attract additional outside funding. We
provided a draft of this report to HUD and Labor for their review and
comments. Officials from HUD's Office of Community Planning and
Development (CPD) provided comments. They stated that they did not
disagree with the report's overall findings, but added that HUD's
oversight activity was limited by resource constraints. Labor's Assistant
Secretary for Employment and Training wrote that she agreed with our
recommendation to develop and monitor post program performance outcome
measures and share data with grantees. She also wrote that Labor agreed
that there could be value in multiyear funding and that the department was
considering such an approach for existing grantees.
Background
CPD manages the Youthbuild program. CPD programs generally fall into three
categories: (1) formula grants, of which the largest are the Community
Development Block Grant and the HOME Investment Partnership Program, which
account for the majority of CPD funding; (2) competitive homeless
programs; and (3) competitive nonhomeless programs, including Youthbuild.
While no staff are dedicated solely to Youthbuild oversight, CPD monitors
the program as part of its oversight responsibilities for 22 programs and
initiatives, which 829 staff (233 in headquarters and 596 in field
offices) administered as of the beginning of fiscal year 2007. In fiscal
year 2006, Congress appropriated $19.1 billion for the CPD programs and
initiatives, of which Youthbuild represents a small portion.^8 The first
program appropriation for Youthbuild was $40 million for fiscal year 1993.
Appropriations reached a high of $65 million in fiscal year 2002, but
dropped to $49.5 million in fiscal year 2006, the last year in which HUD
was authorized to award grants (see fig. 1).^9
8Appropriations for CPD programs and initiatives in fiscal year 2006 were
significantly higher than the $8 billion in prior fiscal years because of
the approximately $11.5 billion Congress added for disaster assistance to
five Gulf Coast states affected by hurricanes.
Figure 1: Youthbuild Appropriations, Fiscal Years 1993-2006
HUD has provided program funding for Youthbuild through an annual
competitive grant application process. Any applicant can receive up to a
$700,000 grant; however, new applicants and rural/underserved applicants
primarily receive grants up to $400,000. Grant awards can last up to 30
months (or longer if the grantee is given an extension), but some grantees
use the funds at a much quicker rate and usually reapply every year, while
others may take the full 30 months.
Prospective grantees are state, private, nonprofit, and local entities
that implement the Youthbuild program by serving low-income youth, ages 16
to 24, who demonstrate educational need--either by virtue of being
high-school dropouts or by justifying their need for inclusion in the
program. According to program regulations, at least 75 percent of the
participants must come from very low-income families and have dropped out
of high school.^10 The program allows up to 25 percent of the youth to
have a high-school diploma or have a slightly higher income; however, data
developed by YouthBuild USA show that most organizations report that less
than 25 percent of participants have high-school diplomas or General
Equivalency Diplomas (GED) upon entering their programs. Also, according
to data from YouthBuild USA, a number of the youths have been adjudicated
(judicially determined to be delinquent), were on welfare upon entering
the program, or lived in public housing (see fig. 2). Other studies we
reviewed identified similar participant characteristics. Program
regulations require participants to spend 50 percent of their program time
receiving job training on construction sites, building affordable housing,
and 50 percent receiving academic training. Recently, some of the
Youthbuild grantees have added nonconstruction programs to their
offerings, such as in the areas of computer repair and health care.
Funding for the additional programs has come from AmeriCorps.^11
9HUD completed the grant awards process in November 2006.
^10Very low-income families are those with incomes 50 percent or less of
area median income.
^11The Corporation for National and Community Service, a public-private
partnership, administers AmeriCorps. The program allows participants to
earn education awards to help pay for postsecondary education in exchange
for performing community service.
Figure 2: Characteristics of Participants Entering YouthBuild
USA-Affiliated Programs, 1997-2005
Note: Participants could have one or more of depicted characteristics.
In 1994, YouthBuild USA, located in Somerville, Massachusetts, became the
primary technical assistance contractor for the Youthbuild program.
YouthBuild USA is a national support center and intermediary for the
program. As such, it provides training, develops materials and handbooks
to help grantees replicate the program, conducts peer-to-peer seminars,
and disseminates materials on best practices. YouthBuild USA also has
established an affiliated network that grantees can choose to join. As of
September 2006, 107 of 173 Youthbuild grantees with active grants were
affiliates. Since 2002, YouthBuild USA has been using a Web-based system,
WebSTA, to collect participant characteristic and performance data from
its affiliates.
Among the purposes of transfering the YouthBuild program to Labor was
better alignment of existing federal workforce and youth training
programs. Labor's Employment and Training Administration offers programs
that assist disadvantaged youths, similar to the programs that Youthbuild
grantees offer. The Workforce Investment Act (WIA) of 1998 established a
variety of programs to serve low-income youths who face barriers to
employment. Labor allocates funds for the youth service programs to state
and local areas based on a distribution formula. The grants are made to
states, and the states in turn allocate funds to approximately 600 local
workforce investment boards. The amount of funding available to states is
determined by the distribution of unemployed individuals and disadvantaged
youths by state. From 2000 through 2005, Labor administered the Youth
Opportunity Grant program, which also was authorized by WIA. This grant
program focused on improving education and employment opportunities for
at-risk youths, ages 14 to 21, in high-poverty areas. Funding for the
program was eliminated beginning in fiscal year 2004. In addition, Labor
administers the Job Corps program, which began in 1964, and provides
education, training and support services to economically challenged
youths, ages 16 through 24, who face multiple barriers to employment.
Private companies--chosen through competitive contracting processes--and
other federal agencies operating under interagency agreements with Labor,
operate the Job Corps centers (located throughout the United States and
Puerto Rico) on behalf of Labor.
HUD's Grant-Making Procedures Incorporated Legislative Criteria
Each year, HUD staff rated and ranked grant applications for serving
economically distressed areas based on factors outlined in the enacting
legislation and detailed in an annual NOFA. The Housing and Community
Development Act of 1992 required HUD to establish selection criteria such
as the qualifications or potential capabilities of applicants, the
feasibility of the proposed Youthbuild program, the potential for
developing a successful Youthbuild program, and the applicants' commitment
to obtain outside resources.^12
Accordingly, HUD annually issued a NOFA that described the grant
application process, outlined criteria for grant awards, and included
scoring factors (see fig. 3). The major factors considered were the
capacity and relevant experience of the organization (performance), need
and extent of the problem, applicant's soundness of approach, leveraging
of nonhousing funds, and ability to show how it would achieve and measure
results. Under these factors, grant applicants had to consider area
poverty and high-school dropout rates, describe their management and
training capabilities, and offer proof of access to property and the
existence of partners to build low-income housing. HUD issued its last
NOFA for the Youthbuild program on March 8, 2006, and announced the
winning grantees in November 2006.
^12HUD program officials also complied with the fairness, transparency,
and nondisclosure requirements of sections 102 and 103 of the HUD Reform
Act of 1989.
Figure 3: NOFA Point-Scoring System for Fiscal Year 2005 Applications
Since fiscal year 2001, HUD has grouped applicants into three separate
scoring categories--new, general (that are typically awarded to
established Youthbuild organizations), and rural/underserved area
grantees. Figure 4 shows the awards by category. The total numbers of
grants as well as the grants by the individual categories have varied by
year.
Figure 4: HUD Awards by Funding Category for Fiscal Years 2001-2006
The awards process also has been competitive. HUD historically funded less
than one-third of all the applicants (see fig. 5). Since 1996, the number
of applicants funded ranged from about 7 to 44 percent.
Figure 5: Youthbuild Grant Applications and the Number Funded by Fiscal
Year (1996-2006)
HUD Oversight of Grantees' Performance Has Been Limited
Although HUD requires grantees to submit closeout reports, which are
intended to help evaluate the programs, it has not verified, aggregated,
or analyzed data from those reports. Further, the closeout reports do not
include other data that could help determine participant characteristics
upon entry or the types of jobs they received after completing the
program. YouthBuild USA, the technical services provider for HUD, collects
quarterly information, but it has not aggregated the information to obtain
performance information over the span of a grant. HUD also has encountered
significant delays in developing a database intended to gather additional
performance information from grantees. Finally, citing resource
constraints, HUD has conducted limited on-site oversight of grantees. As a
result of a lack of verified and aggregated data or analysis and limited
oversight, HUD largely was unable to tell how well the individual
Youthbuild grantees performed.
HUD Has Not Collected or Analyzed Data That Could Help It Assess Grantees
HUD requires grantees to submit performance reports, including closeout
reports, but HUD has not developed or fully utilized
information--particularly from the closeout reports--that could help it
assess Youthbuild grantee programs and has lacked resources to conduct
comprehensive oversight of grantees. Such information serves as the basis
for management and oversight of programs. For instance, HUD states in the
instructions for filling out the closeout reports that the reports will be
used to monitor and evaluate progress of grantees and programs. Further,
the Government Performance and Results Act of 1993 (GPRA) shifted the
focus of federal managers from the number of tasks completed or services
provided to a more direct consideration of the outcomes or results of the
program. GPRA provides a performance-based management framework for
agencies to set goals; measure progress toward those goals; deploy
strategies and resources to achieve them; and ultimately, use performance
information to make the programmatic decisions necessary to improve
performance. Accurate and complete performance data that document outcomes
are needed to inform Congress and the public about the effectiveness of
programs and help agencies conduct effective oversight. The combination of
baseline information and intermediate outputs or outcomes can be used to
show progress or contribution toward final outcomes, that is, the intended
results of carrying out the program.^13 For example, in Youthbuild,
baseline information such as the number of participants entering a program
with and without GEDs or high-school diplomas could be combined with
intermediate outputs and outcomes such as graduation rates and jobs
attained, which could then be linked to desired program outcomes such as
the number of Youthbuild participants attaining economic self-sufficiency.
However, HUD has not set up a system to verify, aggregate, or analyze the
data from closeout reports--information and analyses that would form the
basis for assessments of grantee performance. Citing resource constraints,
HUD filed the reports in the individual grant files in headquarters and
field offices and did not further review them. In addition, some basic
baseline data are missing from the closeout reports, including the number
of participants who enter the programs with a GED or high-school diploma.
Such information is necessary to get an accurate count of the number of
participants receiving a GED degree or diploma through Youthbuild programs
and also provides a basis for comparison with which performance can be
measured. Furthermore, while closeout reports require that some
performance outputs be collected, including the number of participants
receiving GEDs or high-school diplomas, placed in jobs, or entering higher
education programs, the reports do not collect other output information
such as how many participants were placed in construction-related
employment.
^13GPRA; S. Rep. No. 58, 103rd Cong., 1st Sess. (1993); and OMB Circular
A-11.
Citing resource constraints and a need to comply with the Office of
Management and Budget (OMB) performance reporting requirements, HUD
contracted with YouthBuild USA in 2004 to collect quarterly grantee
performance information for all Youthbuild grantees. Programs administered
by six federal agencies, including HUD and Labor, are subject to "common
measures" reporting which OMB established to provide agencies with the
ability to describe, in a similar manner, their core performance outputs
across different grantee programs.^14 The common measures include
employment or education placements, degrees or certificates attained, and
literacy and numeracy skills increased (increases in educational
functioning). Consequently, YouthBuild USA provides HUD with program
information including participant gender and ethnicity data, GED
attainment statistics, grade level improvements, job or school placements,
and the number of new and rehabilitated housing units that were completed.
HUD uses the reports to provide performance information to Congress and
meet OMB reporting requirements. However, because the information is
captured on a quarterly basis, it only represents the performance of a
program at a particular point in time and YouthBuild USA has not
aggregated the information to obtain data for the term of individual
grants. As a result, HUD has been largely unable to track program
outcomes, but its data will be needed for ongoing program assessments.
Moreover, HUD has encountered significant delays in developing a database
intended to provide additional performance outcomes on grantee programs.
Recognizing this deficiency, HUD officials said that they have been
planning for about 3 years to implement an electronic data collection
system to help HUD measure the success of Youthbuild programs, along with
its other nonhomeless competitive grant programs. However, the system's
implementation has been delayed because of funding and technical problems.
As a result of not systematically collecting or analyzing needed
information from closeout reports, relying predominately on limited
point-in-time program information, and not being able to utilize a planned
database to aggregate performance outcomes, HUD is limiting its ability to
assess grantees. Further, although the YouthBuild Transfer Act shifted
oversight of grants issued in fiscal year 2007 to Labor, HUD will be
required to continue monitoring existing Youthbuild grants for
approximately the next 3 years. Therefore, the data HUD has at its
disposal or could develop will continue to be valuable for ongoing
assessments of the program.
^14In addition to HUD and Labor, the Departments of Education, Health, and
Human Services, Veterans Affairs, and Interior are subject to the common
measures.
Citing Resource Constraints, HUD Performed Limited Oversight of Grantees
In addition to not having or utilizing available performance data, HUD has
exercised limited on-site review and oversight of grantee performance.
According to HUD's Office of Inspector General, on-site monitoring is an
essential tool for HUD (through CPD) to assess program performance and
identify and address potential program problems. Such monitoring is also
useful in motivating grantees to exercise sound judgment in carrying out
their grant activities. GAO's Standards for Internal Control in the
Federal Government note that controls generally should be in place to
ensure that ongoing monitoring occurs during normal operations. Monitoring
should be performed continually, be ingrained in an entity's operations,
be used to assess the quality of performance over time, and support the
prompt resolution of any identified problems.^15
CPD goals require on-site monitoring of only a small percentage of grants.
For fiscal year 2003, the plan called for the field offices to monitor 35
percent of formula grants; 20 percent of competitive homeless grants; and
7 percent of competitive nonhomeless grants, including Youthbuild. HUD has
since changed its baseline monitoring goals (requiring monitoring of 20
percent of grantees with active, open grants established since 2004) to
give its field offices more flexibility to meet monitoring priorities
based on local grantee knowledge. As a result, goals are no longer set by
program. Field offices decide on the number of formula versus competitive
grantees to be monitored based on risk analyses that evaluate each grantee
on factors such as the size of the grant, perceived project management
capacity, grantee responsiveness, and the extent of citizen complaints.
HUD officials cited a lack of resources as the primary reason for not
requiring the monitoring of a larger percentage of grantees. In 2006,
according to HUD officials, field offices typically had one person working
part time responsible for overseeing the individual Youthbuild grantees in
the state, visiting no more than one or two grantees per year. As a
result, field staff have not been able to comprehensively monitor grantees
and verify the accuracy of the output data submitted by the grantees.
^15GAO issued these standards as required by 31 U.S.C. S3512(c). Also see
GAO, Standards for Internal Control in the Federal Government,
[26]GAO/AIMD-00-21 .3.1 (Washington, D.C.: November 1999) and Internal
Control Management and Evaluation Tool, [27]GAO-01- 1008G (Washington,
D.C.: August 2001).
In general, we found that the monitoring reports that we reviewed (which
HUD field offices produced) on grantees primarily addressed various
compliance issues, rather than reporting on the overall performance of a
particular grant. For example, several reports identified grantees'
failure to receive prior approval from HUD for changing proposed job
training construction sites. Other reports identified inadequate record
keeping, insufficient documentation, and lack of sufficient financial
internal controls on the part of grantees. However, although a few reports
did comment on some individual performance outputs, such as job placements
or GEDs attained, most did not.
The lack of analysis of performance information of grantees also has
implications for assessing the Youthbuild program overall, particularly
because HUD has devoted few staff to overall program management and
oversight. Since 2000, in headquarters, Youthbuild had a program director
with one or two staff managing the program each year. During fiscal year
2006, with the expected program transfer to Labor, the HUD program manager
was assigned to other duties, and a part-time program manager was assigned
to oversee the 2006 grant process, in which HUD received 308 applications
and awarded 74 grants. Although they are aware of the limited oversight of
Youthbuild, CPD division officials said it has provided appropriate staff
resources for program oversight of Youthbuild grantees, commensurate with
the program's size--the annual appropriation for which represents less
than 1 percent of CPD's usual annual budget of about $8 billion. However,
without sufficient oversight HUD was unable to tell how the individual
Youthbuild grantee programs performed.
Incomplete Data on Participants' Outcomes Precludes an Overall Assessment of
Program's Success
HUD has not performed the programwide analysis needed to determine overall
program success, and weaknesses in other assessments of the programs and
data preclude an overall assessment of Youthbuild success. While limited
in scope, three studies suggest Youthbuild helped youths achieve economic
self-sufficiency. And, although YouthBuild USA tracks performance data,
grantees do not have the follow-up information needed to measure
performance outcomes. To augment limited existing data, we analyzed
information in 245 closeout reports, but the results cannot be generalized
to the entire program. We found a wide range of results in outputs such as
job placements; however, because grantees generally did not follow up on
participants, we were unable to determine long-term outcomes such as
percentages of participants retaining employment. Moreover, while closeout
reports include impediments to success and "best practices," HUD did not
systematically review this information or share it with YouthBuild USA. As
a result, the lack of follow-up data and programwide evaluations make it
very difficult to determine which programs and strategies have worked best
and assess the performance of Youthbuild over time. Labor officials said
they would consider that the closeout reports include outcomes such as
jobs obtained and retained and said that they believe impediments and best
practices should be evaluated and disseminated to grantees.
A Few Studies Document Individual Youthbuild Program Successes but Are Limited
in Scope
A few studies have provided some evidence of the success of individual
Youthbuild programs. Although subject to many limitations, the studies
suggest that identified programs have achieved certain Youthbuild
objectives. However, without comparable information for the entire
population of Youthbuild grantees, it is not possible to determine the
extent to which these outcomes and reported successes are reflective of
what was achieved by the grantees not included in the studies. More
specifically:
o In June 2004, researchers from Brandeis and Temple Universities
published a study entitled, "Life after YouthBuild - 900
YouthBuild Graduates Reflect on Their Lives, Dreams, and
Experiences," in collaboration with YouthBuild USA. The study was
based on a survey of graduates from 11 Youthbuild grant sites and
included in-depth interviews with 57 graduates from eight of these
sites. The researchers determined that, for those 882 participants
(for a 23 percent response rate) who responded to the survey, many
are doing well and were helped considerably by the Youthbuild
program. Many participants found jobs or furthered their education
and have volunteered in community activities. In addition, a
number of participants surveyed indicated they also obtained
intangible benefits from the program, such as increased
self-esteem and had developed positive attitudes about themselves
and their community. A limited nonresponse analysis was conducted
by YouthBuild USA to compare the 882 survey responses and the 57
interviews with one site that had a 98 percent response rate.
However, using one site that had a high response rate is not
representative and cannot be generalized to the whole sample. As
such, it is not known if the 77 percent who did not respond were
doing more or less well than those who responded to the survey.
o In August 2003, HUD issued a study entitled, "Evaluation of the
Youthbuild Program," in which a HUD contractor compared program
results and costs per participant from the Youthbuild Program and
four other federal employment and training programs--Job Corps,
JOBSTART, both designed to serve disadvantage youth, and the youth
components of the Job Training Partnership Act and Supported Work
Demonstration programs.^16 The researchers conducted their work at
20 Youthbuild sites, selected in part because they had received
funding for at least two grants, obtained a variety of
organizational sponsorships, built new or rehabilitated housing,
and covered a variety of funding category sizes. The study found
that 36 percent of the Youthbuild participants at these sites
obtained jobs after enrolling and 29 percent obtained either a GED
or high-school diploma. In addition, 12 percent of the Youthbuild
graduates in the 20 sites reportedly went onto higher education.
However, results at the five federal programs were not directly
comparable because researchers obtained data from Youthbuild
program participants at the time they exited the program while
data from the other programs' were obtained from tracking
participants up to 3 years after leaving the program. In addition,
the limited scope of this study precludes an overall assessment of
the program's success.
o In February 2003, the state of Minnesota issued a study
entitled, "Minnesota Youthbuild Program: A Measurement of Costs
and Benefits to the State of Minnesota." This study analyzed the
costs and benefits of the state-funded Youthbuild program in
Minnesota. The researchers measured post-program participant
performance outcomes by tracking program year participants beyond
graduation. They found that Youthbuild graduates earned more than
double the minimum wage, whereas youth with similar
characteristics who had not been through the program typically
earned only the minimum wage. In addition, the graduates who had
been adjudicated had a considerably lower rate of recidivism
(rearrest, reconviction, or return to prison) than other youth
involved with the correctional system.^17 Using state
administrative data to estimate the amount of revenues and savings
generated through increased tax revenues and reduced prison costs,
the study estimated first-year benefits to the state of about $1.5
million dollars versus costs of about $877,000. Over a 4- year
period, the study estimated the total net benefit to the state at
about $7.3 million (by 2006). The study was limited to costs and
benefits directly related to state programs and did not include
potential benefits such as reductions to welfare rolls,
state-sponsored medical care, or other reductions in human
services costs.
YouthBuild USA WebSTA Data Have Limitations, Particularly for
Assessing Outcomes
Since 2002, YouthBuild USA has been using a Web-based
system--WebSTA--to collect participant characteristics and
performance data from its affiliates. According to YouthBuild USA
officials, as of December 2006, about 90 percent of its affiliates
were entering data into WebSTA to some extent. However, the value
of YouthBuild USA's WebSTA data is limited, largely because
grantees self-report the data, not all grantees are meeting the
requirement to send in information, and the information is not
reported at the grant level. That is, if affiliates have more than
one Youthbuild grant, the data for each grant are not reported
separately. As a result, the data are not verified, comprehensive,
or comparable. Also, because the grantees submit the information
quarterly, the data provide only a point-in-time snapshot of the
program's performance and cannot be summed to provide annual data.
Recognizing these limitations, according to YouthBuild USA's
President, upgrades are planned to be implemented in early 2007
that will allow grantees to enter data in WebSTA by individual
grant, which in turn will allow YouthBuild USA to analyze
grant-level performance.
We also found that the limited depth of the information collected
restricts the value of WebSTA information for performance
assessments. For example, WebSTA collects information on
participant attendance, retention and graduation rates, and
initial job and college placements and participant characteristics
(such as welfare, public housing, and adjudication status) when
participants enter Youthbuild programs. However, many grantees do
not follow up on participants once they leave the programs. As a
result, they often do not know if students placed in jobs were
still working at those jobs weeks or months later. Also, several
program directors we visited said that most of their program
graduates who go on to higher education sign up for free
remedial-level courses at nearby colleges. Although they have not
had the funding to systematically follow the students' progress,
some said they generally have found that the students have had
varying success in advancing to college-level credit courses.
Without grantee-level follow-up systems, WebSTA cannot track
program outcomes that would reflect whether the graduates are
achieving economic self-sufficiency--such as the percentage of
program graduates who remove themselves from the welfare rolls,
move out of public housing, or stay out of the court system. The
system also does not collect follow-up information on the type of
jobs attained (including construction), pay raises, benefits
received, and college or trade degrees or certifications received.
Analyzing such performance outcomes could provide, HUD, Labor, and
other interested parties with the measures needed to evaluate
overall program performance and success.
Closeout Reports We Analyzed Provided Little Detail on Long-Term
Outcomes Because Grantees Are Not Required to Follow up on Graduates
Because HUD had not aggregated or analyzed data for Youthbuild, we
obtained and analyzed 245 closeout reports that grantees submitted
to HUD. Figure 6 provides selected performance information that we
summarized from the closeout reports. However, because HUD has not
established objective performance goals for the program, which
would serve as baseline criteria against which to judge
performance, the information related to the experience of the
12,863 participants who entered the program under the 245 grants
gave us limited perspectives about how individual programs were
working. In addition, because we had information on only 245 (or
46 percent) of the grants, the results cannot be generalized to
the entire program.
As shown in figure 6 the percentages for participants graduating,
receiving a GED, or being placed in employment varied widely. As
previously discussed, HUD has not set benchmarks, and critical
data elements (baseline data) are missing in the closeout reports.
For instance, it is not possible to obtain an accurate number of
diplomas and GEDs attained from the closeout reports because the
reports do not contain data on the number of participants that
enter the program with a high-school diploma or GED. As result, we
could not determine how well the programs (except for those with
the very highest percentages) performed individually or in
comparison with each other.
^16The Job Training Partnership Act program was replaced by the Workforce
Investment Act, which was implemented in 2000. This evaluation included
data from 1996 through 1999 when the Job Training Partnership Act programs
were still in effect.
^17Minnesota Youthbuild program participants entered unsubsidized
employment with an average starting wage of $11.60 an hour. The minimum
wage of $5.15 an hour was used as a baseline for the comparison group.
Fourteen (5 percent) of the participants with one or more offenses prior
to enrollment were rearrested, reconvicted, or returned to a state
correctional facility within 2 years after enrollment in the Minnesota
Youthbuild program, compared with a 20 percent rate of return for
juveniles within 2 years of release from Minnesota correctional
facilities.
Figure 6: Based on Closeout Reports, Youthbuild Performance Measures
Indicate Varied Outputs
^aNot all grantees provided information for each category.
^bIncludes 1,239 participants that were reported as still active because
they were carried over to a subsequent grant or allowed to stay on the
books to receive continuing education and other support services. In
addition, the drop outs, graduates, and still active categories do not sum
to the number of participants category because all grantees did not report
information for each category.
^cRegulations allow up to 25 percent of the participants who enroll to
already have a GED or high-school diploma. As a result, this number may be
underrepresentative since the data do not specify the number of
participants that entered the program with a GED or high-school diploma.
^dThe same participants can be included in the employment and higher
education categories.
The closeout reports also do not provide consistent and reliable data on
the number of units rehabilitated or built by Youthbuild participants. A
basic purpose for which Youthbuild was authorized was to expand the supply
of affordable housing. According to the closeout reports, 158 grantees
reported that their participants built a total of 1,483 new units, while
139 grantees reported that their participants rehabilitated a total of
2,238 units.^18 Some of the Youthbuild program directors with whom we met
said that during a typical reporting period they would complete from one
to four new or gut-rehabilitated housing units. Other directors said that
when a program reports about 200 units rehabilitated, it generally means
the students are performing minor repetitive tasks such as changing locks
or painting a limited number of interior walls. As a result, the number of
units rehabilitated likely is overstated in the closeout reports. Although
construction training is a major part of the Youthbuild program, HUD's
closeout reports also do not track the number of graduates obtaining
construction-related jobs or specify the types of jobs obtained
(construction or otherwise).
In addition, HUD officials told us that grantees are not required to
follow up on participants after they leave the program. While some grantee
officials with whom we met recently had attempted to establish follow-up
systems, most grantees cited funding constraints as the reason for not
following up on graduates. Several officials further noted that it was
difficult to locate many participants after they had left the program.
However, as a result, it is not possible to obtain comprehensive
post-program results.
Finally, some program directors with whom we spoke cautioned that any
follow-up system should not be strictly focused on students obtaining
higher education degrees or obtaining high-paying jobs. They emphasized
that such goals were beyond the abilities of some of their participants,
but that moving from a fifth-grade to a ninth-grade reading level or
obtaining a service sector job sometimes was a real victory and should be
counted as a success.
Limited Information We Developed from Site Visits Pointed to Varied Results for
Programs, and Also Provided Little Detail on Long-Term Outcomes
The information we obtained from our visits also demonstrates varied
results for continued employment and academic success, and little detail
on long-term success. As with our closeout report analysis, the
information cannot be generalized to the universe of Youthbuild
grantees--although YouthBuild USA officials told us that they believe we
visited some of the more innovative grantees. For example, although not
required, some of the nine grantees we visited used other funding sources,
such as United Way or AmeriCorps funds, to establish a participant
follow-up system. As a result, some of the grantees were able to provide
us with varying amounts of information on the status of program graduates.
For example:
^18Some grantees reported that they used their program to both construct
new units as well as rehabilitated others under the same grant.
o One grantee, which was able to track individuals who completed
the program since 2001, provided us with data that showed 70 of
the 82 program participants were either employed or pursuing
higher education. Seven of the 70 participants were working and
going to school. The hourly wages of those employed ranged from $7
to $13.75. However, the grantee did not have information on the
types of employment.
o Another grantee that recently established a follow-up system
showed that 11 of the 26 program graduates who started in 2004 had
obtained employment. The positions ranged from pizza delivery
person to union carpenter. However, only the union carpenter and a
graduate hired by the program as a construction trainer were
working in the construction field. The hourly wages of the
graduates ranged from $5.50 to more than $16 per hour for a
graduate hired by the grantee.
o Another grantee used WEBSTA to track continuing education and
work history, including each job held. The program director told
us that such a follow-up system allows the program to spot trends,
which in turn can be used to fine-tune the program. Of 60
graduates from 2002 to 2005, the program continually has tracked
53 and reported that 8 are receiving higher education, 24 are
working in nonconstruction jobs, and 16 are working in
construction jobs. The pay for the graduates ranged from $6.50 to
$15 per hour. The grantee continues to keep in touch with the
graduates by helping them with housing, legal aid, school advice,
tutoring, and help toward achieving a GED.
Grantees are not required to collect such post-program follow-up
information. Yet, both grantees and program participants
emphasized that the Youthbuild program created a very supportive
atmosphere that many participants had never experienced prior to
joining a program. During our visits, we met with students who
credited the program with improving their lives by either helping
get them off drugs, get out of the gang lifestyle, or disengage
from other activities that likely could have resulted in
incarceration or worse. The studies we mentioned previously had
similar findings. However, it is difficult to gain insight into
how many program participants remain drug free, crime free, in
advanced degree programs, or working in construction or other
trades without post-program performance data. It is also difficult
to gain insight into which program strategies are most effective
for serving these youth.
HUD Does Not Review or Disseminate Impediments and Best Practices
Identified in Closeout Reports That Could Help Improve Program
Performance
Citing resource constraints, HUD did not systematically review the
impediments and best practices identified in closeout reports or
share the information in them with YouthBuild USA. In particular,
identifying best practices (or the converse, impediments) can
provide a model for improvement for other organizations with
similar missions. The closeout reports contain a section
addressing what the grantees considered to be best practices. For
example:
o Several grantees set-up educational opportunities, such as GED
preparation, at local colleges for participants. Two grantees
noted that being on a college campus provided a positive
experience for the participants. Two grantees noted that they made
an arrangement with a local technical college to offer free
enrollment to all participants that successfully completed the
Youthbuild program. Another grantee noted that working through
community college GED programs allowed them to lower their
per-student cost and serve more students.
o One grantee established an alumni council consisting of elected
representatives from each of its seven graduating classes. All
graduates receive quarterly mailings that include an alumni
newsletter, new job and educational opportunities, and a quarterly
calendar of events.
o Another grantee has maintained a program serving graduates for
up to 12 months to provide on-going employment support, including
assistance in securing a second job in the event that the initial
placement did not work out.
o One grantee started a "drug free" club and found that students
really wanted to participate and were anxious to become drug free
and would publicly announce their success.
o Another grantee worked extensively recruiting area college
students to volunteer to help with tutoring and program community
service activities.
Closeout reports also contain information on impediments that
grantees encountered in implementing and operating the programs.
Impediments to program success varied in scope and severity. Some
of the more significant impediments mentioned by numerous grantees
included the following.
o Turnover of key staff, especially on-site construction managers;
o Negative and apathetic attitudes among participants;
o Alcohol and drug abuse issues among participants;
o Difficulties in dealing with municipal organizations to obtain
building permits and contractual difficulties, which delayed
construction training activities; and
o Acceptance of students with very low educational levels, which
made obtaining high GED levels very difficult.
HUD's contract with YouthBuild USA specifically calls for the
contractor to assist HUD in the management, supervision, and
coordination of Youthbuild programs by strengthening Youthbuild
program design and disseminating information on best program
practices. Although analyzing the impediment and best practices
falls within the scope of HUD's contract with YouthBuild USA, two
CPD program officials said that HUD never considered sharing the
reports with its technical services provider. YouthBuild USA's
chief operating officer said that the organization would welcome
access to the closeout reports. The officer said his staff already
spent many hours each month trying to identify best practices from
Youthbuild grantees and would be anxious to analyze the closeout
reports for both the problems and successes identified and forward
the information to interested grantees. YouthBuild USA's President
told us that she not only agreed that reviewing the closeout
reports for problems and "best practices" would be beneficial, but
that she believes the closeout reports needed to "be analyzed to
get to the bottom line because the total impact (outcomes) of a
grant could not be determined from data provided on a rolling
quarterly basis."^19 As a result of not sharing the closeout
reports with YouthBuild USA, HUD may have missed opportunities to
improve the program as a whole and help grantees improve
performance.
Labor Officials Indicated They Will Consider Using Information
from Closeout Reports to Assess and Improve YouthBuild
Performance
According to Labor officials, they will consider analyzing
information from the closeout reports to assess and improve the
performance of the YouthBuild program. Until 2006, Labor assessed
past performance of its youth employment and training grantees
through measures required under WIA. For example, under the WIA
Youth and Job Corps programs, Labor tracked and analyzed 6-month
retention and earnings wage change information. Labor officials
told us that they would use OMB's common measures to track and
analyze the performance of YouthBuild grantees but said that they
have not finalized the information they would require YouthBuild
grantees to collect. The officials indicated to us that they would
consider developing post-program performance outcomes such as the
types of employment graduates attained and retained (including
construction), wage rates, and college or trade degrees or
certifications received. Although Labor may incur some additional
costs in developing such information, the ability to measure
program success would improve. Labor officials also noted that the
information currently contained in the closeout reports would be
considered with other factors they deem as pertinent to assessing
grantees' performance. However officials said they would also
likely consider modifying the closeout reports to include other
program participant data, such as degrees obtained prior to
program entry and details on the types of construction jobs
obtained upon program completion.
^19As we previously discussed, the quarterly data cannot be summed to
provide an annualized report.
Labor officials also acknowledged that information on program
impediments and best practices should be evaluated and passed to
grantees and that performance data needed to be aggregated,
verified, and analyzed to determine which programs and strategies
worked best. As of October 2006, Labor had not determined whether
its Employment and Training Administration or an outside
contractor would have primary responsibility for collecting,
aggregating, and disseminating the information.
Youthbuild Grantees Have Experienced Varied Success in Obtaining
Outside Funding Sources, and Many Cited Continued HUD Funding as
Crucial
Our analysis of available closeout reports showed that the
grantees have had varying success in obtaining additional funding
sources. In assessing grant applications, HUD gave consideration
to program applicants that were able to secure outside funding.
Applicants received points for leveraging nonhousing resources
compared with the amount of Youthbuild funds requested in the
application. Overall, the median amount spent by a HUD Youthbuild
grantee was $400,000 in HUD funds; the HUD grant represented about
48 percent of the average total spending per grantee, which
included additional funding sources of about $430,000. However,
success in obtaining additional funds varied widely, from 21
grants reporting no additional funding sources to 40 reporting
more than $1 million (see fig. 7).
Figure 7: Outside Funding Amounts Varied Widely for Grants We
Reviewed
According to information in the closeout reports and the officials
from the grantees that we visited, the additional funding they
obtained largely came from federal, state, and local agencies,
businesses, and charitable organizations. According to a
YouthBuild USA official, states have provided funding to local
Youthbuild grantees over the past several years. For instance, the
Massachusetts State Department of Education has provided funding
in its budget for the state's Youthbuild program. The
Massachusetts funds are provided to existing Youthbuild programs,
based on a formula the state developed in conjunction with a state
coalition of Youthbuild grantees. In Minnesota, 10 Youthbuild
programs in 2005 shared $754,000 in state Youthbuild program
funding, which the state supplemented with more than $3 million in
matching funds from local partners. In Minnesota as elsewhere,
urban grantees have had greater opportunities to obtain funding or
in-kind support from local businesses and charitable organizations
than rural grantees that typically have had access to fewer
businesses and might not be a focus of charitable activity.
Regardless of their success in obtaining outside funds, several
grantees we visited stated that their Youthbuild program could not
be sustained without the HUD grant because the HUD funding allowed
them to attract the leveraged funds. Also, several grant officials
said that many existing and potential contributors would not be
willing to continue funding if they perceived that the program
would not be able to continue without the HUD grant. One exception
(which we did not visit) was in Minnesota where the state
Youthbuild program director said that only 4 of the state's 10
programs have had HUD funds in the past, including 2 programs
which received HUD funding in 2004 through a HUD grant
administered by the state. The program director added that the
structure of the Minnesota state Youthbuild program is similar to
that of HUD's, although the programs are typically smaller, having
from 10 to 25 students.
Information from YouthBuild USA generally agrees with grantees'
contention about the need for continuous HUD funding. Overall,
YouthBuild USA found that 60 percent of programs that HUD did not
fund for 2 consecutive years ceased operations, and 90 percent of
programs not funded for 3 consecutive years ceased operations.
According to YouthBuild USA, 173 of 462 grantees funded since 1994
currently are operating with active HUD grants. YouthBuild USA,
also found that grantees with continuous HUD funding achieved
higher outputs, such as GED attainments and job placements, than
the ones that did not receive one or more follow-on grants.
According to some grantee officials, that success is partly due to
continuous funding that allows grantees to develop infrastructure
and maintain experienced staff to establish a better program. In
its fiscal year 2005 Performance and Accountability Report, HUD
noted the benefits of making grants to previous awardees: "Having
established programs and experience running a Youthbuild program,
these grantees are more efficient in enrolling students, resulting
in a greater number of youth trained." Furthermore, Youthbuild
program directors with whom we met emphasized the intangible
benefit that such established programs provide such as a family
atmosphere that allows graduates to return for advice and support
as needed--a benefit that goes away when programs close.
HUD has not kept statistics to find out why some grantees did not
reapply for additional grants or why grantees did not qualify for
follow-up grants. Through discussions with a nonprobability
selection of 23 of the 100 grantees that closed programs, we found
that some did not reapply because they were no longer interested
in dealing with at-risk youth and others closed due to poor
management. But most of the closed grantees with whom we spoke
said they closed because they lost HUD funding for one or more
years and were unable to obtain sufficient private funds to make
up for the lost funding. Of the sites we contacted, two were still
in operation after ceasing to receive HUD funds--but they appeared
to be the exception. Both were already operating youth technical
training programs prior to their first HUD Youthbuild grant.
In addition, officials from several grantees that also received
funding from AmeriCorps stated that they preferred the AmeriCorps
concept of providing dedicated funds for 3 years. These officials
stated that knowing that funding levels were longer-term provided
more continuity to their program, enabled better planning, and
allowed them to build a cushion through leveraged funding in the
event that they did not receive a HUD grant for a year or two.
Finally, when the question of multiyear funding was raised at a
Web-based listening session in December 2006, designed to help
Labor identify Youthbuild program administration issues and
concerns, most participating grantees spoke out in favor of
multiyear funding.
Conclusions
HUD has not taken advantage of several opportunities to use
existing information for purposes of program assessment and
oversight. Such opportunities should not be ignored, particularly
because HUD has limited resources with which to assess and oversee
current grantees. Specifically, HUD has not verified, aggregated
or analyzed closeout reports, which grantees are required to
submit. HUD also has conducted limited oversight of grantees,
citing resource constraints; the impending transfer of oversight
to Labor bodes no change to this specific situation. However,
given these conditions, analysis of existing closeout reports
would help provide crucial information on the performance of the
Youthbuild programs, particularly in the absence of a centralized
HUD database that could be used to track program performance. At a
minimum, the closeout reports form a logical basis for providing
useful information that HUD, Labor, and other interested parties
can use to judge the performance of Youthbuild. HUD still will
maintain an active involvement for several more years, so
coordination between Labor and HUD will be critical to producing
reliable assessments of how successful the program has been and
will be. HUD also does not share reported constraints and best
practices with the technical services contractor it pays for
performing such work and disseminating best practices. By not
utilizing performance data and sharing information, HUD has missed
opportunities to determine whether the overall Youthbuild program
has achieved its intended results, among them enabling
disadvantaged young adults to gain economic self-sufficiency.
Therefore, Congress and the public also lack the information
needed to make such determinations.
Labor has experience in managing youth training and education
programs and collecting performance information, and Congress
transferred Youthbuild to Labor in part to better align the
program with existing federal workforce development and youth
training programs. In offering perspectives for Labor to consider,
we stress the value that closeout reports have in the absence of
more comprehensive and programwide data, although the existing
reports lack certain elements that could help program managers
determine and report on outcomes. The reports do contain valuable
information on impediments and best practices, which could provide
both program managers and grantees with much broader perspectives
on how successful the program has been and suggestions for
improvements. Labor has indicated that it would consider
developing post-program performance outcomes such as the types of
jobs graduates attained and retained (including construction),
wage rates, and college or trade degrees or certifications
received. As a result, while there may be some additional costs
involved, the ability to measure program success would improve.
And, because alternatives to the HUD data are limited, the
grant-level assessment discussed above could act as the building
blocks of an overall program assessment. Labor also has agreed
that information on impediments and best practices would be
valuable and ought to be passed on to grantees, but has not yet
determined how this might be done.
Finally, data on grantee success in attracting other funding,
although limited, offer more perspectives for Labor to consider
because outside funding helps leverage limited federal dollars.
While many grantees have obtained extensive outside funding and
support, YouthBuild USA has found that very few operated for more
than a few years without continued HUD funding. Tellingly, their
data show that 90 percent of programs not funded by HUD for 3
consecutive years ceased operations. According to YouthBuild USA
officials, their data indicate that grantees with repeat awards
also produced better outcomes such as a higher level of job
placements. Further, when HUD funding stopped, potential
contributors became concerned that grantees would no longer stay
in business and were reluctant to jeopardize their own funds.
These data, while not definitive, do suggest that multiyear
funding has the potential to produce dividends for the program.
Many grantees expressed similar belief in the benefits of longer
term funding at a Department of Labor YouthBuild program listening
session.
Recommendations for Executive Action
To improve the reporting and assessment of performance for
Youthbuild grantee programs and develop the bases for an overall
assessment of the program, we are making the following three
recommendations:
o that the Secretary of HUD analyze closeout reports by grant and
share information on identified problems and "best practices" with
its technical services contractor and Labor;
o that the Secretary of Labor develop and monitor post-program
performance outcome measures for the YouthBuild program, such as
the types of employment graduates attained and retained, wage
rates, and degrees or certifications received, and share the data
with the grantees; and
o that the Secretary of Labor consider whether multiyear funding
could be useful in helping YouthBuild grantees attract additional
outside funding.
Agency Comments and Our Evaluation
We provided HUD and Labor with a draft of this report for review
and comment. HUD's comments are summarized below. Labor's
Assistant Secretary for Employment and Training provided written
comments that are presented in appendix I.
Officials from HUD's Office of Community and Planning and
Development provided comments. They stated that they did not
disagree with GAO's overall findings, but added that HUD's
oversight responsibility was limited by resource constraints. They
cited the fact that the Youthbuild program represents 0.63 percent
of the Community Planning and Development Division's entire
portfolio. Nevertheless, because HUD will have monitoring
responsibilities for existing grants for 3 years or more, we
believe it is important that HUD analyze the closeout reports in
order to provide Labor with information about how current grantees
are performing.
Labor's Assistant Secretary for Employment and Training wrote that
she believes that the report will be very useful to Labor as it
assumes responsibility for administering the YouthBuild program.
She noted that Labor agreed with our recommendation to develop and
monitor post program performance outcome measures for the
YouthBuild program and share these data with grantees. She added
that Labor was building a Web-based Management Information System
to collect and report on participant information. Finally, she
wrote that Labor agreed that there could be value on multiyear
funding and that the Department was considering such an approach
for existing grantees.
We are sending copies of this report to other interested
congressional committees and the Secretaries of Housing and Urban
Development and Labor. We will also make copies available to
others upon request. The report will also be available at no
charge on GAO's Web site at http://www.gao.gov .
If you or your staff have questions regarding this report, please
contact me at (202) 512-4325 or [email protected] . Contact
points for our Offices of Public and Congressional Affairs may be
found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix II.
William B. Shear
Director, Financial Markets and Community
Investment
Appendix I: Comments from the U.S. Department of Labor
Appendix II: GAO Contact and Staff Acknowledgments
GAO Contact
William B. Shear, (202) 512-4325, [email protected]
Staff Acknowledgments
In addition to the individual named above, Andy Finkel, Assistant
Director; Rich LaMore, John McGrail, Marc Molino, Luann Moy,
Barbara Roesmann, and Tom Taydus made key contributions to this
report.
GAO�s Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in
meeting its constitutional responsibilities and to help improve
the performance and accountability of the federal government for
the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at
no cost is through GAO's Web site ( www.gao.gov ). Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of
newly posted products every afternoon, go to www.gao.gov and
select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies
are $2 each. A check or money order should be made out to the
Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are
discounted 25 percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax:
(202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or
(202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW,
Room 7125 Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW,
Room 7149 Washington, D.C. 20548
(250279)
GAO's Mission
www.gao.gov/cgi-bin/getrpt?GAO-07-82 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact William B. Shear at (202) 512-4325 or
[email protected].
Highlights of [37]GAO-07-82 , a report to congressional committees
February 2007
YOUTHBUILD PROGRAM
Analysis of Outcome Data Needed to Determine Long-Term Benefits
Since 1993, the Department of Housing and Urban Development (HUD) has
provided funding for Youthbuild, a competitive grant program that trains
and educates disadvantaged youth and helps build low-income housing. In
2006, Youthbuild was transferred to the Department of Labor (Labor) to
better align the Youthbuild program with existing youth workforce and
training programs. In response to concerns about the overall quality of
Youthbuild, a Senate report directed GAO to assess the program. GAO's
objectives included (1) evaluating how HUD assessed and oversaw the
program, (2) determining what results the program achieved, and (3)
assessing how successful grantees were in obtaining outside funding. GAO
analyzed Youthbuild performance data, visited Youthbuild sites, and
interviewed agency officials.
[38]What GAO Recommends
GAO recommends that (1) HUD analyze closeout reports by grant and share
information on identified problems and best practices with its technical
assistance contractor and Labor, (2) Labor develop post-program
performance outcomes, and (3) Labor consider multiyear funding of grants
for the program. HUD and Labor provided comments on a draft of this
report. Labor generally agreed with the recommendations. HUD agreed but
cited resource constraints in implementing its recommendation.
While HUD requires grantees to report basic performance data, such as the
number of program participants and graduates and job placements, HUD has
not aggregated or analyzed the data and conducted limited oversight of
grantees. According to HUD officials, they did not have staff available to
analyze the closeout reports that grantees must submit, and a lack of
resources also limited oversight of grantees. The monitoring HUD did
primarily focused on compliance with program requirements such as
documentation rather than on performance. As a result, HUD largely was
unable to tell how the individual Youthbuild grantee programs performed.
Limited outcome data preclude any overall assessment of the performance of
the Youthbuild program; further, the few other analyses available such as
the one GAO did in this study to augment limited existing data cannot be
generalized programwide. GAO analyzed 245 closeout reports, representing
46 percent of the grantees or 12,863 participants. While GAO could
determine percentages of participants who received high school diplomas or
were placed in jobs, GAO could not determine outcomes over time, partly
because the reports lacked baseline information and grantees were not
required to and generally did not follow participants after graduation.
Further, while closeout reports include information about impediments to
program success and "best practices," HUD did not systematically review
this information or share it with its primary technical assistance
contractor. Consequently, the lack of programwide evaluations, follow-up
data, and dissemination of best practices make it very difficult to assess
the performance of Youthbuild over time and determine which programs and
strategies have worked best. Reporting on post-program performance
outcomes, such as the number of participants placed and retained in
construction-related employment, could increase the value of the closeout
reports and better measure program results. Labor officials indicated that
they would consider including such measures for program reporting.
Grantees had varying success in obtaining funds from outside sources, but
YouthBuild USA data suggest that continued (multiyear) HUD funding was
critical to sustaining grantee operations and attracting leveraged funds.
Grantees' success in obtaining additional funds varied widely, from 21
grants reporting no additional funding sources to 40 reporting more than
$1 million. While most grantees have generated outside funding, YouthBuild
USA reported that most grantees have had difficulty continuing operations
without continued HUD funding. Their data show that 90 percent of grantees
ceased operations if not funded for 3 consecutive years by HUD. Further,
YouthBuild USA also noted that grantees with follow-on HUD funding
achieved better performance outcomes, such as higher rates of job
placements, than grantees that did not receive subsequent HUD funding.
References
Visible links
26. http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1
27. http://www.gao.gov/cgi-bin/getrpt?GAO-01-1008G
37. http://www.gao.gov/cgi-bin/getrpt?GAO-07-82
*** End of document. ***