Head Start: Challenges in Monitoring Program Quality and Demonstrating
Results (Letter Report, 06/30/1998, GAO/HEHS-98-186).
Head Start, one of the most popular federal early childhood programs,
has delivered comprehensive services to about 16 million low-income
preschoolers during the past 33 years. Since its inception, Head Start
costs have totaled $35 billion, and its annual funding has increased
substantially in recent years. The administration recently proposed a
significant funding increase for Head Start to expand the program's
annual enrollment to one million children by 2002. At the same time,
Congress, the executive branch, and taxpayers have become more concerned
about ensuring the accountability of federal programs. This report
evaluates how the Department of Health and Human Services ensures that
Head Start programs are held accountable for complying with laws and
regulations and for achieving program results. GAO discusses (1) the
extent to which Head Start's mission, goals, and objectives provide an
overall framework that emphasizes compliance with applicable laws and
regulations and achievement of program results; (2) how well Head
Start's processes ensure compliance with applicable laws and
regulations; and (3) how well Head Start's processes ensure the ability
to determine whether the program's purposes have been achieved. GAO
summarized this report in testimony before Congress; see: Head Start:
Challenges Faced in Demonstrating Program Results and Responding to
Societal Changes, by Carlotta C. Joyner, Director of Education and
Employment Issues, before the Subcommittee on Early Childhood, Youth and
Families, House Committee on Education and the Workforce.
GAO/T-HEHS-98-183, June 9 (13 pages).
--------------------------- Indexing Terms -----------------------------
REPORTNUM: HEHS-98-186
TITLE: Head Start: Challenges in Monitoring Program Quality and
Demonstrating Results
DATE: 06/30/1998
SUBJECT: Performance measures
Preschoolers
Disadvantaged persons
Accountability
Preschool education
Education program evaluation
Aid for education
Comparative analysis
Surveys
IDENTIFIER: Head Start Program
HHS Family and Child Experiences Survey
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
GAO/HEHS-98-186
Cover
================================================================ COVER
Report to Congressional Requesters
June 1998
HEAD START - CHALLENGES IN
MONITORING PROGRAM QUALITY AND
DEMONSTRATING RESULTS
GAO/HEHS-98-186
Head Start Accountability
(104872)
Abbreviations
=============================================================== ABBREV
ACF - Administration for Children and Families
ECLS - Early Childhood Longitudinal Study
FACES - Family and Child Experiences Survey
HHS - Department of Health and Human Services
NCES - National Center for Education Statistics
NHES - National Household Education Survey
OSPRI - on-site program review instrument
Letter
=============================================================== LETTER
B-278560
June 30, 1998
Congressional Requesters
Head Start, one of the most popular federal early childhood programs,
has provided funding for a comprehensive set of services to about 16
million low-income preschool children in the past 33 years.
Administered by the Department of Health and Human Services' (HHS)
Administration for Children and Families (ACF), the program has long
enjoyed both congressional and public support. Since its inception,
Head Start costs have totaled $35 billion, and its annual funding has
increased substantially in recent years. Between fiscal years 1990
and 1998, annual Head Start funding nearly tripled from $1.5 billion
to almost $4.4 billion. The administration recently proposed a
significant funding increase for Head Start to expand the program's
annual enrollment to one million children by 2002. Meanwhile, the
Congress, executive branch, and taxpayers have become more concerned
about ensuring the accountability of federal programs in not only
complying with laws and regulations, but also achieving desired
results.
Given the significant financial investment in Head Start as well as
its increased funding in recent years and the interest in holding
federal programs accountable for their performance, you asked us to
evaluate how HHS ensures that Head Start programs are accountable for
complying with laws and regulations and for achieving program
purposes. "Achieving program purposes" refers to whether desired
outcomes have been achieved and whether Head Start participation has
caused differences in outcomes. Specifically, we addressed
-- the extent to which Head Start's mission, goal, and objectives
provide an overall framework that emphasizes compliance with
applicable laws and regulations and achievement of program
results;
-- how well Head Start's processes ensure compliance with
applicable laws and regulations; and
-- how well Head Start's processes ensure the ability to determine
whether program purposes have been achieved.
We analyzed the Head Start program's mission statement, strategic
goal, program objectives, and regulations. In addition, we examined
monitoring processes, systems, and tools used to ensure
accountability at (1) national headquarters, (2) regional offices
that administer regular Head Start programs at the local level, and
(3) regular Head Start local service providers.\1 We interviewed
federal headquarters officials who oversee and administer the Head
Start program, including those in HHS' ACF and its Head Start Bureau,
Division of Grants Policy, Division of Payment Management, and Office
of Inspector General. We also visited 6 of the 10 ACF regional
offices to (1) interview regional officials who administer and
oversee the Head Start program and, (2) at 5 regional offices, review
selected program files of grantees being monitored because of poor
performance. In addition, we interviewed selected local Head Start
program directors and staff to identify tools, processes, and systems
used to ensure accountability in their local programs and monitor
performance. We did our work between February 1997 and May 1998 in
accordance with generally accepted government auditing standards.
--------------------
\1 The regular Head Start program serves children and families in the
50 states, the District of Columbia, Puerto Rico, and the trust
territories. About 85 percent of Head Start children are served
through the regular Head Start program. Head Start also operates
programs for migrant and Native American populations. Grantees and
their delegate agencies are the local service providers.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
Head Start's mission, goal, and objectives provide an overall
performance assessment framework that emphasizes compliance with
applicable laws and regulations and achievement of program results.
Head Start developed this framework, which reinforces the program's
accountability by linking specific program activities to its overall
strategic mission and goal, in response to legislative requirements,
such as the Results Act, as well as Head Start Bureau policies. By
specifying measurable program performance objectives, Head Start has
the ability to answer questions about its compliance with regulations
and whether it is achieving its purposes. This could help to answer
the critical question of whether the program is having an impact,
that is, making a difference in participants' lives.
Although HHS has processes in place to ensure that grantees comply
with regulations, the implementation of these processes could be
improved, according to our review. For example, HHS periodically
conducts comprehensive on-site inspections of local Head Start
agencies using a process that, if properly implemented, can ensure
regulatory compliance. Both HHS' and our reviews, however, have
identified concerns about the consistency of these inspections due to
differences in reviewers' assessments of whether grantees are
complying with some requirements and due to other factors. In
essence, different reviewers may interpret inspection results
differently. These different conclusions, in turn, may affect the
decisions made about actions to take on the basis of inspection
results.
In recent years, HHS has substantially strengthened its emphasis on
determining whether it has achieved program purposes. Its processes
provide too little information, however, about how well the program
is achieving its intended purposes. HHS has new initiatives that, in
the next few years, will provide information not previously available
on program outcomes, such as gains made by participating children and
their families. This information--from a survey of a nationally
representative sample of families--will describe, for example, the
extent to which participating children have improved their
vocabulary, literacy, and social skills as well as the extent to
which families have become economically and socially self-sufficient.
In the future HHS will collect such data from all Head Start programs
rather than from just a sample of programs, according to agency
officials, but it has no established plan or schedule for doing so.
Until HHS takes this step, it will continue to hold local Head Start
programs accountable only for complying with regulations--not for
demonstrating progress in achieving program purposes. Moreover,
although HHS' survey will allow Head Start to show whether children
and their families have progressed in achieving program purposes,
HHS' planned analysis of survey results will not allow it to
determine with certainty that Head Start participation caused
children's or their families' improvements. Instead of comparing
survey results with those from a group of children and families
similar in all respects except for their Head Start participation,
HHS will compare results with other groups. This approach will not
allow HHS to isolate Head Start participation as a causal factor in
children's and families' progress. Therefore, HHS will not be able
to determine program impact, that is, whether the program is making a
difference in children's and families' lives.
BACKGROUND
------------------------------------------------------------ Letter :2
Head Start was created in 1965 as part of President Johnson's War on
Poverty. The program provides comprehensive services, such as
educational, medical, nutritional, mental health, dental, and social
services, to low-income children and their families in all 50 states,
the District of Columbia, Puerto Rico, and the U.S. territories.
Head Start also provides services for migrant and Native American
populations. Unlike some other federal social service programs that
are funded through the states, HHS awards Head Start grants directly
to local agencies, called grantees, which deliver program services.
Grantees numbered about 1,456 in fiscal year 1997. Grantees may
contract with other organizations, called delegate agencies, to run
all or a part of their programs. Grantees had such contracts with
about 517 delegate agencies in fiscal year 1997; four grantees had
contracts with 175 of these 517 delegate agencies. Grantees and
delegate agencies include public and private school systems,
community action agencies and other private nonprofit organizations,
local governments, and Indian tribes.
HHS distributes funds to grantees on the basis of, among other
things, the amount of funds a particular grantee received in
preceding years and a proportion of the amount of additional Head
Start funds available. About 60 percent of all regular Head Start
grantees have participated in the program for over 25 years. Once
approved for funding through a competitive application process,
grantees no longer compete for funding in future years. Grantees
must, however, submit applications to continue receiving funds for
their programs. Grantees must generally pay at least 20 percent of
total program costs with nonfederal funds. State or local
governments or private sources often provide such funds in the form
of cash or in-kind contributions such as building space or equipment.
ACF administers the Head Start program through the Head Start Bureau
and ACF's regional offices nationwide.\2 The Head Start Bureau
develops program policy, goals, and objectives for the program and
compiles reports on the program for the Congress and the public.
ACF's regional offices implement the Head Start Bureau's policies as
well as administer and oversee local Head Start agencies, which
includes assessing local agencies' compliance with applicable laws
and regulations.
Since its inception, Head Start has had long-standing and widespread
support. In recent years, however, HHS' Inspector General and we
have expressed concerns about program operations and effectiveness,
and the Congress has acted to strengthen program accountability. For
example, the Inspector General issued a series of reports in the
early 1990s seeking to improve, among other things, grantees'
financial management and HHS' program compliance capabilities. In
addition, we have issued four reports in as many years on Head Start.
One report, which addressed grantees' views on barriers to providing
services, found an insufficient number of qualified grantee staff to
meet the complex needs of children and their families.\3 Grantees
were also having trouble finding suitable facilities they could
afford, according to our review. Our second report found that Head
Start centers often provide a broader range of services to children
and their families than do other early childhood centers; however,
the quality of Head Start services has been uneven.\4 According to
our third report, research that had been conducted was insufficient
for determining whether Head Start has had a positive impact on
participants' lives.\5 That report recommended that HHS include in
its research plans an assessment of the impact of regular Head Start
programs. HHS responded to this recommendation by stating that it
would evaluate the feasibility of conducting impact studies. Our
most recent report described participant characteristics, services,
and funding.\6 We concluded, among other things, that the 33-year-old
program is at a crossroads because the context in which it operates
today differs greatly from that of 33 years ago when it was
established. For example, Head Start's traditional part-day,
part-year programs may not meet the needs of today's working
families, and more providers are offering services today than when
Head Start was established.
The Congress has recently acted to improve Head Start's and other
federal programs' accountability. In passing the Government
Performance and Results Act of 1993 (Results Act), the Congress
sought to shift the focus of federal management from staffing,
activity levels, and tasks completed toward results. The Results Act
requires federal agencies to develop (1) strategic plans, (2) annual
performance plans that establish goals and measures, and (3)
performance reports for the Congress detailing progress made in
meeting annual performance goals. In addition, in reauthorizing the
Head Start program, the Head Start Act Amendments of 1994 required
HHS to develop specific performance measures for the program to
assess the achievement of desired outcomes. The Amendments also
added a regulatory enforcement mechanism to the program. Grantees
that do not meet time requirements for complying with program
regulations now face termination of their funding. About 90 grantees
have either relinquished their grants or been terminated since 1994.
Although the majority of these grantees voluntarily relinquished
their funding, ACF officials told us that they can often convince
grantees to relinquish funding rather than face termination. To help
HHS carry out its oversight responsibilities, the 1994 Amendments
also required the agency, beginning in 1995, to provide the Congress
with annual reports on grantees' compliance with program regulations,
including information on grantee efforts to correct program
deficiencies. ACF plans to submit the first of these reports to the
Congress in June 1998. ACF officials told us that workload demands
prevented them from submitting these reports on time.
--------------------
\2 Other HHS offices with responsibilities affecting the Head Start
program include the Division of Payment Management, which disburses
grant funds to grantees for HHS and other federal agencies, and the
Division of Grants Policy, which develops policy and guidance for all
discretionary grant programs administered by ACF.
\3 Early Childhood Programs: Local Perspectives on Barriers to
Providing Head Start Services (GAO/HEHS-95-8, Dec. 21, 1994).
\4 Early Childhood Centers: Services to Prepare Children for School
Often Limited (GAO/HEHS-95-21, Mar. 21, 1995).
\5 Head Start: Research Provides Little Information on Impact of
Current Program (GAO/HEHS-97-59, Apr. 15, 1997).
\6 Head Start Programs: Participant Characteristics, Services, and
Funding (GAO/HEHS-98-65, Mar. 31, 1998).
HEAD START'S STATED MISSION,
GOAL, AND OBJECTIVES PROVIDE A
FRAMEWORK FOR ACCOUNTABILITY
------------------------------------------------------------ Letter :3
The Head Start Bureau has developed a performance assessment
framework that reinforces program accountability by clearly linking
the daily activities of local Head Start grantees to the program's
overall strategic mission and goal. This framework not only
emphasizes the importance of statutory and regulatory compliance, but
also the importance of achieving demonstrable outcomes. Head Start
developed this framework in response to congressional mandates and as
a result of its own efforts. In particular, the Results Act and the
Head Start Act Amendments of 1994 require the program to develop
methods to measure its progress in meeting overall program purposes.
In addition, the program's Advisory Committee on Head Start Quality
and Expansion\7 recommended that a performance measurement system be
developed to ensure that local Head Start grantees provide quality
services to children and families. The Committee proposed that such
a system include (1) identifying measurable outcomes, (2) selecting
performance indicators and collecting indicator data, and (3)
analyzing collected data. Although Head Start's framework is
relatively new and largely untested, it is a major step toward being
able to systematically demonstrate that local grantees comply with
laws and regulations and that Head Start programs are collectively
working toward achieving national program goals. It also improves
Head Start's ability to measure the overall impact of the program on
children's lives.
--------------------
\7 The Secretary of HHS created the Committee in June 1993 to review
the Head Start program and make recommendations for improvement and
expansion.
HEAD START'S MISSION, GOAL,
AND OBJECTIVES FOCUS PROGRAM
ACTIVITIES AND SERVICES ON
RESULTS
---------------------------------------------------------- Letter :3.1
Head Start's strategic mission and goal are based on a philosophy
that emphasizes the benefits of a comprehensive interdisciplinary
program. Head Start enrollees receive educational, medical,
nutritional, mental health, dental, and social services. Especially
during Head Start's early years, the program provided services that
participants probably would not have otherwise received. The
program's mission and goal provide the context for the delivery of
these services. Head Start's mission "to help prepare young
economically disadvantaged children for success in school and life"
provides the purpose for delivering these services; its goal to
"improv[e] children's social competence" addresses the services'
intended results. Head Start's strategic mission and goal reflect
ACF's strategic goal of promoting the "healthy development, safety
and well-being of children and youth." In turn, ACF's strategic goal
reflects HHS' strategic goal of "improving the economic and social
well-being of individuals, families and communities in the United
States."
To guide program efforts toward meeting their purposes, the Head
Start Bureau developed five performance-based objectives that reflect
the different areas (such as cognitive, emotional, and social
development; physical and mental health; and nutritional needs) that
it believes make up social competence. The Bureau plans on using
these five objectives to help assess, over time, its programs'
quality, effectiveness, and results. As envisioned by the Results
Act, Head Start's objectives shift the focus from the conduct of
program activities toward a broader focus aimed at achieving results.
Specifically, two of Head Start's five objectives focus on outcomes
and are included in ACF's annual performance plan as annual
performance goals. The remaining three objectives focus on program
activities that the agency believes are critical to achieving its two
outcome objectives. Figure 1 shows Head Start's five program
objectives. It reflects the Bureau's belief that its five objectives
form the foundation for improving children's social competence. The
Bureau's two outcome objectives appear at the top of the pyramid.
The Bureau's three objectives that focus on program processes or
activities appear at the base of the pyramid.
Figure 1: Head Start Program's
Strategic Goal and Objectives
(See figure in printed
edition.)
Source: Head Start Bureau, ACF, HHS.
HEAD START'S PERFORMANCE
MEASURES DESIGNED TO TRACK
PROGRESS IN ACHIEVING
PROGRAM OBJECTIVES
---------------------------------------------------------- Letter :3.2
In developing the five program objectives, the Bureau sought to unify
and organize performance measures that it developed to assess the
quality and effectiveness of local grantees' Head Start programs.
The 1994 Head Start Act Amendments require HHS to develop measures to
assess local grantee program services and administrative and
financial management practices annually and over longer periods. The
measures are intended to help identify program strengths and
weaknesses at both the regional and national levels. In addition,
the Amendments directed the performance measures to be designed so
that grantees could use them for performing self-assessments and so
that others could use them for conducting peer reviews of local
programs.
Head Start developed a total of 24 measures to comply with the 1994
Amendments. The measures are designed to gauge the quality of
health, educational, and nutritional services provided to enrolled
children and assess local grantees' ability to manage their programs.
Each of the 24 measures reflects one of the five program objectives.
Like the objectives, the performance measures focus on either
outcomes or processes and activities--outcome measures reflect
outcome objectives, while process measures reflect process and
activity objectives. For example, the Bureau developed three
performance measures to assess progress toward its outcome objective
of "strengthen[ing] families as the primary nurturers of their
children." One of these performance measures is determining whether
"Head Start parents demonstrate improved parenting skills." One of
the Bureau's four performance measures for demonstrating progress in
meeting the objective of linking children and families to needed
community services is determining whether enrolled children's parents
have secured child care so that these parents can pursue employment,
education, or job training. In this case, both the objective and
performance measure focus on local grantees' program activities.
Table 1 shows the Bureau's 24 performance measures and their related
program objectives.
Table 1
Head Start Program Performance Measures
by Program Objective
Program objective Performance measures
------------------ --------------------------------------
Enhance children's Head Start children demonstrate
growth and improved
development 1. emergent literacy, numeracy, and
language skills
2. general cognitive skills
3. gross and fine motor skills
4. positive attitudes toward learning
5. social behavior and emotional well-
being
6. physical health
Strengthen Head Start parents
families as the 7. demonstrate improved parenting
primary nurturers skills
of their children 8. improve their self-concept and
emotional well-being
9. make progress toward their
educational, literacy, and employment
goals
Provide children Head Start
with educational, 10. programs provide developmentally
health, and appropriate educational environments
nutritional 11. staff interact with children in a
services skilled and sensitive manner
12. programs support and respect
children's cultures
13. children receive needed medical,
dental, and mental health services
14. children receive meals and snacks
that meet their daily nutritional
needs
15. programs provide individualized
services for children with
disabilities
Link children and Head Start parents
families to needed 16. link with social service agencies
community services to obtain needed services
17. link with educational agencies to
obtain needed services
18. link with health care services to
obtain needed care
19. secure child care in order to
work, go to school, or gain job
training
Ensure well- Head Start
managed programs 20. programs are well managed
that involve 21. parents are involved actively in
parents in decisions about program operations
decision-making 22. programs employ qualified staff
23. programs support staff development
and training
24. programs comply with Head Start
regulations
----------------------------------------------------------
For each of the 24 measures, the Bureau has identified indicators and
data sources. As used in Head Start's performance assessment
framework, indicators are statements that relate how data will be
used in relation to a particular performance measure. The Bureau has
developed from one to seven indicators for each measure. For
example, the Bureau uses three indicators to determine whether
participating children's parents have secured child care so that
these parents can pursue employment, education, or job training. The
indicators are the (1) number and percentage of programs that provide
child care, (2) number and percentage of programs that provide
linkages to child care, and (3) number and percentage of program
children's parents that report having stable child care services. To
obtain data for each indicator, the Bureau plans on relying on its
management information systems or initiating new data collection
efforts. In some cases, the Bureau will collect data from each Head
Start grantee; in other cases, the Bureau will use only a sample of
grantees to obtain needed information.
LOCAL HEAD START AGENCIES
MUST COMPLY WITH PROGRAM
REGULATIONS
---------------------------------------------------------- Letter :3.3
Performance standards (hereafter referred to as program regulations)
are the regulations that define local program activities; grantees
must adhere to these regulations in operating their programs.
According to the Bureau, the regulations consistently define the
quality of services that grantees must deliver and constitute the
single most important statement of the expectations and requirements
that grantees must meet. HHS' program regulations generally reflect
Head Start's performance measures, according to our review.
HHS ensures local program quality by monitoring and enforcing
compliance with these regulations, Head Start officials said.
According to HHS, grantee compliance with the agency's program
regulations forms the foundation for obtaining positive program
outcomes. The Bureau plans on demonstrating its progress in meeting
its objectives by analyzing, among other things, compliance
information it gathers during on-site inspections of local Head Start
grantees and by conducting national surveys of Head Start children
and their families.
IMPLEMENTATION OF SYSTEM FOR
ENSURING REGULATORY COMPLIANCE
COULD BE IMPROVED
------------------------------------------------------------ Letter :4
Head Start uses several processes to assess and enforce local Head
Start agencies' compliance with program regulations. On-site
inspections are the main enforcement mechanism. The law requires HHS
to conduct such an inspection after a grantee's initial operating
year and at least once during each 3-year period thereafter.
Conducting on-site inspections not only allows HHS to observe local
program operations first hand, but also allows it to collect other
data by examining grantee records such as children's immunization
schedules. HHS has tailored an approach to conducting inspections
that is intended to result in a comprehensive review of local program
operations; concerns have been raised, however, about the consistency
of its implementation. Inspections may not be conducted consistently
among regions and by inspectors due to the subjectivity involved in
conducting inspections and the inspectors' levels of expertise,
according to our review. HHS has tried to improve the consistency of
its inspections, but the impact of these efforts is still unclear.
ON-SITE INSPECTIONS ARE
DESIGNED TO COMPREHENSIVELY
ASSESS GRANTEE COMPLIANCE
WITH PROGRAM REGULATIONS
---------------------------------------------------------- Letter :4.1
To conduct on-site inspections, HHS uses Head Start program
specialists from ACF regional offices located nationwide. These
staff, in addition to peer reviewers, including local Head Start
agency directors and consultants with particular expertise, inspect
Head Start programs operating in their respective regional office's
jurisdiction. The use of expert reviewers who have different
specialties, such as education or nutrition, ensures that
knowledgeable individuals are collecting data on the different Head
Start program components. On-site inspections generally take 1 week
to complete and conclude with a meeting among inspection team members
and grantee managers and staff to discuss the team's observations and
findings.
Team members use a specially designed on-site program review
instrument (OSPRI) to conduct inspections. The instrument helps
ensure that team members conduct a comprehensive assessment of
grantees' operations. It reflects Head Start's program objectives
and covers such program components as education, health and social
services, program administration and finances, parental involvement,
and facilities. Within each of these components, the instrument
lists review items, or criteria, that team members use to assess
grantees' operations. Team members must assess grantee operations
using a total of 256 criteria covering all program areas.\8 Team
members record their individual judgment on the OSPRI about whether a
grantee has met each criterion; they must judge grantees as either
"in compliance" or "not in compliance" for each criterion. Table 2
lists program components on the OSPRI and selected related criteria
for assessing compliance.
Table 2
Program Components on the OSPRI and
Selected Related Criteria
Component Criteria
------------------ --------------------------------------
Education There is a supportive social and
emotional climate.
Health A health services advisory committee
helps plan, operate, and evaluate the
health services program.
Mental health A mental health professional is
available to the program and to
children.
Nutrition There is a written nutrition plan,
annually updated.
Social services There are established procedures for
emergency assistance and crisis
intervention.
Parent involvement Parent training/orientation is
provided to prevent child abuse and
neglect and to protect abused and
neglected children.
Disabilities Resources to implement the
services disabilities services plan are
adequate.
Eligibility and The program maintains a waiting list
recruitment that ranks children by the program's
selection criteria to assure children
enter the program as vacancies occur.
Administration The grantee has and implements a
written procedure that assures
specific program objectives and
activities are completed in a timely
manner.
Staffing The grantee adequately supervises its
requirements and staff.
program options
Financial and The grantee has written accounting
property procedures.
management
----------------------------------------------------------
On the basis of team members' individual judgments, the Head Start
program specialist must decide whether a grantee is in compliance
with program regulations. If the specialist decides that a grantee
is not in compliance, then a judgment on the severity of
noncompliance must also be made. Under program guidelines, a grantee
that is not in compliance with regulations must be classified as
either (1) noncompliant or (2) deficient. Noncompliant grantees are
those that have not complied with regulations in one or more program
components. For example, a grantee may be classified noncompliant
with the health component because children's dental or medical
records are incomplete. Deficient grantees are those that have not
complied with regulations and whose failure to comply impedes the
grantee's ability to provide quality services to children and
families. For example, a grantee may be classified deficient if its
program has a staffing problem that affects the education, social,
and health services provided to children.
A noncompliant grantee must correct identified problems within 90
days and notify its ACF regional office that it has done so. A
grantee classified as deficient, however, faces far more severe
consequences than a grantee classified as noncompliant. A deficient
grantee must prepare a quality improvement plan describing how it
will correct all identified deficiencies and submit it to the
respective ACF regional office for approval. The grantee has up to 1
year to correct identified deficiencies, during which HHS makes
training and technical assistance available to the grantee to help
correct deficiencies. If the grantee does not correct all
deficiencies within 1 year, the ACF regional office must terminate
the program. Regional ACF staff monitor grantees to ensure that they
take corrective action.
ACF staff use a national database called the Head Start Management
Tracking System to help them monitor grantee compliance. Head Start
regional office staff enter information gathered from their on-site
inspections into the database, including the compliance status of
grantees regarding the 256 criteria used to conduct the inspections.
--------------------
\8 ACF is revising the version of the OSPRI that was in use during
our review. The revised version will be based on the new performance
standards, which became effective in Jan. 1998. ACF expects to
complete its OSPRI revision by fiscal year 1999; until then, ACF is
using an interim version. Criteria used to assess fiscal compliance
will not change from the original version of the OSPRI, according to
ACF officials.
ON-SITE INSPECTIONS COULD BE
MORE CONSISTENT
---------------------------------------------------------- Letter :4.2
Both ACF regional staff and outside researchers have raised concerns
about the consistency of on-site inspections. In 1993, a study
prepared under contract for ACF noted wide variation among regions in
the number of the OSPRI items for which grantees were judged as out
of compliance.\9 The researchers concluded that this wide variation
was probably not caused solely by differences in grantees'
activities, but also stemmed at least partially from differences in
the way regional office staff judged compliance. A study in 1996 by
the same contractor also identified ensuring consistency in
interpreting inspection results as a major challenge for Head
Start.\10 In our interviews with ACF regional staff, they also
expressed concern that on-site inspections may not be conducted
consistently among regions and by reviewers.
Several factors probably contribute to this inconsistency. First,
some of the variation may derive from the OSPRI. For example, the
OSPRI includes somewhat subjective items, such as "the parent
involvement program adequately provides methods and opportunities for
involving parents in experiences in child growth and development" and
"the grantee provides adequate supervision of its staff" as well as
more objective questions such as "completion of all recommended
immunizations." Second, the lack of guidance on the relative
importance of specific OSPRI items may further obscure the inspection
process. Third, some of the variation may result from differences in
inspectors' expertise. In the 1996 study, some focus group
participants stated that inspectors without the necessary expertise
are sometimes selected. Other factors, such as the time and work
necessary to monitor (and, if necessary, terminate) poor performers,
may discourage inspectors from classifying grantees as deficient.
According to ACF officials, the inspection process includes controls
to help ensure consistent inspections and ACF has taken several steps
to improve the process. For example, to help lessen the subjectivity
of judgments about compliance, part of the inspection process
includes team meetings during which inspectors can discuss their
opinions about grantee performance. In addition, training is
provided to inspectors before they conduct an on-site inspection, and
ACF sometimes uses consultants with particular expertise. ACF also
hosts annual conferences where regional ACF staff can share best
practices, identify training needs, and discuss other ways to
increase the consistency of inspection teams' judgments. ACF is also
trying to clarify priorities regarding grantee compliance in its
revision of the OSPRI. Instead of concentrating on assessing
agencies' compliance with individual program regulations, ACF is
adopting an approach in which inspectors would focus on reviewing the
systems a grantee uses to implement regulations. Inspectors would
then assess compliance with individual regulations to the extent that
inspectors find problems using this systems approach. The impact of
these efforts, however, is still unclear.
Although the full impact of the inconsistency of on-site inspections
is unknown, inconsistent inspections could lead to uneven treatment
of grantees. In addition, ACF officials told us that they use data
from the Head Start Management Tracking System--which includes
information from on-site inspections--to help make program decisions
and prepare congressionally mandated reports. Use of data from the
tracking system, therefore, may not be as valuable as it could be to
the Congress and ACF in making decisions about Head Start program
policy.
--------------------
\9 Indices of Head Start Program Quality, Pelavin Associates, Inc.
(Washington, D.C.: Sept. 1993).
\10 Revision of the Head Start Federal Monitoring System, Pelavin
Research Institute (Washington, D.C.: Jan. 3, 1997).
OTHER PROCESSES USED TO
MONITOR GRANTEE REGULATORY
COMPLIANCE ALSO HAVE
LIMITATIONS
---------------------------------------------------------- Letter :4.3
In addition to conducting on-site inspections, regional ACF staff
also monitor grantees' compliance with regulations by annually
reviewing their financial audit reports and their grant
applications.\11 \12 This monitoring supplements the triennial
on-site inspections. In general, financial audits are not designed
to provide detailed assessments of grantees' compliance with all
program regulations. Rather, financial auditors generally determine
whether grantees have established and implemented internal controls
that function to help ensure that programs comply with regulations.
Auditors may select and review samples of financial transactions to
determine whether a grantee has followed established procedures and
program regulations. If a grantee administers more than one federal
grant, as is often the case with large nonprofit agencies, school
districts, and municipalities, relatively small grants may not be
reviewed in as much detail, if at all, as larger grants. As a
result, Head Start programs that constitute only a small portion of a
grantee's total federal funding may not be reviewed or may not have
transactions related to the program selected for compliance review.
In addition to a lack of detail, financial audit reports may not
provide timely information for monitoring current grantee operations.
Grantees have 9 months to submit financial audit reports for any
given year.\13 It may take several additional months before officials
in the HHS Office of Inspector General review the audit report,
summarize findings, and submit the information to regional ACF staff,
who monitor grantees' resolutions of audit findings. Grantees that
fail to resolve audit findings within 6 months may be classified as
"high risk" and be required to submit additional financial reports or
be subject to additional financial controls.\14 However, high-risk
grantees generally do not face termination of their funding unless
they are also classified as deficient, which usually involves an
on-site inspection. As a result, ACF may wait up to 3 years until
the next regularly scheduled triennial inspection before it
classifies a high-risk grantee deficient and requires it to develop a
quality improvement plan and face termination from the program. ACF
officials encourage regional staff to conduct inspections, they said,
before regularly scheduled triennial inspections to determine whether
high-risk grantees should also be classified deficient.
Another tool for monitoring compliance is the review and approval of
Head Start grant applications and program budgets, which is done by
ACF regional staff. Grantees are generally required to submit a
noncompetitive grant application annually, including a proposed
budget. Regional staff told us that late applications for refunding
may indicate that a grantee is having difficulty complying with
program regulations. Regional staff cannot compare budgets with
actual expenditures at the needed detailed level, however, because
grantees do not have to report detailed expenditure data.
--------------------
\11 In general, the Single Audit Act requires that grantees that
receive federal funding undergo periodic financial audits generally
on an annual, or in certain cases, biennial basis. An independent
auditor conducts these audits, which include a review of grantees'
accounting and financial management systems.
\12 ACF regional staff also review other financial reports submitted
by grantees, including federal cash transaction reports and financial
status reports. The cash transaction reports provide data on
grantees' federal cash balances, while the status reports provide
aggregate data on grantees' expenditures. Late submissions of these
reports indicate that a grantee may be having difficulty complying
with program regulations and may highlight a need to more closely
monitor the grantee, according to several regional staff.
\13 Before June 30, 1997, a federal grantee had 13 months from the
end of its grant year to submit an independent financial audit for
that year.
\14 For example, a grantee may only be allowed to receive funds on a
cost-reimbursable basis or be required to provide documentation to
support program expenditures.
HEAD START PROCESSES IMPROVING
BUT STILL INSUFFICIENT TO
ASSESS WHETHER PROGRAM PURPOSES
ACHIEVED
------------------------------------------------------------ Letter :5
HHS has taken significant steps toward developing a comprehensive
strategy to assess whether the Head Start program is achieving its
purposes. Initially, HHS plans to measure program outcomes using the
results from a nationwide survey of a representative sample of
children enrolled in Head Start programs and their families. HHS
will use survey results to project the degree to which it is
accomplishing program purposes on a national level. In the future,
HHS plans to assess whether Head Start is achieving its purposes by
measuring individual grantees' progress in achieving desired
outcomes, rather than relying on a sample of grantees. HHS believes
this strategy is also sufficient to determine program impact.
Although the strategy will move HHS closer to being able to make such
a determination, HHS could do more to ensure that it accurately
measures the program's actual impact with a greater degree of
confidence. HHS would incur costs in such an effort; however, the
significant financial commitment the federal government has made to
the Head Start program warrants such an investment.
INITIATIVES WILL PROVIDE
FRAMEWORK FOR ASSESSING
NATIONAL OUTCOMES BUT NOT
LOCAL AGENCY OUTCOMES
---------------------------------------------------------- Letter :5.1
Overall, HHS has a methodologically and conceptually sound approach
to assessing outcomes. It has designed initiatives to assess its
progress in meeting its two outcome-focused objectives: (1)
enhancing children's growth and development and (2) strengthening
families as the primary nurturers of their children. HHS has
developed multiple performance measures to use in assessing progress
in meeting these objectives. For each measure, HHS has established
one or more performance indicators that will track the percentage of
change from the previous year.
Because data on these indicators were not available, HHS has
developed a strategy to obtain them, namely, the Family and Child
Experiences Survey (FACES), a study of a representative sample of
Head Start children and their families designed to generate
national-level data. FACES will provide most of the outcome data
needed to assess whether outcome-focused objectives are being met on
an ongoing basis. According to HHS officials, a spring 1997 FACES
pilot resulted in collecting data from a sample of 2,400 families
with children enrolled in 160 randomly selected centers in 40 Head
Start programs nationwide. Subsequently, the full study has involved
data collected in the fall of 1997 and spring of 1998. The overall
design is to collect data on 3,200 children and their families at
program entry, exit (or completion of each year of Head Start), and
at the end of kindergarten. In conducting the study, researchers
plan on using well-established and widely used scales, assessments,
and observational checklists and specially tailored questionnaires.
These instruments will collect data on children's vocabulary,
literacy and mathematical skills, perceptual-motor development, and
social and communicative competence as well as information about the
families.
By using well-established instruments, HHS will be able to assess
specific outcomes, such as literacy improvement or expressive
language. Meanwhile, HHS' use of a variety of instruments and scales
will facilitate assessing many dimensions of cognitive, emotional,
and physical growth and development. Its sampling approach is
designed to provide a national perspective on how well Head Start
children and families are doing. Furthermore, HHS has ensured that
the performance measures in its strategic framework are represented
in the national survey.
Although HHS has focused on outcomes at the national level, it has
not focused on outcomes at the local agency level, even though many
Head Start agencies receive substantial funding. In fiscal year
1997, funding ranged from about $120 million in New York City to
about $135,000 for one grantee in Minnesota, with 66 percent of the
local agencies receiving grants of at least $1 million. Grantees and
their respective delegate agencies, however, do not have to use the
measures associated with the two outcome-focused objectives to assess
whether outcomes have been achieved. Instead, according to ACF
officials, the performance measures are intended at this time to
provide a self-assessment tool for individual programs for strategic
planning purposes. Later on, ACF intends to require grantees to
demonstrate their progress in achieving outcomes, according to ACF
officials. They indicated that the transition from grantees'
complying with standards to assessing outcomes is a challenging one,
and they expect it will take time to work with the grantees to enable
them to assess outcomes. Assessing local programs' outcomes will be
difficult, according to HHS, because each agency operates in a unique
community. According to ACF, some grantees are already assessing
outcomes and ACF plans to use findings from their experiences to help
other grantees. ACF has not yet developed a plan or a timetable,
however, for this transition.
HHS INITIATIVES WILL NOT
ADEQUATELY ASSESS PROGRAM
IMPACT
---------------------------------------------------------- Letter :5.2
According to HHS, its strategy for assessing outcomes will also allow
it to determine Head Start's impact on children's growth and
development. HHS' plans will not provide the needed information
about program impact, however, because they do not include a research
study or set of studies that will definitively compare the outcomes
achieved by Head Start children and their families with those
achieved by similar non-Head Start children and families.
HHS PLANS MANY
COMPARISONS TO DRAW
CONCLUSIONS ABOUT IMPACT
-------------------------------------------------------- Letter :5.2.1
HHS has identified comparisons it believes will provide a basis to
draw conclusions about Head Start's impact on children's
developmental competencies. HHS will rely mainly on the FACES
initiative for making these comparisons. HHS has acknowledged,
however, that the absence of a control or comparison group makes it
difficult to explicitly attribute any participant progress to the
Head Start program. HHS proposes, as a solution to this problem, to
compare FACES data with existing data on the national population of
children of the same age and those from low-income families who have
not attended Head Start.
Some of the assessment tools used in FACES are nationally normed
instruments, which is why HHS is relying on them for its comparisons
with children in the national population.\15 By using these
instruments, HHS can compare the FACES results with the average
scores of all children of the same age. As Head Start collects data
on children at the end of each year of Head Start participation and
at the end of kindergarten, it will be able to compare the rate of
development of Head Start children with all preschoolers of the same
ages. For example, even though Head Start children may be behind
their age mates developmentally, they may show equivalent amounts of
growth between the fall and spring of the Head Start year or from 1
year to the next.
Another comparison HHS has identified would infer Head Start's impact
from differences in children who participated in the program to
different degrees or for varying durations. For example, if children
who attended Head Start classes for many hours a week instead of a
relatively few hours or for 2 years instead of 1 year showed greater
developmental gains, then the difference could be evidence of Head
Start's impact.
In addition, HHS has described comparing children who have attended
Head Start with those who have not as part of its effort to assess
impact. Data on the Head Start children would come from FACES; data
on non-Head Start children would come from the National Center for
Education Statistics' (NCES) National Household Education Survey
(NHES) on the developmental status and preschool program attendance
histories of a sample of children. HHS could not compare these
children using any measure that requires direct observation by a
trained interviewer or teacher, but it would be able to compare
parent reports of children's developmental accomplishments and
difficulties.
Another comparison option for HHS, which would not necessarily
include data from the FACES study, involves NCES' Early Childhood
Longitudinal Study (ECLS). HHS is collaborating with NCES on this
study, implemented in fall 1997 after a 5-year planning effort. The
study will collect data on a nationally representative sample of
kindergarten children in public and private schools. The study will
collect data from parents and children, including descriptions of
children's preschool experiences and performance on standardized
tests in areas such as achievement and psychomotor development.
Although information on the early childhood educational experience of
children will be limited, according to HHS, it will be possible to
compare the school progress of children who had attended Head Start
before entering the study with that of low-income children without a
center-based early childhood education experience.
--------------------
\15 Norms are obtained by administering a test to a sample of people
and deriving the distribution of scores for that group. Some of the
tests used by Head Start have been normed using samples selected to
represent the national population for a particular age group.
LIKELY DIFFERENCES AMONG
CHILDREN IN COMPARISONS
LIMIT HHS'
INTERPRETATIONS
-------------------------------------------------------- Letter :5.2.2
The noncomparability of children in the comparison groups will limit
HHS' ability to assess Head Start's impact. Without comparable
groups, outcomes could be mistakenly attributed to Head Start
participation when these outcomes were really caused by other factors
related to differences in the comparison groups.
HHS' plans to compare FACES results with national norms are limited
to the extent that the norms are not based on children like those in
Head Start programs but are generally based on children more
representative of the national population. Thus, if the children in
the group used to establish the norms are unlike the children in Head
Start, conclusions about program impact will be unclear.
Furthermore, comparing norms over time to assess gains assumes that
the groups have an equivalent learning rate, which may not be the
case. The lack of assurance that groups are comparable similarly
limits HHS' plans to compare selected data with the NHES and ECLS
samples. Although comparing Head Start results with other study
results moves in the direction of assessing impact, it is not
definitive enough.
Given the size of the Head Start program and plans for its expansion,
investing in studies that will definitively assess its impact is
important. As HHS has acknowledged, the most conclusive way to
determine program impact is to compare a group of Head Start
participants with an equivalent group of nonparticipants. The
preferred method for establishing the equivalency of groups at the
outset is to randomly assign participants to either a Head Start
group or a comparison group. No matter how extensive the efforts to
document the equivalency of groups formed in other ways, the groups'
equivalency will remain uncertain. If the groups are not equivalent,
outcomes attributed to Head Start will be in doubt. For instance, a
recent evaluation of the Comprehensive Child Development Program,\16
a demonstration project involving comprehensive early childhood
services like Head Start's, found positive changes in participating
families. The study compared participants with comparable
nonparticipants, and researchers discovered that nonparticipating
families had positive changes similar to the participating families.
They concluded, therefore, that the positive changes could not be
attributed to program participation. Researchers in this study had
confidence that the groups were comparable when the participants
entered the program because participants had been randomly assigned
to groups.
To obtain information about impact, HHS could conduct a study or set
of studies using random assignment of Head Start-eligible children
and their families. HHS could design studies to compare outcomes of
Head Start participants with those of nonparticipants.
Nonparticipants would not necessarily receive no services but could
participate in other educational settings, child care, or in any
other programs. HHS could also explore the impact of different Head
Start approaches used by different programs by comparing the outcomes
in these local Head Start programs with each other as well as with
other non-Head Start programs.
--------------------
\16 National Impact Evaluation of the Comprehensive Child Development
Program, Abt Associates (Cambridge, Mass.: June 1997).
CONCLUSIONS
------------------------------------------------------------ Letter :6
Head Start has, through the years, provided a comprehensive array of
services and, as prompted by the Results Act, has in recent years
substantially strengthened its emphasis on determining the results of
those services. Furthermore, recent legislation has established
significant consequences for Head Start grantees that do not comply
with requirements and placed new requirements on agencies to assess
achievement of program purposes. The Head Start program's
performance in meeting challenges for these new requirements is
uneven, however.
HHS has established a comprehensive process to ensure that Head Start
grantees comply with program regulations and has recently taken steps
to address weaknesses in its on-site inspection process to ensure the
consistency of inspections. Given that on-site inspections are HHS'
main mechanism for ensuring regulatory compliance, the integrity of
this process is essential to an effective enforcement program.
In addition, improving the consistency of this process would give HHS
more confidence in the information management systems that track
these data. In turn, HHS would be in a better position to provide
the Congress with the information and reports it requested when the
Congress last reauthorized Head Start. These data can also provide
valuable information at the national level that would be useful for
developing policy and practices for guiding grantees toward improved
compliance and service delivery.
Head Start's initiatives, in particular, FACES, are headed in the
right direction because of their increased focus on outcomes and
research that could be expanded to compare outcomes for children in
Head Start with those for similar nonparticipating children and
families. HHS' current processes, however, focus only on
national-level outcomes based on data collected through nationally
representative samples. Although this type of outcome assessment
would be new and therefore challenging to grantees, knowing whether
local programs achieve results is important. Given the significant
federal investment in the Head Start program, collecting such data is
worthwhile.
Head Start's planned processes to assess program impact--that is,
whether a difference in outcome has resulted from Head Start
participation--do not provide a definitive assessment of the
program's overall impact. The federal government's significant
financial investment in the Head Start program, including plans to
increase the number of children served and enhance the quality of the
program, warrants definitive research studies, even though they may
be costly. HHS has no plans for a research study or set of studies
that will definitively compare the outcomes achieved by Head Start
children and their families with those achieved by similar non-Head
Start children and families. Consequently, questions about Head
Start's impact will remain unanswered.
RECOMMENDATIONS TO THE
SECRETARY OF THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
------------------------------------------------------------ Letter :7
To ensure that individual Head Start grantees are held accountable
for achieving program purposes, we recommend that HHS develop and
implement a plan for assessing individual grantees' performance based
on their achieving the outcomes associated with HHS' performance
objectives. Such a plan could include, for example, guidance and
suggested methods for grantees to use in assessing the degree to
which children show improvement in critical outcome areas such as
cognitive skills, literacy, and gross motor skills.
To determine whether the Head Start program is making a difference in
the lives of those it serves, we recommend that HHS assess the impact
of regular Head Start programs by conducting a study or studies that
will definitively compare the outcomes achieved by Head Start
children and their families with those achieved by similar non-Head
Start children and families.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :8
In commenting on a draft of our report, HHS affirmed its commitment
to accountability and results for the Head Start program. It agreed
with our recommendation that a plan be developed and implemented to
assess individual grantees' performance but noted that such a plan
should be developed cautiously to ensure that the measures have no
unintended consequences such as diverting programs from their
historic commitment to serving a community's most disadvantaged
children. We agree that such plans should be developed carefully and
in consultation with experts. Meanwhile, a timetable should be
established that delineates when and how grantees would be held
accountable and when HHS will provide guidance to grantees on types
of measures and instruments.
Regarding our recommendation that the Secretary conduct a study or
set of studies to assess the impact of regular Head Start programs,
the Department expressed the belief that its current research
strategy of multiple studies and comparisons will produce sufficient
information about impact. We agree with HHS that its strategy of
multiple studies and comparisons will provide useful information
about outcomes achieved by Head Start children and that the
comparisons with non-Head Start children can be used to draw some
limited conclusions about Head Start's impact. HHS has acknowledged
that these conclusions will have limitations, however, because of the
study designs. We believe HHS' disagreement with our recommendation
reflects in part a misunderstanding of its nature and how it could be
implemented. HHS' comments imply that we recommended that its only
research activity should be a single, nationally representative
impact study that would assess the impact of Head Start by comparing
children in Head Start with children receiving no early childhood
education services. We are not now nor have we ever recommended that
the Department abandon all other research activities to focus
exclusively on a single study to assess impact. Instead, we
recommended in an earlier report that the Department include in HHS'
research plan an assessment of the impact of regular Head Start
programs, and we are now recommending that HHS conduct a study or
studies, along with its other research efforts, that will
definitively assess the program's impact.
HHS also expressed doubt that studies using random assignment of
children to Head Start to ensure that Head Start participants are
comparable with nonparticipants could feasibly be done and that such
studies would provide more definitive information about the program's
impact. HHS has acknowledged, however, that it is working with local
partners to determine the feasibility of implementing studies that
use a random assignment design to assess Head Start's effectiveness.
We disagree with HHS' conclusion that designs using random assignment
are not feasible. Although we acknowledge the difficulty of
designing and implementing studies with random assignment, some of
the ethical, methodological, and logistical difficulties that HHS
cites reflect in part several questionable assumptions about how we
would expect such studies to be done. First, HHS notes that mandates
to serve the most needy of eligible families and to incorporate a
minimum percentage of children with disabilities guide Head Start's
recruitment. HHS cites the idea of recruiting more children than
would be served using a random assignment study design as a
difficulty in implementing the recommendation. It is not clear how
or why this would be an unsurmountable problem. A second difficulty
cited--apparently a logistical one--is finding enough eligible
children not in Head Start. Relevant to both of these issues is a
point HHS has frequently made in other documents: that large numbers
of children are eligible for Head Start but are not being served.
Plans to expand the program are based on this assumption of unmet
need. To the extent that a significant unmet need for Head Start
exists, then using a lottery to determine which among the many needy
children would receive the service should not be unsurmountable. A
third issue hinges on HHS' assertion that the kind of study we
recommended would require that the children not participating in Head
Start would have to receive no early childhood education services at
all or exactly identical non-Head Start services. A study using
randomization could have more than one comparison group. In fact, by
using multiple alternative comparison groups, HHS could compare
outcomes achieved by Head Start participants with outcomes of
children in other programs. Furthermore, children do not necessarily
have to receive no services to be part of a comparison group; that
is, families in a comparison group could choose to participate in
child care settings or other preschool programs.
Finally, HHS maintained that the mobility of low-income populations
would probably result in children leaving treatment in unpredictable
numbers and at unpredictable points in time or crossing among program
and comparison groups. Although attrition and mobility can occur, a
well-
designed study would incorporate these kinds of factors and develop
mechanisms to account for their effect. For example, follow-up
studies would allow researchers to obtain information that would
allow them to account for the extent of attrition or mobility.
Furthermore, collecting information on attrition would be an
interesting characteristic to examine for determining the similarity
of attrition patterns.
Despite our areas of disagreement with HHS, we are encouraged that
the agency is continuing to study the feasibility of conducting
studies that include random assignment and is willing to consider
seeking additional advice from experts in research design. We hope
that the results of these activities will lead to studies producing
more definitive conclusions about Head Start's impact than can be
drawn from the studies currently planned. Such studies would be
extremely useful additions to Head Start's considerable body of
ongoing research for assessing and improving the program. The full
text of HHS' comments appears in appendix I.
---------------------------------------------------------- Letter :8.1
We are sending copies of this report to the Secretary of Health and
Human Services, appropriate congressional committees, and other
interested parties. Please call me at (202) 512-7014 or Harriet
Ganson, Assistant Director, at (202) 512-9045 if you or your staff
have any questions about this report. Major contributors to this
report are listed in appendix II.
Carlotta C. Joyner
Director, Education and
Employment Issues
List of Requesters
The Honorable Dan Coats
Chairman, Subcommittee on Children and Families
Committee on Labor and Human Resources
United State Senate
The Honorable Christopher J. Dodd
Ranking Minority Member
Subcommittee on Children and Families
Committee on Labor and Human Resources
United States Senate
The Honorable Bill Goodling
Chairman, Committee on Education and the Workforce
House of Representatives
The Honorable William Clay
Ranking Minority Member
Committee on Education and the Workforce
House of Representatives
The Honorable Frank Riggs
Chairman, Subcommittee on Early Childhood, Youth and Families
Committee on Education and the Workforce
House of Representatives
The Honorable Matthew G. Martinez
Ranking Minority Member
Subcommittee on Early Childhood, Youth and Families
Committee on Education and the Workforce
House of Representatives
The Honorable Dale Kildee
House of Representatives
The Honorable Randy Cunningham
House of Representatives
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
GAO CONTACTS AND STAFF
ACKNOWLEDGMENTS
========================================================== Appendix II
GAO CONTACTS
Harriet C. Ganson, Assistant Director, (202) 512-9045
Veronica Scott, Evaluator-in-Charge, (202) 512-7039
STAFF ACKNOWLEDGMENTS
In addition to those named above, the following individuals made
important contributions to this report: Karen A. Whiten, Assistant
Director; Mary Roy, Senior Evaluator; Donnesha Y. Correll, Issue
Area Assistant; C. Jeff Appel, Adviser; and Liz Williams, editor.
*** End of document. ***