Child and Family Services Reviews: Better Use of Data and	 
Improved Guidance Could Enhance HHS's Oversight of State	 
Performance (20-APR-04, GAO-04-333).				 
                                                                 
In 2001, the Department of Health and Human Services' (HHS)	 
Administration for Children and Families (ACF) implemented the	 
Child and Family Services Reviews (CFSR) to increase states'	 
accountability. The CFSR uses states' data profiles and statewide
assessments, as well as interviews and an on-site case review, to
measure state performance on 14 outcomes and systemic factors,	 
including child well-being and the provision of caseworker	 
training. The CFSR also requires progress on a program		 
improvement plan (PIP); otherwise ACF may apply financial	 
penalties. This report examines (1) ACF's and the states'	 
experiences preparing for and conducting the statewide		 
assessments and on-site reviews; (2) ACF's and the states'	 
experiences developing, funding, and implementing items in PIPs; 
and (3) any additional efforts that ACF has taken beyond the CFSR
to help ensure that all states meet federal goals related to	 
children's safety, permanency, and well-being.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-04-333 					        
    ACCNO:   A09808						        
  TITLE:     Child and Family Services Reviews: Better Use of Data and
Improved Guidance Could Enhance HHS's Oversight of State	 
Performance							 
     DATE:   04/20/2004 
  SUBJECT:   Accountability					 
	     Child abuse					 
	     Child adoption					 
	     Children						 
	     Data collection					 
	     Data integrity					 
	     Federal/state relations				 
	     Foster children					 
	     Performance measures				 
	     Program evaluation 				 
	     State-administered programs			 
	     Surveys						 
	     HHS Adoption and Foster Care Analysis		 
	     and Reporting System				 
                                                                 
	     HHS Child and Family Services Reviews		 
	     System						 
                                                                 
	     NCCAN National Child Abuse and Neglect		 
	     Data System					 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-04-333

United States General Accounting Office

GAO

                       Report to Congressional Requesters

April 2004

                                CHILD AND FAMILY
                                SERVICES REVIEWS

Better Use of Data and Improved Guidance Could Enhance HHS's Oversight of State
                                  Performance

GAO-04-333

Highlights of GAO-04-333, a report to Congressional Requesters

In 2001, the Department of Health and Human Services' (HHS) Administration
for Children and Families (ACF) implemented the Child and Family Services
Reviews (CFSR) to increase states' accountability. The CFSR uses states'
data profiles and statewide assessments, as well as interviews and an
on-site case review, to measure state performance on 14 outcomes and
systemic factors, including child well-being and the provision of
caseworker training. The CFSR also requires progress on a program
improvement plan (PIP); otherwise ACF may apply financial penalties. This
report examines (1) ACF's and the states' experiences preparing for and
conducting the statewide assessments and on-site reviews; (2) ACF's and
the states' experiences developing, funding, and implementing items in
PIPs; and (3) any additional efforts that ACF has taken beyond the CFSR to
help ensure that all states meet federal goals related to children's
safety, permanency, and well-being.

GAO recommends that the Secretary of HHS ensure that ACF uses the best
available data to measure state performance, clarify PIP guidance, and
help regional offices better integrate their oversight activities. In
commenting on this report, HHS generally agreed with GAO's findings and
said that it is committed to continually monitoring and improving the CFSR
process.

www.gao.gov/cgi-bin/getrpt?GAO-04-333.

To view the full product, including the scope and methodology, click on
the link above. For more information, contact Cornelia Ashby at (202)
512-8403 or [email protected].

April 2004

CHILD AND FAMILY SERVICES REVIEWS

Better Use of Data and Improved Guidance Could Enhance HHS's Oversight of State
Performance

ACF and many state officials perceive the CFSR as a valuable process and a
substantial undertaking, but some data enhancements could improve its
reliability. ACF staff in 8 of the 10 regions considered the CFSR a
helpful tool to improve outcomes for children. Further, 26 of 36 states
responding to a relevant question in our survey commented that they
generally or completely agreed with the results of the final CFSR report,
even though none of the 41 states with final CFSR reports released through
2003 has achieved substantial conformity on all 14 outcomes and systemic
factors-see figure below. Additionally, both ACF and the states have
dedicated substantial financial and staff resources to the process.
Nevertheless, several state officials and child welfare experts we
interviewed questioned the accuracy of the data used in the review
process. While ACF officials contend that stakeholder interviews and case
reviews complement the data profiles, many state officials and experts
reported that additional data from the statewide assessment could bolster
the evaluation of state performance.

State Performance on the 14 CFSR Outcomes and Systemic Factors

Program improvement planning is under way, but uncertainties have affected
the development, funding, and implementation of state PIPs. Officials from
3 of the 5 states we visited said ACF's PIP-related instructions were
unclear, and at least 9 of the 25 states reporting on PIP implementation
in our survey said that insufficient funding and staff were among the
greatest challenges. While ACF has provided some guidance, ACF and state
officials remain uncertain about PIP monitoring efforts and how ACF will
apply financial penalties if states fail to achieve their stated PIP
objectives.

Since 2001, ACF's focus has been almost exclusively on the CFSRs and
regional staff report limitations in providing assistance to states in
helping them to meet key federal goals. While staff from half of ACF's
regions told us they would like to provide more targeted assistance to
states, and state officials in all 5 of the states we visited said that
ACF's existing technical assistance efforts could be improved, ACF
officials acknowledged that regional staff might still be adjusting to the
new way ACF oversees child welfare programs.

Contents

  Letter

Results in Brief
Background
The CFSR Is a Valuable yet Substantial Undertaking, but Data

Enhancements Could Improve Its Reliability
Program Improvement Planning Under Way, but Uncertainties

Challenge Plan Development, Implementation, and Monitoring
ACF's Focus Rests Almost Exclusively on Implementing the CFSR
Conclusions
Recommendations for Executive Action
Agency Comments

                                       1

                                      3 5

                                       8

18 28 31 32 33

Appendix I Objectives, Scope, and Methodology 37

Objectives 37
Scope and Methodology 37

Appendix II List of Outcomes and Systemic Factors and Their

  Associated Items 41

Outcome factors 41
Systemic Factors 43

Appendix III	Dates on Which Site Visit States Reached CFSR Milestones

Appendix IV	Comments from the Department of Health and Human Services

Appendix V GAO Contacts and Acknowledgments 52

GAO Contacts 52
Staff Acknowledgments 52

  Related GAO Products 53

Tables

Table 1: Items Most Frequently Assigned a Rating of Area Needing
Improvement among the 41 States Reviewed in 2001, 2002, and 2003

Table 2: Number of States Including Each of the PIP Strategy Categories
Used in This Study Table 3: ACF Regulations and Guidance on Program
Improvement Planning

                                       11
                                       20
                                       22

Figures                                                                 
                      Figure 1: Four Phases of the CFSR Process             6 
               Figure 2: State Performance on the 14 CFSR Outcomes and     
                                   Systemic Factors                        10 
             Figure 3: Variation in the Start of States' PIP Development   24 
            Figure 4: Average Time Lapse between States' PIP Due Dates and 
                                  ACF's PIP Approval                       25 
                Figure 5: Most Common Challenges Affecting States' PIP     
                                    Implementation                         26 

Abbreviations

ACF Administration for Children and Families
AFCARS Adoption and Foster Care Analysis and Reporting System
ASFA Adoption and Safe Families Act
CFSR Child and Family Services Review
CIP court improvement program
FTE full-time equivalent
HHS Department of Health and Human Services
NCANDS National Child Abuse and Neglect Data System
PIP program improvement plan
SACWIS Statewide Automated Child Welfare Information System

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States General Accounting Office Washington, DC 20548

April 20, 2004

Congressional Requesters

Recent media reports have called into question a number of states'
abilities to protect the more than 800,000 children estimated to spend
some time in foster care each year. To help states investigate reports of
child abuse and neglect, provide placements to children outside their
homes, and deliver services to help keep families together, the federal
government provides approximately $7 billion dollars to states annually.
In addition, the federal Department of Health and Human Services (HHS)
monitors states' compliance with key federal goals, specified in part by
the Adoption and Safe Families Act (ASFA) of 1997, to keep children safe
and ensure their placement in stable and permanent homes.

In 2001, HHS-through its Administration for Children and Families' (ACF)
Children's Bureau-began implementing a new federal review system, known as
the Child and Family Services Reviews (CFSR), to hold states accountable
for improving child welfare outcomes. Unlike prior federal reviews-which
determined states' adherence to certain process measures-ACF designed the
CFSRs as an outcome-oriented approach to assess children's safety; their
timely placement in permanent homes; and their mental, physical, and
educational well-being. ACF also developed a number of factors to assess
states' performance in systemic areas, such as staff training and foster
parent licensing.1 According to ACF, this CFSR process combines statewide
assessments, which the states complete using a profile of their recent
child welfare data; on-site reviews of child welfare

1The CFSR measures state performance on 45 performance items, which
correspond to

7 outcomes and 7 systemic factors. The outcomes relate to children's
safety, permanency, and well-being, and the systemic factors address state
agency management and responsiveness to the community. Six national
standards, as reported in the Adoption and Foster Care Analysis and
Reporting System (AFCARS) and the National Child Abuse and Neglect Data
System (NCANDS), apply to 5 of the 45 items. Three of these standards are
based on the 75th percentile of all states' performance-adoption;
stability of foster care placements; and length of time to achieve
reunification, guardianship, or permanent placement with relatives-because
a higher incidence is desirable. However, the remaining three
standards-recurrence of maltreatment, incidence of child abuse/neglect in
foster care, and foster care re-entries-are based on the 25th percentile
of state performance, because lower incidence is a desired outcome for
these measures.

case files; and interviews with stakeholders2 to provide both qualitative
and quantitative information that helps describe and clarify complex child
welfare issues. Further, states are required to develop and implement
program improvement plans (PIP) to improve their child welfare practice
and capacity when they are found to be deficient. Pursuant to CFSR
regulations, ACF can withhold federal funds if states do not show adequate
progress implementing their PIPs.

To better understand the CFSR process, you asked us to examine (1) ACF's
and the states' experiences preparing for and conducting the statewide
assessments and on-site reviews; (2) ACF's and the states' experiences
developing, funding, and implementing items in their PIPs; and (3)
additional efforts, if any, that ACF has taken beyond the CFSR to help
ensure that all states meet federal goals of safety, permanency, and
well-being for children.

To conduct our work, we surveyed all 50 states, the District of Columbia,
and Puerto Rico regarding their interactions with ACF, the resources and
staffing required to prepare for the review and develop and implement
their PIPs, and any assistance ACF provided throughout the process. We
achieved a 98 percent response rate from this survey, as Puerto Rico was
the only non-respondent. To supplement this information, we also engaged
in intensive post-survey follow-up phone calls with key states in each
phase of the CFSR process. Further, we visited 5 states-California,
Florida, New York, Oklahoma, and Wyoming-to obtain first-hand information
on states' experiences. We selected these states for diversity in their
location, size, program administration, performance on the CFSR, and the
timing of their review. We also examined all 31 approved PIPs available as
of January 1, 2004, to categorize common strategies states developed to
improve their performance. In addition, we conducted interviews with ACF's
senior officials and regional staff from all 10 regions, ACF contractors,
directors and staff from all 10 national resource centers, and key child
welfare experts.3 We conducted our work between

2The term stakeholder refers to two groups: (1) agency stakeholders, such
as judges or advocates, whose responsibilities are closely related to the
work of the child welfare agency and who can comment on the agency's
overall performance on outcomes and systemic factors, and (2)
case-specific stakeholders, such as parents, caseworkers, children, or
others who are interviewed to provide first-hand information that
supplements reviewers' assessment of paper or electronic case files.

3Each of the 10 resource centers operates under a 5-year cooperative
agreement with HHS to help states implement federal child welfare
legislation and provide training and technical assistance to states.

Results in Brief

May 2003 and February 2004 in accordance with generally accepted
government auditing standards. A more detailed discussion of our scope and
methodology appears in appendix I.

ACF and many state officials perceive the CFSR as a valuable process-
highlighting many areas needing improvement-and a substantial undertaking,
but some state officials and child welfare experts told us that data
enhancements could improve its reliability. ACF staff in 8 of the 10
regions considered the CFSR a helpful tool to improve outcomes for
children, and officials from 8 regions noted that the CFSRs were more
intensive and comprehensive than other types of reviews they had conducted
in the past. Further, 26 of the 36 states responding to a relevant
question in our survey commented that they generally or completely agreed
with the results of the final CFSR report, even though none of the 41
states with final CFSR reports released through 2003 has achieved
substantial conformity on all 14 outcomes and systemic factors. In
addition, both ACF and the states have dedicated substantial financial and
staff resources to the process. For example, ACF committed $6.6 million to
hire contractors for review assistance, and based on survey responses from
the 45 states that reported on staff resources, an average of 47 fulltime
staff equivalents per state participated in the statewide assessment
phase. Nevertheless, several state officials and child welfare experts we
interviewed questioned the accuracy of the data used to compile state
profiles and establish the national standards. While ACF officials in the
central office contend that stakeholder interviews and case reviews
complement the data profiles, several state officials and experts reported
that additional data from the statewide assessment could bolster the
evaluation of state performance.

The program improvement planning process for the CFSR, which all states
with final reports have been required to undertake, has been characterized
by many uncertainties, such as those related to federal guidance and
monitoring and the availability of state resources to implement the plans.
Despite such uncertainties, states have included a variety of strategies
in their PIPs to address weaknesses identified in the review. Such
strategies include involving foster and birth parents in identifying
needed services and developing tools to assess the risk of harm to
children. To assist states with PIP development, ACF issued regulations
and guidance to states for developing the plans, but state responses to
this assistance have been mixed. For example, state officials attributed a
lengthy PIP approval process, in part, to unclear regulations and
insufficient guidance. Our analysis of the time lapse between the date
states must submit a PIP and

ACF's PIP approval date shows a range from 45 to 349 business days.
Further, 6 of the 21 states reporting on the PIP approval process said
that ACF did not clearly describe the process to them, and officials in 3
of the 5 states we visited said they were unclear as to ACF's expectations
regarding the format and content of the plans. In response to a question
on PIP implementation challenges, at least 9 of 25 states reported that
insufficient funding, staff, and time, as well as high caseloads, were the
most challenging factors. For example, one survey respondent reported that
unless the state receives additional funding, it would need to lay off
social workers and attorneys, making it difficult to implement some of its
improvement strategies. ACF officials noted that they have clarified PIP
expectations and increased technical assistance over time, but they
acknowledge that they are still unsure how best to monitor state PIP
implementation and have not yet determined when and how to apply the
prescribed financial penalties for states' failures to improve outcomes.

Since 2001, ACF's focus has been almost exclusively on implementing the
CFSRs, and regional staff report limitations in providing assistance to
states in helping them to meet key federal goals. For example, ACF
provides internal training for its staff and technical assistance to
states, but all 18 courses provided to regional staff have dealt with the
various phases of the review. Staff in 2 regional offices told us that
they have not been able to counsel states on such topics as contracting
out for child welfare services and maximizing the receipt of federal child
welfare funds because they have lacked the applicable training. In
addition, while ACF organizes biennial conferences for state and federal
child welfare officials, staff from 5 regions told us that they wanted
more substantive interaction with their ACF colleagues to increase their
overall child welfare expertise. While regional officials conduct site
visits to states, CFSR on-site reviews and PIP-related assistance account
for the majority of regions' time and travel budgets, and ACF staff from 5
regions said that more frequent visits with state personnel-outside of the
CFSR process-would allow them to better understand states' programs and
cultivate relationships with state officials. Further, state officials in
all 5 of the states we visited said that ACF's technical assistance needed
improvement, acknowledging, in some cases, that regional office staff were
stretched thin by CFSR demands and, in other cases, that the assistance
from resource center staff lacked focus or practicality. Although ACF
officials told us the CFSR has become the agency's primary mechanism for
monitoring states and facilitating program improvement, they acknowledged
that regional staff might still be adjusting to the new way ACF oversees
child welfare programs.

Given the emphasis that ACF has placed on the CFSRs-and the resources and
time states have invested to prepare for the reviews and address their
deficiencies through program improvement plans-we are recommending that
the Secretary of HHS ensure that ACF uses the best available data to
measure state performance. We are also recommending that the Secretary
clarify PIP guidance and provide guidance to regional officials on how to
better integrate their many oversight responsibilities. In commenting on a
draft of this report, HHS generally agreed with our findings and
recommendations. HHS acknowledged that the CFSR is a new process that
continues to evolve, and also noted several steps it has taken to address
the data quality concerns we raise in this report. We believe that our
findings fully address these initial actions, as well as the substantial
resources the agency has already dedicated to the review process. However,
to improve its oversight of state performance, our recommendations are
meant to encourage HHS to take additional actions to improve its use of
data in conducting these reviews and enhance PIPrelated guidance and
regional officials' understanding of how to incorporate the CFSR process
into their overall improvement and oversight efforts.

Background 	ACF's Children's Bureau administers and oversees federal
funding to states for child welfare services under Titles IV-B and IV-E of
the Social Security Act, and states and counties provide these child
welfare services, either directly or indirectly through contracts with
private agencies.4 Among other activities, ACF staff are responsible for
developing appropriate policies and procedures for states to follow to
obtain and use federal child welfare funds and conduct administrative
reviews of states' case files to ensure that children served by the state
meet statutory eligibility requirements.

In 2001, ACF launched a new outcome-oriented process, known as the Child
and Family Services Reviews (CFSR), to improve its existing monitoring
efforts, which had once been criticized for focusing exclusively on
states' compliance with regulations rather than on their performance over
a full range of child welfare services. Passage of the 1997 Adoption

4Title IV-B of the Social Security Act, consisting of two subparts, is the
primary source of federal funding for services to help families address
problems that lead to child abuse and neglect and to prevent the
unnecessary separation of children from their families. Funding under
Title IV-E of the Social Security Act is used primarily to pay for the
room and board of children in foster care.

and Safe Families Act (ASFA) helped spur the creation of the CFSR by
emphasizing the outcomes of safety, permanency, and well-being for
children. Subsequently, ACF consulted with state officials, child welfare
experts, and other interested parties, and conducted pilot CFSR reviews in
14 states. In January 2000, ACF released a notice of proposed rule making
and published final CFSR regulations. In March 2001, ACF conducted the
first of its state reviews in Delaware. By March of 2004, ACF had
completed an on-site review in all 50 states, the District of Columbia,
and Puerto Rico. Although ACF plans to initiate a second round of reviews,
an official start date for this process has not yet been determined.

As figure 1 indicates, the CFSR is a four-phase process that involves
state staff as well as central and regional ACF officials. This process
begins with a statewide assessment in the first phase.

Figure 1: Four Phases of the CFSR Process

      ACF and state          ACF's           ACF approves        ACF assesses 
      use statewide        contractor       PIP and state         the state's 
       assessment          prepares a          submits           achievements 
     information to       final report        quarterly               against 
        select 2         based in part      reports to ACF         negotiated 
      locations for     on the findings    for monitoring.         benchmarks 
     on-site review.     of the on-site                                   and 
    (3rd location is       review and                              determines 
     state's largest     releases this                         whether or not 
      metropolitan       report to the                              financial 
         area).              state.                                 penalties 
                                                                       apply. 

Source: GAO analysis.

The assessment of state performance continues in the second phase, most
commonly known as the on-site review, when ACF sends a team of reviewers
to three sites in the state for one week to assess state performance. A
list of all the outcomes and systemic factors and their associated items
appears in appendix II. In assessing performance, ACF relies, in part, on
its own data systems, known as the National Child Abuse and Neglect Data
System (NCANDS) and the Adoption and Foster Care Analysis and Reporting
System (AFCARS), which were designed prior to

CFSR implementation to capture, report, and analyze the child welfare
information collected by the states.5 Today, these systems provide the

national data necessary for ACF to calculate national standards for key
CFSR performance items with which all states' data will be compared.

After the on-site review, ACF prepares a final report for the state-
identifying areas needing improvement, as well as the outcomes and
systemic factors for which the state was determined not to be in
substantial conformity-and provides the state with an estimated financial
penalty.6 As a result, the state must develop a 2-year PIP with action
steps to address its noted deficiencies and performance benchmarks to
measure progress. Once ACF approves the PIP, states are required to submit
quarterly progress reports, which ACF uses to monitor improvement.
Pursuant to CFSR regulations, federal child welfare funds can be withheld
if states do not show adequate progress as a result of PIP implementation,
but these penalties are suspended during the 2-year implementation term.
As of January 2004, no financial penalties had been applied, but according
to data on the 41 states for which final CFSR reports have been released
through December 2003, potential penalties range from $91,492 for North
Dakota to $18,244,430 for California.

ACF staff in HHS's 10 regional offices provide technical assistance to
states through all phases of the CFSR process, and they are also
responsible for reviewing state planning documents required by Title IV-B,
assisting with state data system reviews, and assessing states' use of
IV-E funds. In addition to these efforts, ACF has established cooperative
agreements with 10 national resource centers to help states implement
federal legislation intended to ensure the safety, permanency, and well

5States began voluntarily reporting to NCANDS in 1990, and in 1995 started
reporting to AFCARS on the demographic characteristics of adoptive and
foster children and their parents, as well as foster children's type of
placement and permanency goals. We recently issued a report on states'
child welfare information systems and the reliability of child welfare
data. See Child Welfare: Most States Are Developing Statewide Information
Systems, but the Reliability of Child Welfare Data Could Be Improved,
GAO-03-809 (Washington, D.C.: July 31, 2003).

6States achieve substantial conformity on outcomes when at least 90
percent of applicable cases are substantially achieved; stakeholder
interviews confirm that state plan and other program requirements are in
place and functioning as described in the applicable regulations or
statute; and performance on items with national standards meets the
applicable threshold. The formula for calculating penalties is based in
part on each state's allocation of federal child welfare funds from Titles
IV-B and IV-E and the number of outcomes and systemic factors for which
substantial conformity has not been achieved.

The CFSR Is a Valuable yet Substantial Undertaking, but Data Enhancements
Could Improve Its Reliability

being of children and families. ACF sets the resource centers' areas of
focus, and although each center has a different area of expertise, such as
organizational improvement or information technology, all of them conduct
needs assessments, sponsor national conference calls with states,
collaborate with other resource centers and agencies, and provide on-site
training and technical assistance to states.7

Members of the 108th Congress introduced a proposal to provide federal
incentive payments directly to states that demonstrate significant
improvements to their child welfare systems. At the time of publication,
the House of Representatives was considering H.R. 1534, the Child
Protective Services Improvement Act, which contains provisions to award
grants to states with approved PIPs and additional bonuses to states that
have made considerable progress in achieving their PIP goals for the
previous year.

ACF and many state officials perceive the CFSR as a valuable process-
highlighting many areas needing improvement-and a substantial undertaking,
but some state officials and child welfare experts told us that data
enhancements could improve its reliability. ACF staff in 8 of the 10
regions considered the CFSR a helpful tool to improve outcomes for
children. Further, 26 of the 36 states responding to a relevant question
in our survey commented that they generally or completely agreed with the
results of the final CFSR report, even though none of the 41 states with
final CFSR reports released through 2003 has achieved substantial
conformity on all 14 outcomes and systemic factors. In addition, both ACF
and the states have dedicated substantial financial and staff resources to
the process. However, several state officials and child welfare experts we
interviewed questioned the accuracy of the data used to compile state
profiles and establish the national standards. While ACF officials in the
central office contend that stakeholder interviews and case reviews
compliment the data profiles, many state officials and experts reported
that additional data from the statewide assessment could bolster the
evaluation of state performance.

7Resource centers may provide unlimited assistance to states by phone or
e-mail, but onsite training and technical assistance are restricted to 10
days each year. States must first obtain regional office approval before
on-site training and technical assistance can occur. ACF officials in both
the regions and in headquarters indicated that they have been flexible
with extensions beyond the 10-day cap.

The CFSR Is a Valuable Process for ACF and the States

ACF and state officials support the objectives of the review, especially
in focusing on children's outcomes and strengthening relationships with
stakeholders, and told us they perceive the process as valuable. ACF staff
in 8 of the 10 regions considered the CFSR a helpful tool to improve
outcomes for children. Also, ACF officials from 8 regional offices noted
that the CFSRs were more intensive and more comprehensive than the other
types of reviews they had conducted in the past, creating a valuable tool
for regional officials to monitor states' performance. In addition, state
officials from every state we visited told us that the CFSR process helped
to improve collaboration with community stakeholders. For example, a court
official in New York said that the CFSR acted as a catalyst for improved
relations between the state agency and the courts, and he believes that
this has contributed to more timely child abuse and neglect hearings.
Additionally, a state official in Florida said that the CFSR stimulated
discussions among agency staff about measuring and improving outcomes,
particularly the data needed to examine outcomes and the resources needed
to improve them. Furthermore, state staff from 4 of the 5 states we
visited told us the CFSR led to increased public and legislative attention
to critical issues in child welfare. For example, caseworkers in Wyoming
told us that without the CFSR they doubted whether their state agency's
administration would have focused on needed reforms. They added that the
agency used the CFSR findings to request legislative support for the
hiring of additional caseworkers.

In addition to affirming the value associated with improved stakeholder
relations, the ACF officials we talked to and many state officials
reported that the process has been helpful in highlighting the outcomes
and systemic factors, as well as other key performance items that need
improvement. According to our survey, 26 of the 36 states that commented
on the findings of the final CFSR report indicated that they generally or
completely agreed with the findings, even though performance across the
states was low in certain key outcomes and performance items. For example,
not one of the 41 states with final reports released through 2003 was
found to be in substantial conformity with either the outcome measure that
assesses the permanency and stability of children's living situations or
with the outcome measure that assesses whether states had enhanced
families' capacity to provide for their children's needs. Moreover, across
all 14 outcomes and systemic factors, state performance ranged from
achieving substantial conformity on as few as 2 outcomes and

systemic factors to as many as 9.8 As figure 2 illustrates, the majority
of states were determined to be in substantial conformity with half or
fewer of the 14 outcomes and systemic factors assessed.

Figure 2: State Performance on the 14 CFSR Outcomes and Systemic Factors

States' performance on the outcomes related to safety, permanency, and
well-being-as well as the systemic factors-is determined by their
performance on an array of items, such as establishing permanency goals,
ensuring worker visits with parents and children, and providing accessible
services to families. The CFSR showed that many states need improvement in
the same areas, and table 1 illustrates the 10 items most

8California and Puerto Rico were determined to be in substantial
conformity on 2 outcomes and systemic factors, while North Dakota achieved
substantial conformity on 9.

frequently rated as needing improvement across all 41 states reviewed
through 2003.

Table 1: Items Most Frequently Assigned a Rating of Area Needing
Improvement among the 41 States Reviewed in 2001, 2002, and 2003

                                                             Number of states
                                                    assigned a rating of area
                                                     Item needing improvement

Assessing the needs and services of child, parents, and foster parents

Assessing mental health of child

Establishing the most appropriate permanency goal for the child

Demonstrating efforts to involve child and family in caseplanning
activities

Ensuring stability of foster care placements

Achieving a child's goal of adoption

Providing a process that ensures that each child has a written case plan
to be developed jointly with the child's parent(s) that includes the
required provisions

Ensuring workers conduct face-to-face visits with parent(s)

Providing services that are accessible to families and children in all
political jurisdictions covered in the state's Child and Family Services
Plan

Ensuring workers conduct face-to-face visits with child

Source: GAO analysis of the 41 states' final CFSR reports released through
December 31, 2003.

ACF and the States Report That Reviews Have Been a Substantial Undertaking

Given the value that ACF and the states have assigned to the CFSR process,
both have spent substantial financial resources and staff time to prepare
for and implement the reviews. In fiscal years 2001-2003, when most
reviews were scheduled, ACF budgeted an additional $300,000 annually for
CFSR-related travel. In fiscal year 2004, when fewer reviews were
scheduled, ACF budgeted about $225,000. To further enhance its capacity to
conduct the reviews, and to obtain additional logistical and technical
assistance, ACF spent approximately $6.6 million annually to hire
contractors. Specifically, ACF has let three contracts to assist with
CFSR-related activities, including training reviewers to conduct the
on-site reviews, tracking final reports and PIP documents, and, as of
2002, writing the CFSR final reports. Additionally, ACF hired 22 new staff
to build central and regional office capacity and dedicated 4 full-time
staff and 2 state government staff temporarily on assignment with ACF to
assist with the CFSR process. To build a core group of staff with CFSR
expertise,

ACF created the National Review Team, composed of central and regional
office staff with additional training in and experience with the review
process. In addition, to provide more technical assistance to the states,
ACF reordered the priorities of the national resource centers to focus
their efforts primarily on helping states with the review process.

Like ACF, states also spent financial resources on the review. While some
states did not track CFSR expenses-such as staff salaries, training, or
administrative costs-of the 25 states that reported such information in
our survey, the median expense to date was $60,550, although states
reported spending as little as $1,092 and as much as $1,000,000 on the
CFSR process.9 For example, California officials we visited told us that
they gave each of the state's three review sites $50,000 to cover the
salary of one coordinator to manage the logistics of the on-site review.
Although ACF officials told us that states can use Title IV-E funds to pay
for some of their CFSR expenses, only one state official addressed the use
of these funds in our survey, commenting that it was not until after the
on-site review occurred that the state learned these funds could have been
used to offset states' expenses.10 Furthermore, 18 of 48 states responding
to a relevant question in our survey commented that insufficient funding
was a challenge-to a great or very great extent-in preparing for the
statewide assessment, and 11 of 40 states responded that insufficient
funding was similarly challenging in preparing for the on-site review.
Officials in other states reported that because available financial
resources were insufficient, they obtained non-financial, in-kind
donations to cover expenses associated with the on-site review. For
example, a local site coordinator in Oklahoma informed us that while the
state budgeted $7 per reviewer each day of the on-site review for food,
refreshments, and other expenses, she needed to utilize a variety of
resources to supplement the state's budget with donations of supplies and
food from local companies and agency staff.

9These values are state-reported and reflect officials' estimates of costs
associated with all CFSR-related activities except those incurred during
PIP implementation. In reporting on their expenses, states were instructed
to include the value of training, travel, infrastructure, technology,
food, administrative supplies, and any other expenses associated with the
CFSR process. States were also asked to provide supporting documentation
for this particular question, but most states were unable to provide
documentation. Many states reported that they did not track CFSR-related
expenses. The 25 states that did provide estimates were in different
phases of the CFSR.

10ACF provided preliminary data, based on an inquiry of its regional
offices, showing that 25 states have charged some CFSR-related costs to
Title IV-E.

States reported that they also dedicated staff time to prepare for the
statewide assessment and to conduct the on-site review, which sometimes
had a negative impact on some staffs' regular duties. According to our
survey, 45 states reported dedicating up to 200 full-time staff
equivalents (FTEs), with an average of 47 FTEs, to the statewide
assessment process.11 Similarly, 42 states responded that they dedicated
between 3 and 130 FTEs, with an average of 45 FTEs, to the on-site review
process. To prepare for their own reviews, officials in all 5 states we
visited told us that they sent staff for a week to participate as
reviewers in other states. Additionally, local site coordinators in 4 of
the 5 states we visited reported that planning for the CFSR on-site review
was a full-time job involving multiple staff over a period of months. An
official in Florida also told us that the extensive preparation for the
CFSR dominated the work of the quality assurance unit, which was about 20
people at that time. In addition to preparing for the review, staff in all
5 of the states we visited, who served as reviewers in their own states,
reported that to meet their responsibility as case reviewers they worked
more than 12-hour days during the week of the on-site review. For some
caseworkers, dedicating time to the CFSR meant that they were unable or
limited in their ability to manage their typical workload. For example,
Wyoming caseworkers whose case files were selected for the on-site review
told us that they needed to be available to answer reviewers' questions
all day every day during the on-site review, which they said prevented
them from conducting necessary child abuse investigations or home visits.

Child welfare-related stakeholders-such as judges, lawyers, and foster
parents-also contributed time to the CFSR, but some states found it took
additional staff resources to initiate and maintain stakeholder
involvement over time. According to our survey, 46 states reported that an
average of about 277 stakeholders were involved with the statewide
assessment, and 42 states responded that an average of about 126
stakeholders participated in the on-site review.12 However, state
officials told us that it was difficult

11The number of FTEs participating in each phase of the CFSR is
state-reported. While states were not given specific instructions for how
to calculate FTEs, they were asked to report only on the phases of the
CFSR that they had started or completed. Therefore, states' responses
varied depending on the phase of the CFSR process they were in and the
methods they used to calculate FTEs.

12These values are state-reported and represent states' estimates
regarding the number of stakeholders that participated in each phase of
the CFSR process. In answering this question, states were provided with a
list of stakeholders, such as advocates and court officials, to guide
their estimates, but they were not asked to indicate the duration of
stakeholder involvement during each phase.

to recruit and maintain stakeholder involvement because long time lapses
sometimes occurred between CFSR phases. For example, more than a year can
elapse between completion of the statewide assessment and the initiation
of PIP development-both key points at which stakeholder participation is
critical. Nonetheless, all 5 of the states we visited tried to counter
this obstacle by conducting phone and in-person interviews, holding focus
groups, or giving presentations in local communities to inform and recruit
stakeholders to be involved in the CFSR process. As a result, some
stakeholders were involved throughout the process. For example, a tribal
representative in Oklahoma assisted with the development of the statewide
assessment, was interviewed during the onsite review, and provided
guidance on the needs of Indian children during PIP development.

States and Child Welfare Experts Report That Several Data Improvements
Could Enhance CFSR Reliability

State officials in all 5 states, as well as child welfare experts,
reported on several data improvements that could enhance the reliability
of CFSR findings. In particular, they highlighted inaccuracies with the
AFCARS and NCANDS data that are used for establishing the national
standards and creating the statewide data profiles, which are then used to
determine if states are in substantial conformity. These concerns echoed
the findings of a prior GAO study on the reliability of these data
sources, which found that states are concerned that the national standards
used in the CFSR are based on unreliable information and should not be
used as a basis for comparison and potential financial penalty.13 Several
of the state officials we visited and surveyed also questioned the
reliability of data given the variation in states' data-reporting
practice, which they believe may ultimately affect the validity of the
measures and may place some states at a disadvantage. Furthermore, many
states needed to resubmit their statewide data after finding errors in the
data profiles ACF would have used to measure compliance with the national
standards.14 According to our national survey, of the 37 states that
reported on resubmitting data for the statewide data profile, 23 needed to
resubmit their statewide data at least once, with 1 state needing to
resubmit as many as five times to accurately reflect revised data. Four
states reported in our survey that they did not resubmit their data
profiles because they did not know they had

13See GAO-03-809.

14ACF provides states with their statewide data about 6 months prior to
the on-site review, during which time states are allowed to make
corrections to the data and resubmit the updated data so it can be used
when determining state conformity with CFSR measures.

this option or they did not have enough time to resubmit before the
review.

In addition to expressing these data concerns, child welfare experts as
well as officials in all of the states we visited commented that existing
practices that benefit children might conflict with actions needed to
attain the national standards. Specifically, one child welfare expert
noted that if a state focused on preventing child abuse and neglect while
the child was still living at home, and only removed the most at-risk
children, the state may perform poorly on the CFSR reunification
measure-assessing the timeliness of children's return to their homes from
foster care arrangements-because the children they did remove would be the
hardest to place or most difficult to reunite. Additionally, officials in
New York said that they recently implemented an initiative to facilitate
adoptions. Because these efforts focus on the backlog of children who have
been in foster care for several years, New York officials predict that
their performance on the national standard for adoption will be lower
since many of the children in the initiatives have already been in foster
care for more than 2 years.

These same state officials and experts also commented that they believe
the on-site review case sample of 50 cases is too small to provide an
accurate picture of statewide performance, although ACF officials stated
that the case sampling is supplemented with additional information. Of the
40 states that commented in our survey on the adequacy of the case sample
size for the on-site review, 17 states reported that 50 cases were very or
generally inadequate to represent their caseload.15 Oklahoma officials we
visited also commented that they felt the case sample size was too small,
especially since they annually assess more than 800 of their own
cases-using a procedure that models the federal CFSR-and obtain higher
performance results than the state received on its CFSR. Furthermore,
because not every case in the states' sample is applicable to each item
measured in the on-site review, we found that sometimes as few as one or
two cases were being used to evaluate states' performance on an item.
Specifically, an ACF contractor said that several of the CFSR items
measuring children's permanency and stability in their living conditions

15According to our calculations-which assumed that the attribute of
interest occurred in about 50 percent of the cases-a sample size of 50
would produce percentage estimates with a 95 percent margin of error of
approximately plus or minus 14 percentage points. This level of
variability is a limitation when attempting to interpret estimates based
on this sample size.

commonly have very few applicable cases. For example, Wyoming had only two
on-site review cases applicable for the item measuring the length of time
to achieve a permanency goal of adoption, but for one of these cases,
reviewers determined that appropriate and timely efforts had not been
taken to achieve finalized adoptions within 24 months, resulting in the
item being assigned a rating of area needing improvement.16 While ACF
officials acknowledged the insufficiency of the sample size,17 they
contend that the case sampling is augmented by stakeholder interviews for
all items and applicable statewide data for the five CFSR items with
corresponding national standards, therefore providing sufficient evidence
for determining states' conformity.

All of the states we visited experienced discrepant findings between the
aggregate data from the statewide assessment and the information obtained
from the on-site review, which complicated ACF's determination of states'
performance. Each of the 5 states we visited had at least one, and
sometimes as many as three, discrepancies between its performance on the
national standards and the results of the case review. We also found that
in these 5 states, ACF had assigned an overall rating of area needing
improvement for 10 of the 11 instances where discrepancies occurred. ACF
officials acknowledged the challenge of resolving data discrepancies,
noting that such complications can delay the release of the final report
and increase or decrease the number of items that states must address in
their PIPs. While states have the opportunity to resolve discrepancies by
submitting additional information explaining the discrepancy or by
requesting an additional case review, only one state to date has decided
to pursue the additional case review.18 In addition, among the states we

16Because one of the two cases applicable to the adoption measure was
assigned a rating of area needing improvement, 50 percent of the cases for
this item were assigned a rating of area needing improvement. As a result,
the item was given an overall rating of area needing improvement since
both cases would have needed to be assigned a rating of strength for this
item to meet the 85 percent threshold necessary to assign an overall
rating of strength.

17An ACF statistician also confirmed that the CFSR sample is too small to
generalize to the states' populations and that the three sites, from which
cases are selected, also are not representative.

18Virginia requested an additional case review to resolve a discrepancy
between the statewide data and on-site review findings for the item
measuring the state's performance on foster care re-entries. According to
an ACF regional official, the state met the national standard for this
item but the case review findings showed the state did not meet the
threshold for this measure. At the time of publication, ACF and the state
were still finalizing plans to conduct the additional case review, and
until the review is completed, the state cannot receive its final report.

visited, for example, one state acknowledged that it has not opted to
pursue the supplemental case review because doing so would place
additional strain on its already limited resources.

Several state officials and experts told us that additional data from the
statewide assessments-or other data sources compiled by the states- could
bolster the evaluation of states' performance, but they found this
information to be missing or insufficiently used in the final reports.
According to our survey, of the 34 states that commented on the adequacy
of the final report's inclusion of the statewide assessment, 10 states
reported that too little emphasis was placed on the statewide
assessment.19 Specifically, 1 state reported that the final report would
have presented a more balanced picture of the state's child welfare system
if the statewide assessment were used as a basis to compare and clarify
the on-site review findings. Further, child welfare experts and state
officials from California and New York-who are using alternative data
sources to AFCARS and NCANDS, such as longitudinal data that track
children's placements over time-told us that the inclusion of this more
detailed information would provide a more accurate picture of states'
performance nationwide. North Carolina officials also reported in our
survey that they tried to submit additional longitudinal data-which they
use internally to conduct statewide evaluations of performance-in their
statewide assessment, but HHS would not accept the alternative data for
use in evaluating the state's outcomes. An HHS official told us that
alternative data are used only to assess state performance in situations
where a state does not have NCANDS data, since states are not mandated to
have these systems.

Given their concerns with the data used in the review process, state
officials in 4 of the 5 states believed that the threshold for achieving
substantial conformity was difficult to achieve. One state official we
visited acknowledged that she believed child welfare agencies should be
pursuing high standards to improve performance, but she questioned the
level at which ACF established the thresholds. While an ACF official told
us that different thresholds for the national standards had been
considered, ACF policy makers ultimately concluded that a more rigorous
threshold would be used. ACF officials recognize that they have set a high
standard. However, they believe it is attainable and supportive of their
overall approach to move states to the standard through continuous
improvement.

19The remaining 24 respondents reported that the emphasis was "just about
right."

Program Improvement Planning Under Way, but Uncertainties Challenge Plan
Development, Implementation, and Monitoring

In preparation for the next round of CFSRs, ACF officials have formed a
Consultation Work Group of ACF staff, child welfare administrators, data
experts, and researchers who will propose recommendations on the CFSR
measures and processes. The group began its meetings prior to our
publication, but no proposals were available.

Forty-one states are engaged in program improvement planning, but many
uncertainties, such as those related to federal guidance and monitoring
and the availability of state resources, have affected the development,
implementation, and funding of the PIPs. State PIPs include strategies
such as revising or developing policies, training caseworkers, and
engaging stakeholders, and ACF has issued regulations and guidance to help
states develop and implement their plans. Nevertheless, states reported
uncertainty about how to develop their PIPs and commented on the
challenges they faced during implementation. For example, officials from 2
of the states we visited told us that ACF had rejected their PIPs before
final approval, even though these officials said that the plans were
similar in the level of detail included in previously approved PIPs that
regional officials had provided. Further, at least 9 states responding to
our survey indicated that insufficient time, funding, and staff, as well
as high caseloads, were the greatest challenges to PIP implementation. As
states progress in PIP implementation, some ACF officials expressed a need
for more guidance on how to monitor state accomplishments, and both ACF
and state officials were uncertain about how the estimated financial
penalties would be applied if states fail to achieve the goals described
in their plans.

State Plans Include a Variety of Strategies to Address Identified
Weaknesses

State plans include a variety of strategies to address weaknesses
identified in the CFSR review process. However, because most states had
not completed PIP implementation by the time of our analysis, the extent
to which states have improved outcomes for children has not been
determined.20 While state PIPs varied in their detail, design, and scope,
according to our analysis of 31 available PIPs, these state plans have
focused to some extent on revising or developing policies; reviewing and
reporting on agency performance; improving information systems; and

20At the time of our analysis, only Delaware and North Carolina had
completed the 2-year term of their PIPs, and to date, ACF is still
analyzing the states' progress and has not determined if there has been
overall improvement or if it will apply financial penalties.

engaging stakeholders such as courts, advocates, foster parents, private
providers, or sister agencies in the public sector.21 Table 2 shows the
number of states that included each of the six categories and
subcategories of strategies we developed for the purposes of this study,
and appendix I details our methodology for this analysis.

21Although 41 states were developing or implementing PIPs at the time of
publication, we reviewed the 31 available PIPs that ACF had approved as of
January 1, 2004.

Table 2: Number of States Including Each of the PIP Strategy Categories
Used in This Study

PIP strategy category Description (number of states that included the
strategy in their PIP)

Policies and procedures	Review, modify, or develop/implement any policy,
procedure, or case practice standard (31)

Enhance foster home/parent licensing standards (7)

Develop child and family assessment tools, such as protocols for
risk/safety determinations (28)

Identify and adopt any promising practices (19)

Data collection and analysis 	Review and report on agency performance
through self-assessments or internal audits/review (31)

Apply federal CFSR or similar process for internal statewide case reviews
(16)

Improve information and data collection systems (31)

Staff supports 	Train and develop caseworkers (through dissemination and
training on policy or through revisions to overall curriculum) (30)

Assess and monitor staff responsibilities, skills, or performance (24)

Recruit additional staff/retain staff (14)

Lower caseloads (11)

Increase caseworker pay (1) Foster parent supports/services and resources
for children and families Train and develop foster families'/providers'
skills and capacities (27)

Recruit and retain foster families (22)

Increase involvement of foster or birth families in case (18)

Expand service array for children and families (includes developing or
enhancing transportation systems to transport siblings and parents for
visits, creating one-stop centers for assistance, modifying visitation
services, and providing any additional support services) (27)

Engage stakeholders such as courts, advocates, foster homes, private
providers, or sister agencies in public sector, e.g., mental health (can
include consultation, training, or formal partnering to improve services
or placements) (31)

Create or improve monitoring of contracts with private providers to
enhance service delivery (includes development of performance based or
outcome-based contracts or other evaluations of provider performance) (25)

State legislative supports State request for legislative action to support
any of the above strategies (20)

Federal technical assistance 	State request technical assistance from ACF
or any resource center to support any of the above strategies (27)

Source: GAO analysis.

Our analysis found that every state's PIP has included policy revisions or
creation to improve programs and services. For example, to address
unsatisfactory performance in the prevention of repeat maltreatment,
California's PIP includes a policy change, pending legislative approval,
granting more flexibility to counties in determining the length of time
they spend with families to ensure child safety and improve family
functioning before closing cases. Additionally, 30 of the plans included
caseworker

training; 20 of the plans included requests for state legislative action;
and 27 of the plans included requests for federal technical assistance
from ACF or the resource centers. For example, California planned to ask
its legislature to allow the agency to consolidate standards for foster
care and adoption home studies, believing this would facilitate the
adoption of children in the state. In addition, New York worked with its
legislature to secure additional funding to improve the accessibility of
independent living and adoption services for children and families.

Our analysis also showed that many states approached PIP development by
building on state initiatives in place prior to the on-site review. Of the
42 surveyed states reporting in our survey on this topic, 30 said that
their state identified strategies for the PIP by examining ongoing state
initiatives. For example, local officials in New York City and state
officials in California told us that state reform efforts-borne in part
from legal settlements-have become the foundation for the PIP. In New
York, the Marisol case was one factor in prompting changes to the state's
child welfare financing structure. 22 Subsequently, the state legislature
established a quality enhancement fund. Much of this money today supports
strategies in New York's PIP, such as permanency mediation to support
family involvement in case planning and new tools to better assess
children's behavioral and mental health needs. California state officials
also informed us that state reform efforts initiated by the governor prior
to the CFSR, such as implementing a new system for receiving and
investigating reports of abuse and neglect and developing more early
intervention programs, became integral elements in the PIP.

22The plaintiffs in this class action lawsuit filed in 1995 were 11
children, all of whom were abused, sometimes severely. The plaintiffs
sought to restructure New York City's child welfare system and charged
that the city mishandled their cases and deprived them of their rights
under the state and federal constitutions as well as various state and
federal laws. Marisol A. v. Giuliani, 929 F. Supp. 662, 669 (S.D.N.Y.
1996). As part of its settlement with the parties, the state agreed to
monitor the city's staffing requirements and conduct audits of the city's
treatment of children in nine specified areas. Marisol A. v. Giuliani, 185
F.R.D. 152, 159-160 (S.D.N.Y. 1999).

Insufficient Guidance ACF has provided states with regulations and
guidance to facilitate PIP Hampered State Planning development, but some
states believe the requirements have been unclear. Efforts, but ACF Has
Some of the requirements for program improvement planning are outlined
Taken Steps to Clarify in the following table.

Expectations and Improve Technical Assistance

     Table 3: ACF Regulations and Guidance on Program Improvement Planning

Stage Requirements

Development of the States are required to develop a PIP to address all

PIP 	of the outcomes and systemic factors determined not to be in
substantial conformity as a result of a CFSR. Although ACF does not
require a specific format for the PIP, it does offer states a suggested
format, called the PIP matrix, to use.

Approval of the PIP 	States must include certain information in the PIP,
including measurable goals, action steps required to correct each
deficiency identified in the final report, anticipated dates of completion
for each action step, amount of progress the state will make toward
national standards, benchmarks to measure state progress implementing the
PIP, and technical assistance needs.

Implementation and ACF will monitor the state's progress in completing
monitoring of the PIP the provisions of the approved PIP by evaluating
Time frame

States must submit the PIP to ACF for approval within 90 calendar days
from the date that the state received written notification that it is not
operating in substantial conformity.

No time frame is specified for ACF review and approval of the PIP.
However, ACF guidance says that the regional office should give prompt
attention to the PIP when it is submitted and take the minimum amount of
time necessary to review it and respond to the state. If ACF notifies the
state that the PIP is unacceptable, the state has 30 calendar days to
resubmit the plan for approval.

States must submit status reports to ACF on a quarterly basis, and the PIP
must be implemented within 2 years.

quarterly status reports and reviewing the state's Annual Progress and
Services Report, which incorporates the state's progress in implementing
the PIP.

Source: ACF regulations at 45 C.F.R. 1355.35 and guidance.

Some states in our survey indicated that the guidance ACF had provided did
not clearly describe the steps required for PIP approval. In addition,
some state officials believe that even ACF's more recent efforts to
improve PIP guidance have also been insufficient. Of the 21 states
reporting on the PIP approval process in our survey, 6 states-4 reviewed
in 2001 and 2 reviewed in 2002-said that ACF did not clearly describe its
approval process, and another 8 states reported that ACF described the
approval process as clearly as it did unclearly. Further, several states
commented in our survey that several steps in the approval process were
unclear to them, such as how much detail and specificity the agency
expects the plan to include, what type of feedback states could expect to
receive, when states could expect to receive such feedback, and whether a
specific format was required. Officials in the states we visited echoed
survey

respondents' concerns with officials from 3 of the 5 states informing us
that ACF had given states different instructions regarding acceptable PIP
format and content. For example, California and Florida officials told us
that their program improvement plans had been rejected prior to final
approval, even though they were based on examples of approved plans that
regional officials had provided. In addition, California officials told us
that they did not originally know how much detail the regional office
expected in the PIP and believed that the level of detail the regional
office staff ultimately required was too high. Although some steps may be
duplicative, officials in California said that the version of their plan
that the region accepted included 2,932 action steps-a number these
officials believe is too high given their state's limited resources and
the 2-year time frame to implement the PIP.

ACF officials have undertaken several steps to clarify their expectations
for states and to improve technical assistance, but state responses to
this assistance have been mixed. For example, in 2002, 2 years after ACF
released the CFSR regulations and a procedures manual, ACF offered states
additional guidance and provided a matrix format to help state officials
prepare their plans. ACF officials told us the agency is also helping
states through a team approach to providing on-site technical assistance.
Under this approach, when ACF determines a state is slow in developing its
PIP, the agency sends a team of staff from ACF and resource centers to the
state to provide intensive on-site technical assistance. An official from
West Virginia who had received this team assistance reported that ACF's
support was very beneficial. Further, ACF has attempted to encourage state
officials to start developing program improvement plans before the final
report is released. To do so, the agency has provided training to state
officials and stakeholders almost immediately after the completion of the
on-site review. ACF has sent staff from the resource center for
Organizational Improvement to provide such training. An official from
Utah, however, reported that the resource center training on PIP
development had been general, and she wished the resource center staff had
better tailored its assistance and provided more examples of strategies
other states are pursuing to improve.

Analysis of state survey responses indicates that starting to develop
improvement plans early can make the 90-day time frame to prepare a PIP
seem adequate. Of 9 states reporting that they started developing their
PIP before or during the statewide assessment phase, 5 said that 90 days
was adequate. Nonetheless, 21 of 35 state survey respondents reported that
the 90-day time frame was insufficient. For example, one respondent
reported that 90 days is too short a time to perform tasks necessary for
developing a

program improvement plan, such as analyzing performance data, involving
diverse groups of stakeholders in planning decisions, and obtaining the
approval of state officials. Survey results indicate that increasing
numbers of states are developing their PIPs early in the CFSR process,
which may reflect ACF's emphasis on PIP development. The following figure
shows that of the 18 states reviewed in 2001, only 2 started developing
their PIPs before or during the statewide assessment phase. Among states
reviewed in 2003, this share increased to 5 of 9.

Figure 3: Variation in the Start of States' PIP Development

Evidence suggests that lengthy time frames for PIP approval have not
necessarily delayed PIP implementation, and ACF has made efforts to reduce
the time the agency takes to approve states' PIPs. For example, officials
in 3 of the 5 states we visited told us they began implementing new action
steps before ACF officially approved their plans because many of the
actions in their PIPs were already under way. In addition, according to
our survey, of the 28 states reporting on this topic, 24 reported that
they had started implementing their PIP before ACF approved it. Further,
our analysis shows that the length of time between the PIP due date, which

statute sets at 90 days after the release of the final CFSR report, and
final ACF PIP approval has ranged considerably-from 45 to 349 business
days. For almost half of the plans, ACF's approval occurred 91 to 179
business days after the PIP was due. As shown in figure 4, however, our
analysis indicates that ACF has recently reduced the time lapse by 46
business days between states' PIP due dates and ACF's PIP approval.

Figure 4: Average Time Lapse between States' PIP Due Dates and ACF's PIP
Approval

Notes: Data for states reviewed in 2003 are not included because only 2
states' PIPs had been approved by March 1, 2004.

We used the date that states are required to submit a PIP to calculate the
time lapse. Therefore, the time lapse is calculated from day 90 through
the date ACF ultimately approved the plans. States may have submitted
their plans prior to or after the 90-day time frame designated in
regulations, but ACF's contractor-charged with tracking CFSR-related
milestones-does not track the actual PIP submission dates.

The shorter time lapse for PIP approval may be due, in part, to the ACF's
emphasis on PIP development. According to one official, ACF has directed
states to concentrate on submitting a plan that can be quickly approved.
Another ACF official added that because of ACF's assistance with PIP
development, states are now submitting higher-quality PIPs that require
fewer revisions.

State and Federal Uncertainties Cloud PIP Implementation and Monitoring

Program improvement planning has been ongoing, but uncertainties have made
it difficult for states to implement their plans and ACF to monitor state
performance. Such uncertainties include not knowing whether state
resources are adequate to implement the plans and how best to monitor
state reforms. In answering a survey question about PIP implementation
challenges, a number of states identified insufficient funding, staff, and
time-as well as high caseloads-as their greatest obstacles. Figure 5
depicts these results.

Figure 5: Most Common Challenges Affecting States' PIP Implementation

Note: This is based on responses from 25 states. The results reported in
the figure are a sum of the states reporting that the issue was a
challenge to PIP implementation to a very great extent, great extent,
moderate extent, or some/little extent. States not included answered no
extent, no basis to judge, or not applicable.

In regards to funding, an official from Pennsylvania commented that
because of the state's budget shortfall, no additional funds were
available for the state to implement its improvement plan, so most
counties must improve outcomes with little or no additional resources. A
Massachusetts official reported that fiscal problems in his state likely
would lead the state to lay off attorneys and caseworkers and to cut
funding for family support programs. While state officials acknowledged
that they do not have specific estimates of PIP implementation expenses
because they have not tracked this information in their state financial
systems, many states indicated that to cope with financial difficulties,
they had to be creative and use resources more efficiently to fund PIP
strategies. Of the 26 states responding to a question in our survey on PIP
financing, 12 said that they were financing the PIP strategies by
redistributing current funding, and 7 said that they were using no-cost
methods. In an example of the latter,

Oklahoma officials reported pursuing in-kind donations from a greeting
card company so that they could send thank-you notes to foster parents,
believing this could increase foster parent retention and engagement.

States also reported that PIP implementation has been affected by staff
workloads, but these comments were mixed. In Wyoming, for example,
caseworkers told us that their high caseloads would prevent them from
implementing many of the positive action steps included in their
improvement plan. In contrast, Oklahoma caseworkers told us that the
improvement plan priorities in their state-such as finding permanent homes
for children-have helped them become more motivated, more organized, and
more effective with time management. For example, one caseworker explained
that she is quicker now at locating birth fathers who were previously
uninvolved in the child's life because she uses the Internet to search for
these fathers' names. She said this new way of exploring leads and
information-a strategy that stemmed from PIP development- has been
motivating and rewarding because it has decreased the time spent tracking
down paternal relatives and increased the number of available placements
for the child.

ACF officials expressed uncertainty about how best to monitor states'
progress and apply estimated financial penalties when progress was slow or
absent, and 3 of the 5 states we visited reported frustration with the
limited guidance ACF had provided on the PIPs quarterly reporting process.
For example, 4 regional offices told us that they did not have enough
guidance on or experience with evaluating state quarterly reports. Some
regional offices told us they require states to submit evidence of each
PIP action step's completion, such as training curricula or revised
policies, but one ACF official acknowledged that this is not yet standard
procedure, although the agency is considering efforts to make the
quarterly report submission procedures more uniform. Moreover, ACF staff
from one region told us that because PIP monitoring varies by region, they
were concerned about enforcing penalties. Finally, shortly before
California's quarterly report was due, state officials told us they still
did not know how much detail to provide, how to demonstrate whether they
had completed certain activities, or what would happen if they did not
reach the level of improvement specified in the plan.

Based on data from the states that have been reviewed to date, the
estimated financial penalties range from a total of $91,492 for North
Dakota to $18,244,430 for California, but the impact of these potential
penalties remains unclear. While ACF staff from most regional offices told
us that potential financial penalties are not the driving force behind
state

ACF's Focus Rests Almost Exclusively on Implementing the CFSR

reform efforts, some contend that the estimated penalties affect how
aggressively states pursue reform in their PIPs. For example, regional
office staff noted that one state's separate strategic plan included more
aggressive action steps than those in its PIP because the state did not
want to be liable for penalties if it did not meet its benchmarks for
improvement. State officials also had mixed responses as to how the
financial penalties would affect PIP implementation. An official in
Wyoming said that incurring the penalties was equivalent to shutting down
social service operations in one local office for a month, while other
officials in the same state thought it would cost more to implement PIP
strategies than it would to incur financial penalties if benchmarks were
unmet. Nevertheless, these officials also said that while penalties are a
consideration, they have used the CFSR as an opportunity to provide better
services. One official in another state agreed that it would cost more to
implement the PIP than to face financial penalties, but this official was
emphatic in the state's commitment to program improvement.

To implement the CFSRs, ACF has focused its activities almost entirely on
the four phases of the review process. However, staff in several regions
report limitations in providing assistance to states in helping them to
meet key federal goals. Although regional staff conduct site visits to
states for reasons beyond the CFSR process, conducting the CFSR on-site
reviews and providing PIP-related assistance to states account for the
majority of regions' time and travel budgets, according to ACF officials.
Further, regional office staff said that more frequent visits with state
personnel- visits outside of the CFSR process in particular-would allow
them to better understand states' programs and cultivate relationships
with state officials. In addition, state officials in all five of the
states we visited said that ACF technical assistance needed improvement,
acknowledging that in some cases regional office staff were stretched thin
by CFSR demands and in other cases that assistance from resource center
staff lacked focus. While ACF officials in the central office said that
the CFSR has become the primary method for evaluating states' performance,
they acknowledged that regional staff might still be adjusting to the new
way ACF oversees child welfare programs. Further, they told us that ACF is
currently reevaluating the entire structure of its training and technical
assistance, in part to address these concerns.

ACF officials told us that the learning opportunities in the Children's
Bureau are intentionally targeted at the CFSR, but staff in 3 regions told
us that this training should cover a wider range of subjects-including
topics outside of the CFSR process-so that regional officials could better
meet

states' needs. All 18 of the courses that ACF has provided to its staff
since 2001 have focused on topics such as writing final CFSR reports and
using data for program improvement. ACF officials in the central office
said that the course selection reflects both the agency's prioritization
of the CFSR process and staff needs. To ascertain staff training needs,
ACF surveyed regional staff in October 2002, and ACF officials told us
they used the survey results in deciding which courses to offer. Our
analysis of this survey, however, showed that it focused only on training
topics directly related to the CFSR, so it might offer only limited
information on whether regional officials wanted training on other topics.
Specifically, the survey asked staff to check their top 5 training choices
from among 11 CFSR-related topics. While survey respondents were also
given the opportunity to write in additional training tropics they
desired, only 2 of the survey's 27 respondents did so.23 One indicated a
greater need for training on Indian child welfare issues, and another
expressed a desire to learn more about the entire child welfare system.
Although it is not possible to determine whether more respondents would
have prioritized non-CFSR training areas had the survey been designed to
elicit such information, our interviews with regional staff suggest that
some of them wish to obtain additional non-CFSR training. For example, a
staff member from one region told us she has not been adequately trained
in child welfare and believed that her credibility was damaged when a
state wanted advice that she could not provide on how to help older youth
prepare to exit from foster care.

In addition to offering training, ACF organizes biennial conferences for
state and federal child welfare officials. Nonetheless, staff from 5
regions told us that they wanted more substantive interaction with their
ACF colleagues, such as networking at conferences, to increase their
overall child welfare expertise. Staff from 6 of the 10 regions told us
that their participation in conferences is limited because of funding
constraints.

Further, staff in all 10 regions provide ongoing assistance or ad hoc
counseling to states, but staff from 6 regions told us they would like to
conduct site visits with states more regularly to improve their
relationships with state officials and provide more targeted assistance.
For example, staff in most regions told us that they assist states
predominantly by e-mail and telephone on topics such as interpretation of
Title IV-E eligibility criteria. Additionally, staff in 7 regions said
that they sometimes

23ACF could not provide a response rate.

visit with states to participate in state planning meetings-part of the
annual child and family services planning effort-or to give presentations
at state conferences on topics such as court improvement. However, staff
in 4 regions felt their travel funds were constrained and explained that
they try to stretch their travel dollars by addressing states' non-CFSR
needs, such as court improvements, during CFSR-related visits. While an
ACF senior official from central office confirmed that CFSR-related travel
constituted 60 percent of its 2002 child welfare-monitoring budget, this
official added that CFSR spending represents an infusion of funding rather
than a reprioritization of existing dollars and stated that regional
administrators have discretion over how the funds are allocated within
their regions. In addition, the same official stated that he knew of no
instance in which a region requested more money for travel than it
received.

Concerns from state officials in all 5 of the states we visited echoed
those of regional office staff and confirmed the need for improvements to
the overall training and technical assistance structure, while
respondents' comments on our survey showed more mixed perceptions on the
quality of assistance they received. For example, state officials in New
York and Wyoming commented that ACF staff from their respective regional
offices did not have sufficient time to spend with them on CFSR matters
because regional staff were simultaneously occupied conducting reviews in
other states. Further, an Oklahoma state official commented that
assistance from one resource center was not as specific or helpful as
desired. Specifically, when the state asked the resource center to provide
a summary of other states' policies regarding the intake of abuse and
neglect allegations, the resource center did not provide an analysis of
sufficient depth for the state to explore possible reforms.

According to state survey respondents, however, satisfaction with the
training and technical assistance provided by regional offices varied by
CFSR phase. For example, among states reviewed in 2001, 2002, and 2003,
satisfaction was generally highest in the statewide assessment phase, but
then dropped during on-site review and PIP development before rising again
in the PIP implementation phase. Across all phases of the CFSR process,
however, states reviewed in 2003 had much higher levels of satisfaction
with regional office assistance than those states reviewed in 2001,
suggesting improvements to regional office training and technical
assistance as the process evolved. Further, based on survey data and our
follow-up calls with selected states, satisfaction was also mixed in
regard to resource center provided assistance. For example, among states
reporting in our survey on the quality of assistance provided by the

resource center for Organizational Improvement and the resource center for
Information Technology-the two resource centers that provide specific
support to states regarding data issues and PIP development- satisfaction
was generally lower in every phase among states reviewed in 2003 than
among states reviewed in 2001. The only exception to this was during the
PIP development phase, for which states reviewed in 2003 reported higher
levels of satisfaction with the resource center for Organizational
Improvement than states reviewed in 2001, suggesting positive responses to
the on-site training and technical assistance this resource center has
recently been providing to aid states in their PIP planning efforts.24

ACF officials told us the CFSR has become the agency's primary mechanism
for monitoring states and facilitating program improvement, but they
acknowledged that regional office staff might not have realized the full
utility of the CFSR as a tool to integrate all existing training and
technical assistance efforts. Further, according to ACF officials,
meetings to discuss a new system of training and technical assistance are
ongoing, though recommendations were not available at the time of
publication. Levels of resource center funding, the scope and objectives
of the resource centers' work, and the contractors who operate the
resource centers are all subject to change before the current cooperative
agreements expire at the close of fiscal year 2004.25

Conclusions 	ACF and the states have devoted considerable resources to the
CFSR process, but concerns remain regarding the validity of some data
sources and the limited use of all available information to determine
substantial conformity. Further, no state to date has passed the threshold
for substantial conformity on all CFSR measures. The majority of states
surveyed agreed that CFSR results are similar to their own evaluation of

24According to interviews with ACF regional staff, only one region is
tracking states' satisfaction with the resource centers systematically.
While some regions conduct ad hoc conversations with state officials after
a resource center has provided assistance, ACF does not have a formal
mechanism for collecting or analyzing states' views on the quality of
assistance provided.

25Funding levels for each resource center have varied over their 5-year
cooperative agreements, but our analysis of 1999-2003 data shows that
since the implementation of the CFSRs, the funding for the resource center
for Information Technology and the resource center for Organizational
Improvement-the two resource centers that provide specific support to
states regarding data issues and PIP development-has consistently and
substantially increased.

areas needing improvement. However, without using more reliable data- and
in some cases, additional data from state self-assessments-to determine
substantial conformity, ACF may be over-or underestimating the extent to
which states are actually meeting the needs of the children and families
in their care. These over-or underestimates can, in turn, affect the scope
and content of the PIPs that states must develop in response. As states
face difficult budget decisions, accurate performance information could be
critical to deciding how best to allocate resources. We previously
reported on the reliability of state-reported child welfare data and
recommended that HHS consider additional ways to enhance the guidance and
assistance offered to states to help them overcome the key challenges in
collecting and reporting child welfare data. In response to this
recommendation, HHS said that ACF has provided extensive guidance on how
states can improve the quality of their data and acknowledged that
additional efforts were under way.

In addition, the PIP development, approval, and monitoring processes
remain unclear to some, potentially reducing states' credibility with
their stakeholders and straining the federal/state partnership. Similarly,
regional officials are unclear as to how they can accomplish their various
training and technical assistance responsibilities, including the CFSR.
Without clear guidance on how to systematically prepare and monitor
PIP-related documents, and how regional officials can integrate their many
oversight responsibilities, ACF has left state officials unsure of how
their progress over time will be judged and potentially complicated its
own monitoring efforts.

To ensure that ACF uses the best available data in measuring state
performance, we recommend that the Secretary of HHS expand the use of
additional data states may provide in their statewide assessments and
consider alternative data sources when available, such as longitudinal
data that track children's placements over time, before making final CFSR
determinations.

In addition, to ensure that ACF regional offices and states fully
understand the PIP development, approval, and monitoring processes, and
that regional offices fully understand ACF's prioritization of the CFSR as
the primary mechanism for child welfare oversight, we recommend that the
Secretary of HHS take the following two actions:

Recommendations for Executive Action

o  	issue clarifying guidance on the PIP process and evaluate states' and
regional offices' adherence to this instruction, and

o

Agency Comments

provide guidance to regional offices explaining how to better integrate
the many training and technical assistance activities for which they are
responsible, such as participation in state planning meetings and the
provision of counsel to states on various topics, with their new CFSR
responsibilities.

We received comments on a draft of this report from HHS. These comments
are reproduced in appendix IV. HHS also provided technical clarifications,
which we incorporated where appropriate.

HHS generally agreed with our findings and noted that the CFSR process has
already focused national attention on child welfare reform, but because
the CFSR is the first review of its kind, HHS is engaged in continuous
monitoring and improvement of the process. However, in its technical
comments, HHS commented that while it acknowledges that the CFSR is its
top priority, it disagreed with our statement that HHS's focus rests
exclusively on implementing the CFSR, stating that the Administration for
Children and Families (ACF) also conducts other oversight efforts, such as
Title IV-E eligibility reviews and AFCARS assessments. While we
acknowledged ACF's other oversight activities in the background section of
the report, this report focuses primarily on the CFSR and we reflected the
comments that ACF officials made throughout the course of our work that
the CFSR was the primary tool for monitoring state performance and that it
served as the umbrella for all monitoring activities undertaken by central
and regional ACF staff.

HHS further noted in its technical comments that we were wrong to suggest
that federal staff do not know how to monitor state PIPs or assess
financial penalties. However, we do not report that ACF is unsure of how
to monitor PIPs or how to assess financial penalties-rather, we reported
that ACF regional staff have not received sufficient guidance on how to
best monitor PIPs and that ACF officials have not decided how or when to
apply such penalties, even though two states to date have completed their
initial PIP implementation timeframe and all states reviewed thus far are
engaged in PIP development and implementation.

With regard to our first recommendation, HHS acknowledged that several
steps are under way to address necessary data improvements and said that
states have begun to submit more accurate information in their AFCARS and
NCANDS profiles, with HHS's assistance. HHS also commented that we failed
to properly emphasize the states' responsibility to improve overall data
quality. We believe that our report, as well as our previous

report on child welfare data and states' information systems, addresses
HHS's activities and the steps many states have taken to enhance their
CFSR data. Given that many states have developed independent data
collection tools-and included findings from these instruments in their
statewide assessments-our recommendation is meant to encourage HHS to work
more closely with all states to supplement their AFCARS and NCANDS data in
order to improve the determinations made about state performance.

In addition, HHS commented that our report emphasized the limitations of
the 50 case sample size without focusing on the expenses and the increased
state and federal staff time that would likely be associated with efforts
to increase the sample size. We agree that additional expenses and staff
time would likely be needed to increase the sample size and recommended
that ACF use additional data-beyond the information collected from the 50
case reviews and ACFARS and NCANDS data-to develop a more accurate picture
of state performance. This information could include the data that many
states already collect on their performance, such as longitudinal
information tracking children from their time of entry into the system.

In response to our second recommendation, HHS said that it has continued
to provide technical assistance and training to states and regional
offices, when appropriate. HHS noted that it is committed to continually
assessing and addressing training and technical assistance needs. In this
context, our recommendation was intended to encourage HHS to enhance
existing training efforts and focus both on state and on regional
officials' needs in understanding and incorporating the CFSR process into
their overall improvement and oversight efforts.

We also provided a copy of our draft report to child welfare officials in
the five states we visited-California, Florida, New York, Oklahoma, and
Wyoming. We received comments from California, Florida, New York, and
Oklahoma, all of which generally agreed with our findings and provided
various technical comments, which we also incorporated where appropriate.

We are sending copies of this report to the Secretary of Health and Human
Services, state child welfare directors, and other interested parties. We
will make copies available to others on request. In addition, the report
will be available at no charge on GAO's Web site at http://www.gao.gov. If
you or your staff have any questions or wish to discuss this material
further,

please call me at (202) 512-8403 or Diana Pietrowiak at (202) 512-6239.
Key contributors to this report are listed in appendix IV.

Cornelia M. Ashby, Director, Education, Workforce, and Income Security
Issues

List of Congressional Requesters

The Honorable Tom DeLay
Majority Leader
House of Representatives

The Honorable Charles E. Grassley
Chairman, Committee on Finance
United States Senate

The Honorable George Miller
Ranking Democratic Member
Committee on Education and the Workforce
House of Representatives

The Honorable Charles B. Rangel
Ranking Democratic Member
Committee on Ways and Means
House of Representatives

The Honorable Fortney Pete Stark
Ranking Democratic Member
Subcommittee on Health
Committee on Ways and Means
House of Representatives

The Honorable Benjamin L. Cardin
Ranking Democratic Member
Subcommittee on Human Resources
Committee on Ways and Means
House of Representatives

Appendix I: Objectives, Scope, and Methodology

Objectives 	The objectives of our study were to report on (1) ACF's and
the states' experiences preparing for and conducting the statewide
assessments and on-site reviews; (2) ACF's and the states' experiences
developing, funding, and implementing items in the PIP; and (3) additional
efforts that ACF has taken beyond the CFSR to ensure that all states meet
federal goals of safety, permanency, and well-being.

Scope and	To gather information about ACF's and the states' experiences
with the CFSR and PIP process, we utilized multiple methodologies to
solicit

Methodology 	information from both ACF and the states, including (1) a
Web-based survey to state child welfare agencies; (2) site visits to five
states; (3) a content analysis of all 31 PIPs available as of January 1,
2004; (4), interviews with ACF officials in Washington and all regional
offices, directors of all resource centers, and child welfare experts
nationwide; and (5) a review of CFSR regulations and the available
guidance offered to states. We conducted our work between May 2003 and
February 2004 in accordance with generally accepted government auditing
standards.

Survey 	To gather information about states' experiences with each phase of
the CFSR and PIP process, we distributed a Web-based survey to all 50
states, the District of Columbia, and Puerto Rico on July 30, 2003. We
pretested the survey instrument with officials in the District of
Columbia, Kentucky, and Maryland; and after extensive follow-up, we
received survey responses from all 50 states and the District of Columbia
for a 98 percent response rate.1 We did not independently verify the
information obtained through the survey.

The survey asked a combination of questions that allowed for open-ended
and close-ended responses. Because some states had not yet begun their
statewide assessments and others had already submitted quarterly PIP
progress reports at the time that our survey was released, the instrument
was designed with skip patterns directing states to comment only on the
CFSR stages that they had begun or completed to that point. Therefore, the
number of survey respondents for each question varied depending on the
number of states that had experienced that stage of the CFSR and PIP
processes.

1Puerto Rico did not respond to our survey.

Appendix I: Objectives, Scope, and Methodology

To supplement the survey and elaborate on survey responses, we selected 10
states with which to conduct follow-up phone calls based on their answers
to the survey's open-ended questions.2 These calls helped us obtain more
specific examples about states' experiences preparing for the CFSR;
developing, funding, and implementing a PIP; and working with ACF to
improve their child welfare systems.

Because this was not a sample survey, there are no sampling errors.
However, the practical difficulties of conducting any survey may introduce
errors, commonly referred to as nonsampling errors. For example,
difficulties in how a particular question is interpreted, in the sources
of information that are available to respondents, or in how the data are
entered into a database or were analyzed, can introduce unwanted
variability into the survey results. We took steps in the development of
the questionnaire, the data collection, and the data analysis to minimize
these nonsampling errors. For example, social science survey specialists
designed the questionnaire in collaboration with GAO staff with subject
matter expertise. Then, the draft questionnaire was pretested with a
number of state officials to ensure that the questions were relevant,
clearly stated, and easy to comprehend. When the data were analyzed, a
second, independent analyst checked all computer programs. Since this was
a Web-based survey, respondents entered their answers directly into the
electronic questionnaire. This eliminated the need to have the data keyed
into a database thus removing an additional source of error.

Site Visits

To gather more detailed information about the states' experiences with the
CFSR and PIP process, we selected five states to visit-California,
Florida, New York, Oklahoma, and Wyoming-based on the timing and results
of each state's CFSR, as well as their differences in location, size of
child welfare population, degree of privatization of services, size of
tribal populations, and whether they had state or locally administered
systems. In preparation for the visits and to understand the unique
circumstances in each state, we obtained and reviewed relevant literature
from each of the five states, such as the statewide assessment, the CFSR
final report, and any available PIPs or quarterly reports. Additionally,
we reviewed relevant

2The 10 states participating in our phone follow-up survey were Arkansas,
Iowa, Kansas, Mississippi, North Dakota, New Jersey, Pennsylvania, Rhode
Island, Utah, and West Virginia.

                 Appendix I: Objectives, Scope, and Methodology

past or current litigation that may affect the states' delivery of
services as identified by the National Center for Youth Law's Litigation
Docket (2002).

During our visits to each state, we talked with officials from the state
child welfare agency along with officials and staff from at least one
local agency office that was selected for the CFSR on-site review.
Specifically, in each state we spoke with state and local officials
responsible for guiding the states' efforts throughout the review process;
CFSR on-site reviewers; and stakeholders, including judges, child
advocates, private providers, foster parents, and child welfare staff.
Some detailed information regarding key CFSR milestones among the five
states we visited is included in appendix

III.

Content Analysis of Available PIPs

To learn about states' improvement strategies, we conducted a content
analysis of the 31 available PIPs that ACF had approved by January 1,
2004. For each of these PIPs, we classified the state's action steps as
relating to one or more of the following: policies and procedures, data
collection and analysis, staff supports, foster parent supports or
services and resources for children and families, state legislative
supports, and federal technical assistance. Table 2 in the report
summarizes how we classified the PIP strategies and indicates the number
of states including each strategy in its PIP.

Interviews

To gather information about ACF's experience with the CFSR and PIP
process, we interviewed ACF officials in Washington, D.C., who are
involved in the CFSR process and ACF staff in all 10 of the regional
offices; directors of each resource center; and ACF contractors working on
CFSR-related activities. Further, we observed the final debriefing
sessions for three states-South Carolina, Virginia, and Washington- during
the weeks of their respective on-site reviews.

In addition to our interviews with ACF officials, we also interviewed 10
prominent child welfare experts and researchers, such as those affiliated
with the Chapin Hall Center for Children, the Child Welfare League of
America, the National Coalition on Child Protection Reform, and the
University of California at Berkeley, to learn additional information
about the states' experiences with the CFSR process, including information
about states' concerns with the reliability of CFSR data, states'
involvement of tribes as stakeholders, and the media's coverage of
egregious child welfare cases.

                 Appendix I: Objectives, Scope, and Methodology

Review of ACF Guidance To gather information about CFSR regulations and
the available training

to States 	and technical assistance offered to states, we reviewed ACF's
regulations, its policy memorandums, and the CFSR manual it makes
available to states. In addition, we obtained and reviewed a list of all
of the resource centers' training and technical assistance activities
provided to the five states we visited during our site visits.

Appendix II: List of Outcomes and Systemic Factors and Their Associated
Items

Outcome factors

(Items with an asterisk have associated national standards.)

Child Safety

Outcome 1: Children are, first and foremost, protected from abuse and
neglect.

Item 1: Timeliness of initiating investigations on reports of child
maltreatment

Item 2: Repeat maltreatment

o  Recurrence of maltreatment*

o  Incidence of child abuse and/or neglect in foster care*

Outcome 2: Children are safely maintained in their own homes whenever
possible and appropriate.

Item 3: Services to family to protect child(ren) in home and prevent
removal

Item 4: Risk of harm to child(ren)

Permanency for Children

Outcome 3: Children have permanency and stability in their living
conditions.

Item 5: Foster care re-entries*
Item 6: Stability of foster care placement*
Item 7: Permanency goal for child
Item 8: Reunification, guardianship, or permanent placement with relatives

o  Length of time to achieve permanency goal of reunification*

Appendix II: List of Outcomes and Systemic Factors and Their Associated
Items

Item 9: Adoption

o  Length of time to achieve permanency goal of adoption*
Item 10: Permanency goal of other planned permanent living arrangement

Outcome 4: The continuity of family relationships and connections is
preserved for children.

Item 11: Proximity of foster care placement
Item 12: Placement with siblings
Item 13: Visiting with parents and siblings in foster care
Item 14: Preserving connections
Item 15: Relative placement
Item 16: Relationship of child in care with parents

Child and Family Well-being

Outcome 5: Families have enhanced capacity to provide for their children's
needs.

Item 17: Needs and services of child, parents, foster parents
Item 18: Child and family involvement in case planning
Item 19: Worker visits with child
Item 20: Work visits with parent(s)

Outcome 6: Children receive appropriate services to meet their educational
needs.

Item 21: Education needs of the child

 Appendix II: List of Outcomes and Systemic Factors and Their Associated Items

Outcome 7: Children receive adequate services to meet their physical and
mental health needs.

Item 22: Physical health of child Item 23: Mental health of child.

                                Systemic Factors

Systemic factor 1: Statewide information system

Item 24: State is operating a statewide information system that, at a
minimum, can identify the status, demographic characteristics, location,
and goals for the placement of every child who is (or within the
immediately preceding 12 months has been) in foster care.

Systemic factor 2: Case review system

Item 25: Provides a process that ensures that each child has a written
case plan to be developed jointly with the child's parent(s) that includes
the required provisions

Item 26: Provides a process for the periodic review of the status of each
child, no less frequently than once every 6 months, either by a court or
by administrative review.

Item 27: Provides a process that ensures that each child in foster care
under the supervision of the state had a permanency hearing in a qualified
court or administrative body no later than 12 months from the date the
child entered foster care and no less frequently than every 12 months
thereafter.

Item 28: Provides a process for termination of parental rights proceedings
in accordance with the provisions of the Adoption and Safe Families Act.

Item 29: Provides a process for foster parents, pre-adoptive parents, and
relative caregivers of children in foster care to be notified of, and have
an opportunity to be heard in, any review or hearing held with respect to
the child.

Appendix II: List of Outcomes and Systemic Factors and Their Associated
Items

Systemic factor 3: Quality assurance system

Item 30: The state has developed and implemented standards to ensure that
children in foster care are provided quality services that protect the
safety and health of children.

Item 31: The state is operating an identifiable quality assurance system
that is in place in the jurisdictions where the services included in the
Child and Family Services Plan are provided, evaluates the quality of
services, identifies strengths and needs of the service delivery system,
provides relevant reports, and evaluates program improvement measures
implemented.

Systemic factor 4: Training

Item 32: The state is operating a staff development and training program
that supports the goals and objectives in the Child and Family Services
Plan, addresses services provided under Titles IV-B and IV-E, and provides
initial training for all staff who deliver these services.

Item 33: The state provides for ongoing training for staff that addresses
the skills and knowledge base needed to carry out their duties with regard
to the services included in the Child and Family Services Plan.

Item 34: The state provides training for current or prospective foster
parents, adoptive parents, and staff of state-licensed or approved
facilities that care for children receiving foster care or adoption
assistance under Title IV-E that addresses the skills and knowledge base
needed to carry out their duties with regard to foster and adopted
children.

Systemic factor 5: Service array

Item 35: The state has in place an array of services that assess the
strengths and needs of children and families and determine other service
needs, address the needs of families in addition to individual children in
order to create a safe home environment, enable children to remain safely
with their parents when reasonable, and help children in foster and
adoptive placements achieve permanency.

Item 36: The services in item 35 are accessible to families and children
in all political jurisdictions covered in the State's Child and Family
Services Plan.

Appendix II: List of Outcomes and Systemic Factors and Their Associated
Items

Item 37: The services in item 35 can be individualized to meet the unique
needs of children and families served by the agency.

Systemic factor 6: Agency responsiveness to the community

Item 38: In implementing the provisions of the Child and Family Services
Plan, the state engages in ongoing consultation with tribal
representatives, consumers, services providers, foster care providers, the
juvenile court, and other public and private child- and family-serving
agencies and includes the major concerns of these representatives in the
goals and objectives of the Child and Family Services Plan.

Item 39: The agency develops, in consultation with these representatives,
annual reports of progress and services delivered pursuant to the Child
and Family Services Plan.

Item 40: The state's services under the Child and Family Services Plan are
coordinated with services or benefits of other federal or federally
assisted programs serving the same population.

Systemic factor 7: Foster and adoptive parent licensing, recruitment, and
retention

Item 41: The state has implemented standards for foster family homes and
child care institutions that are reasonably in accord with recommended
national standards.

Item 42: The standards are applied to all licensed or approved foster
family homes or child care institutions receiving Title IV-E or IV-B
funds.

Item 43: The state complies with federal requirements for criminal
background clearances as related to licensing or approving foster care and
adoptive placements and has in place a case-planning process that includes
provisions for addressing the safety of foster care and adoptive
placements for children.

Item 44: The state has in place a process for ensuring the diligent
recruitment of potential foster and adoptive families that reflect the
ethnic and racial diversity of children in the state for whom foster and
adoptive homes are needed.

Appendix II: List of Outcomes and Systemic Factors and Their Associated
Items

Item 45: The state has in place a process for the effective use of
crossjurisdictional resources to facilitate timely adoptive or permanent
placements for waiting children.

Appendix III: Dates on Which Site Visit States Reached CFSR Milestones

                               California  Florida New York Oklahoma  Wyoming 
    Date the state submitted         7/02     5/01     4/01     1/02     6/02 
    its statewide assessment                                         
Date ACF began the on-site     9/23/02   8/6/01  6/18/01  3/18/02   7/8/02 
             review                                                  
Date ACF released the final    1/10/03  4/23/02   1/9/02   7/1/02   4/1/03 
           CFSR report                                               
    Time lapse in release of      75 days 182 days 143 days  71 days 187 days 
          final report                                               
    Date the state's PIP was      4/10/03  7/22/02   4/9/02  9/29/02  6/30/03 
               due                                                   
    Date ACF approved the PIP     6/24/03  4/30/03  4/14/03  1/22/03   1/1/04 
Time lapse in PIP approval     54 days 203 days 265 days  83 days 134 days 
    Date the state submitted                                                  
its first quarterly report    10/29/03  7/30/03  8/11/03  5/15/03  4/30/04

Total number of outcomes and systemic factors not in 12 8 8 9

a

substantial conformity

    Estimated financial penalties $18,244,430 $3,639,552 $2,284,407 $727,300
                                    $137,369

Source: GAO analysis of ACF data.

aThe total number of outcomes and systemic factors evaluated in the CFSR
is 14.

Appendix IV: Comments from the Department of Health and Human Services

Appendix IV: Comments from the Department of Health and Human Services

Appendix IV: Comments from the Department of Health and Human Services

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: GAO Contacts and Acknowledgments

GAO Contacts 	Diana Pietrowiak, (202) 512-6239 Joy Gambino, (202) 512-2838

Staff In addition to those named above, Elizabeth Caplick and Catherine
Roark made key contributions to this report. Amy Buck, Karen
Burke,Acknowledgments Jason Kelly, Stuart Kaufman, Luann Moy, and Jerome
Sandau also provided key technical assistance.

Related GAO Products

Child Welfare: Improved Federal Oversight Could Assist States in
Overcoming Key Challenges. GAO-04-418T. Washington, D.C.: January 28,
2004.

D.C. Family Court: Progress Has Been Made in Implementing Its Transition.
GAO-04-234. Washington, D.C.: January 6, 2004.

Child Welfare: States Face Challenges in Developing Information Systems
and Reporting Reliable Child Welfare Data. GAO-04-267T. Washington, D.C.:
November 19, 2003.

Child Welfare: Enhanced Federal Oversight of Title IV-B Could Provide
States Additional Information to Improve Services. GAO-03-956. Washington,
D.C.: September 12, 2003.

Child Welfare: Most States Are Developing Statewide Information Systems,
but the Reliability of Child Welfare Data Could be Improved. GAO-03-809.
Washington, D.C.: July 31, 2003.

D.C. Child and Family Services: Key Issues Affecting the Management of Its
Foster Care Cases. GAO-03-758T. Washington, D.C.: May 16, 2003.

Child Welfare and Juvenile Justice: Federal Agencies Could Play a Stronger
Role in Helping States Reduce the Number of Children Placed Solely to
Obtain Mental Health Services. GAO-03-397. Washington, D.C.: April 21,
2003.

Foster Care: States Focusing on Finding Permanent Homes for Children, but
Long-Standing Barriers Remain. GAO-03-626T. Washington, D.C.: April 8,
2003.

Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare
Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March 31,
2003.

Foster Care: Recent Legislation Helps States Focus on Finding Permanent
Homes for Children, but Long-Standing Barriers Remain. GAO-02-585.
Washington, D.C.: June 28, 2002.

District of Columbia Child Welfare: Long-Term Challenges to Ensuring
Children's Well- Being. GAO-01-191. Washington, D.C.: December 29, 2000.

Related GAO Products

Child Welfare: New Financing and Service Strategies Hold Promise, but
Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.

Foster Care: States' Early Experiences Implementing the Adoption and Safe
Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.

Foster Care: HHS Could Better Facilitate the Interjurisdictional Adoption
Process. GAO/HEHS-00-12. Washington, D.C.: November 19, 1999.

Foster Care: Effectiveness of Independent Living Services Unknown.

GAO/HEHS-00-13. Washington, D.C.: November 10, 1999.

Foster Care: Kinship Care Quality and Permanency Issues. GAO/HEHS-99-32.
Washington, D.C.: May 6, 1999.

Juvenile Courts: Reforms Aim to Better Serve Maltreated Children.
GAO/HEHS-99-13. Washington, D.C.: January 11, 1999.

Child Welfare: Early Experiences Implementing a Managed Care Approach.
GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.

Foster Care: Agencies Face Challenges Securing Stable Homes for Children
of Substance Abusers. GAO/HEHS-98-182. Washington, D.C.: September 30,
1998.

GAO's Mission

Obtaining Copies of GAO Reports and Testimony

The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.

Order by Mail or Phone 	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud,	Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm

Waste, and Abuse in E-mail: [email protected]

Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470

Jeff Nelligan, Managing Director, [email protected] (202) 512-4800

Public Affairs 	U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***