Child Welfare: States Face Challenges in Developing Information
Systems and Reporting Reliable Child Welfare Data (19-NOV-03,
GAO-04-267T).
To better monitor children and families served by state child
welfare agencies, Congress authorized matching funds for the
development of statewide automated child welfare information
systems (SACWIS) and required that the Department of Health and
Human Services (HHS) compile information on the children served
by state agencies. This testimony is based on our July 2003
report and addresses the following: (1) states' experiences in
developing child welfare information systems and HHS's role in
assisting in their development, (2) factors that affect the
reliability of data that states collect and report on children
served by their child welfare agencies and HHS's role in ensuring
the reliability of those data, and (3) practices that child
welfare agencies use to overcome challenges associated with
SACWIS development and data reliability. For the July 2003
report, we surveyed all 50 states and the District of Columbia
regarding their experiences developing and using information
systems and their ability to report data to HHS. We also reviewed
a variety of HHS documents and visited five states to obtain
firsthand information. Finally, we interviewed HHS officials and
child welfare and data experts and reviewed relevant literature.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-267T
ACCNO: A08884
TITLE: Child Welfare: States Face Challenges in Developing
Information Systems and Reporting Reliable Child Welfare Data
DATE: 11/19/2003
SUBJECT: Children
Data collection
Data integrity
Federal aid to states
Federal/state relations
Management information systems
Child welfare
Colorado
HHS Adoption and Foster Care Analysis
and Reporting System
Iowa
NCCAN National Child Abuse and Neglect
Data System
New York
North Carolina
Oklahoma
Statewide Automated Child Welfare
Information System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-267T
United States General Accounting Office
GAO Testimony
Before the Subcommittee on Human Resources, Committee on Ways and Means,
House of Representatives
For Release on Delivery
Expected at 2:00 p.m. EST
Wednesday, November 19, 2003 CHILD WELFARE
States Face Challenges in Developing Information Systems and Reporting Reliable
Child Welfare Data
Statement of Cornelia M. Ashby, Director, Education, Workforce, and Income
Security Issues
GAO-04-267T
Highlights of GAO-04-267T, a testimony for the Subcommittee on Human
Resources, Committee on Ways and Means, House of Representatives
To better monitor children and families served by state child welfare
agencies, Congress authorized matching funds for the development of
statewide automated child welfare information systems (SACWIS) and
required that the Department of Health and Human Services (HHS) compile
information on the children served by state agencies. This testimony is
based on our July 2003 report and addresses the following: (1) states'
experiences in developing child welfare information systems and HHS's role
in assisting in their development, (2) factors that affect the reliability
of data that states collect and report on children served by their child
welfare agencies and HHS's role in ensuring the reliability of those data,
and (3) practices that child welfare agencies use to overcome challenges
associated with SACWIS development and data reliability. For the July 2003
report, we surveyed all 50 states and the District of Columbia regarding
their experiences developing and using information systems and their
ability to report data to HHS. We also reviewed a variety of HHS documents
and visited five states to obtain firsthand information. Finally, we
interviewed HHS officials and child welfare and data experts and reviewed
relevant literature.
www.gao.gov/cgi-bin/getrpt?GAO-04-267T.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Cornelia M. Ashby at (202)
512-8403 or [email protected].
November 19, 2003
CHILD WELFARE
States Face Challenges in Developing Information Systems and Reporting Reliable
Child Welfare Data
HHS reported that 47 states are developing or operating a SACWIS, but many
states continue to face challenges developing their systems. Most state
officials said they recognize the benefit their state will achieve by
developing SACWIS, such as contributing to the timeliness of child abuse
and neglect investigations; however, despite the availability of federal
funds since 1994, states reported a median delay of 2 1/2 years beyond the
time frames they set for completion. States reported that they encountered
some difficulties during SACWIS development, such as challenges receiving
state funding and creating a system that reflected their work processes.
In response to some of these challenges, HHS has provided technical
assistance to help states develop their systems and conducted on-site
reviews of SACWIS to verify that the systems meet federal requirements.
Despite efforts to implement comprehensive information systems, several
factors affect the states' ability to collect and report reliable
adoption, foster care, and child abuse and neglect data. States responding
to GAO's survey and officials in the five states GAO visited reported that
insufficient caseworker training and inaccurate and incomplete data entry
affect the quality of the data reported to HHS. In addition, states
reported technical challenges reporting data. Despite HHS assistance, many
states report ongoing challenges, such as the lack of clear and documented
guidance on how to report child welfare data. In addition, although states
were mandated to begin reporting data to the Adoption and Foster Care
Analysis Reporting System (AFCARS) in 1995, few reviews of states' AFCARS
reporting capabilities have been conducted to assist states in resolving
some of their reporting challenges.
Some states are using a variety of practices to address the challenges
associated with developing SACWIS and improving data reliability. For
example, 28 states reported using approaches to help caseworkers identify
and better understand the data elements that are required for federal
reporting.
In a related report, we recommended that the Secretary of HHS consider
ways to enhance the guidance and assistance offered to states to help them
overcome the key challenges in collecting and reporting child welfare
data. These efforts could include a stronger emphasis placed on conducting
AFCARS reviews and timelier follow-up to help states implement their
improvement plans or identifying a useful method to provide clear and
consistent guidance. HHS generally agreed with our findings but, in
response to our recommendation, said that we did not recognize the
longterm efforts to provide AFCARS and National Child Abuse and Neglect
Data System related guidance. HHS also noted that the data definitions
need to be updated and revised and said it was in the process of revising
regulations. HHS added that it is firmly committed to continue to support
the states and to provide technical guidance and assistance as resources
permit.
Mr. Chairman and Members of the Subcommittee:
Thank you for inviting me here today to discuss states' development of
automated child welfare information systems. As you are aware, the
Congress required that the Department of Health and Human Services (HHS)
compile information on the children served by state agencies and
authorized federal funds to match those of states for use in the
development of state child welfare information systems. Since 1994,
designated federal matching funds have been available to states to develop
and implement comprehensive case management systems-statewide automated
child welfare information systems (SACWIS)-to manage their child welfare
cases as well as to report child abuse and neglect, foster care, and
adoption information to the federal government. States have the option to
implement a SACWIS or develop different information systems without using
SACWIS funds to support their child welfare agencies and collect
information on their child welfare cases. Regardless of the type of system
a state develops, child welfare caseworkers at the county or local level
are the key personnel who collect and document information on children and
families served by child welfare agencies, in addition to performing a
wide range of services to protect children-such as investigating child
abuse or neglect reports or providing support services to maintain the
children in their homes.
Currently, HHS compiles state-reported child welfare data in two
databases: the Adoption and Foster Care Analysis and Reporting System
(AFCARS) and the National Child Abuse and Neglect Data System (NCANDS).
HHS relies on the information available in its databases to analyze and
track children's experiences in the child welfare system, to determine
states' performance on federal child welfare outcome measures, and to
report to Congress on children's well being and child welfare experiences.
My testimony today will focus on three key issues: (1) states' experiences
in developing child welfare information systems and HHS's role in
assisting in their development; (2) factors that affect the reliability of
data that states collect and report on children served by their child
welfare agencies, and HHS's role in ensuring the reliability of those
data; and (3) practices that child welfare agencies use to overcome
challenges associated with SACWIS development and data reliability. My
comments are based on the findings from our July 2003 report, Child
Welfare: Most States Are Developing Statewide Information Systems, but the
Reliability of Child Welfare Data Could Be Improved (GAO-03-809, July 31,
2003). Those findings were based on our survey of all 50 states and the
District of Columbia regarding their experiences in developing and using
information systems and their ability to report data to HHS. We received
responses from 49 states and the District of Columbia,1 although some
states did not respond to every question. We also reviewed a variety of
HHS documents, including the protocol and reports for its reviews of
SACWIS systems and states' AFCARS reporting capabilities and visited five
states-Colorado, Iowa, New York, North Carolina, and Oklahoma- to obtain
firsthand information on their experiences developing SACWIS and reporting
data to HHS. We selected these states to represent geographic diversity
and different stages of SACWIS implementation. Finally, we interviewed HHS
officials and child welfare and data experts and reviewed relevant
literature.
In summary, HHS reported that 47 states were developing or operating a
SACWIS, but many states continue to face challenges developing their
systems. Most state officials said they recognize the benefit their state
will achieve by developing SACWIS, but added that they have encountered
difficulties in receiving state funding and in creating a system that
reflected their work processes. Despite the availability of federal funds
since 1994, states reported a median delay of 2 1/2 years beyond the time
frames they set for completion. Several factors affect the states' ability
to collect and report reliable adoption, foster care, and child abuse and
neglect data. For example, insufficient caseworker training and inaccurate
and incomplete data entry affect the quality of data reported to HHS.
States also reported technical challenges reporting data. Despite HHS's
assistance, many states reported ongoing challenges, such as the lack of
clear and documented guidance from HHS on how to report child welfare
data. In addition, although states were mandated to begin reporting data
to AFCARS in 1995, few reviews of states' AFCARS reporting capabilities
have been conducted. Some states are using a variety of practices to
address the challenges they face in developing SACWIS and improving data
reliability. For example, 28 states reported using approaches to help
caseworkers identify and better understand the data elements that are
required for federal reporting. To improve the reliability of
state-reported child welfare data, we recommended in our July 2003 report
that the Secretary of HHS consider ways to enhance the guidance and
assistance
1Throughout this testimony, references to state survey responses include
the District of Columbia. Forty-six of these states reported that they are
developing or operating a SACWIS. Nevada, which HHS reported has an
operational SACWIS, did not respond to our survey.
offered to states to help them overcome the key challenges in collecting
and reporting child welfare data.
Background ACF's Children's Bureau is responsible for the administration
and oversight of federal funding to states for child welfare services
under Titles IV-B and IV-E of the Social Security Act. However, the
monitoring of children served by state child welfare agencies is the
responsibility of the state agencies that provide the services to these
children and their families. Child welfare caseworkers at the county or
local level are the key personnel responsible for documenting the wide
range of services offered to children and families, such as investigations
of abuse and neglect, treatment services offered to keep families intact
and prevent the need for foster care, and arrangements made for permanent
or adoptive placements when children must be removed from their homes.
Caseworkers are supported by supervisors, who typically assign new cases
to workers and monitor caseworkers' progress in achieving desired
outcomes, analyzing and addressing problems and making decisions about
cases.
To qualify for federal funding for SACWIS, states must prepare and submit
an advance planning document (APD) to the Children's Bureau, in which they
describe the state's plan for managing the design, development,
implementation, and operation of a SACWIS that meets federal requirements
and state needs in an efficient, comprehensive, and costeffective manner.
In addition, the state must establish SACWIS and program performance goals
in terms of projected costs and benefits in the APD. States are required
to submit separate APDs for the planning and development phases, in
addition to periodic updates.
Since the administration and structure of state child welfare agencies
vary across the nation, states can design their SACWIS to meet their state
needs, as long as states meet certain federal requirements. Federal
funding is available to states for SACWIS that
o meet the requirements for reporting AFCARS data to HHS;
o to the extent practicable, are capable of linking with the state data
collection system that collects information on child abuse and neglect;
o to the extent practicable, are capable of linking with, and retrieving
information from, the state data collection system that collects
information on the eligibility of individuals under Title IV-A-Temporary
Assistance for Needy Families; and
o provides for more efficient, economical, and effective administration
of the programs carried out under a state's plans approved under Titles
IV-B and IV-E of the Social Security Act.
A SACWIS must operate uniformly as a single system in each state and must
encompass all entities that administer programs provided under Titles IV-B
and IV-E. In some cases, HHS will allow the statewide system to link to
another state system to perform required functions, such as linking to
financial systems to issue and reconcile payments to child welfare service
providers. The state's APD must describe how its SACWIS will link to other
systems to meet the requirements in the SACWIS regulations.
In addition to monitoring the APDs of the states that are developing
SACWIS, HHS reviews state information systems through formal SACWIS
assessment reviews and the Child and Family Services Reviews (CFSR)-a
federal review process to monitor states' compliance with child welfare
laws and federal outcome measures. The formal SACWIS reviews are conducted
by the Children's Bureau to determine if a state has developed and
implemented all components detailed in the state's APD and if the system
adheres to federal requirements. The CFSR assesses statewide information
systems, along with other systemic factors, to determine if the state is
operating a system that can readily identify the status, demographic
characteristics, location, and goals for placement of every child who is
in foster care. This systemic factor is reviewed in all states, regardless
of whether the state is developing a SACWIS or the stage of system
development. For the 40 CFSR reports that are available, HHS found that
four states were not in substantial conformity on the statewide
information system indicator.2 These four states must address how they
will come into conformity with this factor in a program improvement plan.
HHS has also conducted SACWIS reviews in two of these states.
2We are currently conducting an engagement on states' and HHS's
experiences in conducting the CFSRs.
Most States Are Developing SACWIS, But Challenges Remain Despite HHS's
Oversight and Technical Assistance
While 47 states are developing or operating a SACWIS, many challenges
remain despite HHS's oversight and technical assistance. Since 1994,
states reported that they have spent approximately $2.4 billion in
federal, state, and local funding on SACWIS. While most state officials we
interviewed and those responding to our survey said that they recognize
the benefits their state will achieve by developing a statewide system,
many states reported that the development of their SACWIS is delayed
between 2 months and 8 years beyond the time frames the states set for
completion, with a median delay of 2 1/2 years. Most states responding to
our survey faced challenges, such as obtaining state funding and
developing a system that met the child welfare agency's needs statewide.
In response to some of these challenges, HHS has provided technical
assistance to help states develop their systems and conducted on-site
SACWIS reviews to verify that the systems meet all federal requirements.
States Are Using Federal and State Funds and Various Participants to
Develop Multicomponent SACWIS
Currently, 47 states are developing or operating a SACWIS and are in
various stages of development-ranging from planning to complete. The
states responding to our survey reported using approximately $1.3 billion
in federal funds3 and approximately $1.1 billion in state and local funds4
for their SACWIS. However, HHS estimated that it allocated approximately
$821 million between fiscal years 1994 and 2001 in SACWIS developmental
funds5 and $173 million between fiscal years 1999 and 2001 in SACWIS
operational funds.6 The total amount of federal funding provided to states
for SACWIS is unknown because states claimed operational costs as a part
3Forty-four states provided information on the total amount of federal
funds they received to develop and operate SACWIS. Alaska, Hawaii,
Missouri, North Carolina, Texas, and Vermont did not report federal
funding information. Nevada did not respond to our survey. State-reported
figures may include some funding allocated in fiscal year 2003, since the
survey was issued in October 2002 and completed as late as December 2002.
4Forty-four states provided information on the total amount of state funds
used to develop and operate SACWIS. Arkansas, Hawaii, Missouri, North
Carolina, Texas, and Vermont did not report state funding information.
Nevada did not respond to our survey. State-reported figures may include
some funding allocated in fiscal year 2003 since the survey was issued in
October 2002 and completed as late as December 2002.
5This figure includes developmental funds allocated by HHS to 49 states
and the District of Columbia. Hawaii did not take any federal money for
SACWIS development.
6This figure includes operational funds allocated to 35 states. States
begin claiming operational costs when some or all components of their
SACWIS are operating in local offices. Operational activities include
routine maintenance, minor enhancements, and other changes that do not
significantly increase or modify the functionality of the system.
of their Title IV-E administrative expenses prior to 1999.7 Although the
federal government matched state funding at an enhanced rate of 75 percent
beginning in 1994, many states did not apply for federal funding or begin
SACWIS development until 1996 or 1997, when more than $467 million-the
bulk of federal funds-were allocated. Most states were still developing
their SACWIS by the time enhanced funding expired in 1997, after which
states could receive a 50 percent federal financial participation for
SACWIS development and operation. Although 47 states are currently
developing or operating a SACWIS, all states except Hawaii received some
federal SACWIS funds. For example, according to figures provided by HHS,
North Carolina and North Dakota received some developmental funds but
encountered difficulties that prevented them from completing their
systems.
In order to track states' SACWIS development, HHS places them in six
categories that identify their stage of development (see table 1). HHS
sometimes recategorizes states into a lower stage of development when
problems are encountered. In addition, while HHS may classify a state
system as complete following an assessment of the state's SACWIS, a state
may make additional changes to the system since SACWIS, like other
computer systems, continually evolve as technology and child welfare
practices change. States can claim federal funding for these changes as
operational expenses. An HHS official reported that such changes do not
need prior approval unless they are in excess of $5 million.
7According to HHS officials, prior to fiscal year 2000, states reported
SACWIS operational expenses as part of their Title IV-E administrative
expenses because the claims sheet states used for reporting did not have a
separate column for SACWIS operational expenditures. In fiscal year 2000,
states were required to use a claims sheet that was reformatted to provide
space for SACWIS operational expenditures. In addition, an HHS official
explained that the difference between the state-reported figures and the
federal figures may be due to states claiming some SACWIS expenses under
different programs, such as Title IV-E administrative funds, rather than
separately as SACWIS expenses.
Table 1: Number of States in Various Stages of SACWIS Development
Stage Number of states
Completea
Operationalb
Partially operationalc
Implementationd
e
Planning
No SACWISf
Source: HHS.
Note: Status is as of October 13, 2003.
aThe SACWIS assessment process is completed, and all functional
requirements and specifications set forth in the APD are either included
in the system or in an accepted corrective action plan.
bAll functional requirements and specifications in the APD are included in
the system, and the system is functional statewide, but state has not
completed a SACWIS assessment or is working on other issues.
cThe state is still rolling out a system to field sites or still adding
functions to systems that are operational statewide.
dIn active design and development, even if delayed while waiting to
resolve problems such as funding.
eWorking through options for a SACWIS.
fHave never pursued SACWIS funding or have abandoned plans to develop a
system.
States have considerable flexibility in the design of their SACWIS.
According to HHS officials, a state should be using its SACWIS as a case
management tool that uses automation to support the various aspects of
state child welfare programs, such as recording child protection,
out-ofhome care, and foster care and adoption services. To further assist
child welfare practice, states have designed their systems to follow the
natural flow of child welfare practice in their state and have added
design features to help track key events during a case. For example, in
Iowa child welfare work is divided between child abuse and neglect
investigations and ongoing case management for children brought into the
care of the child welfare agency. As a result, Iowa designed a SACWIS to
reflect this work process by linking two databases-one to record child
abuse and neglect
information and one to record ongoing case records-that share information
with each other.8
Since many states are in different phases of SACWIS development, their
systems currently support to varying degrees a variety of child welfare
and administrative components (see table 2). According to HHS, while the
components listed in table 2 are required for a state's SACWIS to be
considered compliant with federal guidance-either through an interface or
built within the system-some of the subcomponents, such as a function that
helps caseworkers manage their caseloads, are optional. HHS has encouraged
states to automate as many functions as possible in the SACWIS in an
effort to cut down on the additional paperwork or duplicative steps
inherent in manual data collection.
8Although the Iowa state officials described their SACWIS as including the
child abuse and neglect system, HHS commented on a draft of the July 2003
report that it does not view the child abuse and neglect system as part of
the state's SACWIS. However, HHS said that the state has met the SACWIS
requirement in this area by building an interface between the two systems.
Table 2: Selected SACWIS Child Welfare and Administrative Services
Fully or partially operational in Planned Service SACWIS for SACWIS
Child welfare services
a
Child protection 38
b
Out-of-home care 35
Adoption 34
Independent living 27
Intensive home-based servicesc 27
Administrative services
Workload management 32
IV-E eligibilityd 29
Foster care maintenance payments 28
Adoption assistance payments 25
Contract provider payment 24
Source: GAO survey.
Note: This table is based on responses from 46 states developing or
operating a SACWIS. The rows for the columns "fully or partially
operational" and "planned" do not add to 46 because the respondents may
have answered "not supported," "don't know," or "no answer."
aChild protection includes services such as intake and screening,
investigation, and disposition.
bOut-of-home care includes things such as foster care, group homes, and
residential placement.
cIntensive home-based services include efforts to avoid placing a child in
foster care.
dIV-E funding is available for foster care, adoption, and independent
living services.
To assist with the design of their SACWIS, states relied on a number of
different participants, including internal users, such as caseworkers and
managers, information technology (IT) staff, and contractors. In Oklahoma,
for example, 150 child welfare staff from the field worked closely with
the contractor in intensive work group sessions to design and test the
system. To complement the caseworkers' knowledge of child welfare
practice, 43 states relied on IT staff. Finally, 42 states reported that
they hired private contractors to conduct a large part of SACWIS design
and development.
At the time of our review, HHS reported that four states were not pursuing
SACWIS development, and most of these states reported various reasons in
our survey for not developing a system. In Hawaii, for example, the child
welfare agency chose not to pursue SACWIS because it already had a
statewide system in place that it believed was adequately meeting its
needs and which was collecting and reporting federal child welfare data.
States Accrue Benefits from Using SACWIS, but Several Issues Create Delays
in Completing States' Systems
While most state child welfare agency officials said they recognize the
benefits the state will achieve by developing SACWIS, such as enhancing
their ability to track the whereabouts of foster children, 31 state
agencies lag behind the time frames they set for completion, with 26
states reporting delays ranging from 2 months to 8 years. According to
survey results, automated systems provided easier access to data and
allowed caseworkers to better monitor children in their care, a fact that
may contribute to additional child welfare and administrative benefits,
such as decreased incidences of child abuse and neglect, shortened length
of time to achieve adoption, timeliness of payments to foster families,
and timeliness of payments to foster facilities. New Jersey, which is in
the planning stage, reported in our survey that its goal in developing a
SACWIS is to integrate the more than 40 stand-alone systems that currently
capture information on the children served by their child welfare agency.9
By pulling all of these systems together into a uniform SACWIS, the state
hopes to improve the recording of casework activities in a timely manner
and to develop a tool to better target resources and services. Effectively
integrating these systems will require the state to use a disciplined IT
management approach that includes (1) detailed analyses of users' needs
and requirements, (2) a clearly defined strategy for addressing
information needs, and (3) sufficient technical expertise and resources to
support the effort.
Despite the benefits that many states have accrued with SACWIS, 31 states
reported in our survey that they have been delayed in system completion
beyond their initial deadline and identified a number of challenges that
have led to the delay (see table 3).10 Some of the common difficulties
states
9New Jersey reported in our survey that it had spent approximately $9
million in federal funds and $4 million in state and local funds on system
development. According to HHS, New Jersey first received federal funds in
1996.
10Twelve of the 46 states reporting that they are developing or operating
a SACWIS reported that they have not experienced delays in developing
their systems. In response to the length of the delays reported by 26
states in our survey, ACF commented on a draft of the July 2003 report
that these states may be using different definitions in defining their
delays. However, ACF did not provide further information on how the delays
represented in that report differ from its perception of states'
experiences. In our survey, we asked states to report on the delays that
exceeded the time line outlined in their initial APD.
reported in developing SACWIS included receiving state funding approval,
reaching internal agreement on system development, and creating a system
that reflects child welfare work processes and is user-friendly. Vermont
officials, for example, reported that the state legislature declined to
provide the matching state funds needed to secure federal funding for
SACWIS. As a result, the state could not pursue development.
Table 3: Number of Months States Delayed in SACWIS Development
State Length of delay in monthsa
Alabama
Arkansas
California
Colorado
Connecticut
District of Columbia
Georgia
Idaho
Illinois
Indiana
Kansas
Louisiana
Maryland
Michigan
Minnesota
Mississippi
New Jersey 42
New Mexico 3
Ohio 36
Oregon 70
Rhode Island 14
South Carolina 47
Tennessee 36
Utah 48
Virginia 2
Washington 36 Source: GAO survey.
Note: While 31 states reported in the survey that they have experienced a
delay in SACWIS development, only 26 states reported the length of their
delay. The survey was issued in October 2002 and completed by states as
late as December 2002.
aStates were asked to report the number of months the delays exceeded the
time line outlined in their APD.
Despite user involvement in system design, some states still faced
challenges trying to reach internal agreement among agency officials and
caseworkers on the design of a system, resulting in a delay in
development. In New York-a state where the counties are responsible for
administering child welfare services-the development of SACWIS was stalled
when significant frustration with the system's design led commissioners
from five large counties and New York City to request that the state stop
SACWIS development until a reassessment of the design of and plans for the
implementation of the system was completed.
Similarly, despite states' heavy reliance on contractors, many reported
that securing contractors with knowledge of child welfare practice was a
challenge for timely SACWIS development. Contractors are hired by the
state for their system development knowledge but often are unfamiliar with
child welfare policies and practices, especially since they vary from
state to state. A contractor who has worked with seven states to develop
their SACWIS reported that contractors are asked to learn the child
welfare business practices of a state in a short amount of time and that
states cannot devote many resources, such as caseworkers, to help in the
design process because caseworkers need to devote their time to providing
services to children and families.
Many states reported that creating a system that reflects child welfare
work processes and is user-friendly was a challenge in developing SACWIS.
These issues were also identified in the federal reviews of states'
SACWIS. For example, one state explained in the SACWIS review that it had
designed a system to meet the caseworkers' needs and reflect the nature of
the child welfare work processes by developing a system that required
events to be documented as they occurred. However, this design limited the
SACWIS's functionality because it did not allow the caseworkers to go back
and enter information after an event happened. The state explained that
caseworkers do not use the system in real time, but provide services to
the children and families and then record the information in the system.
The state had to redesign the system to correct for this design flaw.
HHS Provides Some Assistance to Help States Meet SACWIS Requirements
HHS has assisted states in a variety of ways in developing and completing
their SACWIS.11 As a part of its regulatory responsibilities, HHS must
review, assess, and inspect the planning, design, development,
installation, and operation of SACWIS. In addition to reviewing and
monitoring states' APDs, HHS conducts on-site SACWIS reviews to comply
with these responsibilities. HHS officials told us that these reviews are
a detailed and thorough assessment of state systems to ensure the systems'
compliance with SACWIS requirements. In addition, officials reported that
they provide technical assistance during the on-site review to help states
that do not fully conform with the applicable regulations and policies. As
of October 2003, HHS had reviewed 27 SACWIS-5 of which were determined as
meeting all the requirements and classified as complete. HHS officials
told us that since states have the flexibility to build a SACWIS that
meets their needs, a large portion of the formal reviews concentrate on
ensuring that the systems conform to state business practices. For
example, while SACWIS regulations require that a state report all AFCARS
data from their SACWIS, one state HHS reviewed relied on a separate state
system to report data on the children served by the juvenile justice
agency who are eligible for IV-E foster care funds. The state proved it
had developed an automated process to merge data from both systems to
compile a single AFCARS report that included children captured in both
their SACWIS and juvenile justice systems. Therefore, HHS recognized that
this process best met the state's needs and determined the SACWIS to be
complete and meeting all requirements.
Few systems have been determined complete after an on-site review because
of unresolved issues, such as not being able to build links to other state
information systems or not implementing certain eligibility determination
functions. To help states address some of these development challenges,
the SACWIS review team provides the state with recommendations for
complying with SACWIS requirements. In addition, HHS officials reported
that once the draft report with the results of the SACWIS review is
completed, federal staff schedule a conference call with
11With regard to the budget difficulties that states reported facing,
since 1994 the federal government has made a commitment to help states
develop and maintain their SACWIS by matching 75 percent of states'
development funds through 1997 and providing an ongoing match of 50
percent of state funding for the development and maintenance of their
systems. However, since the states' legislatures must make the initial
commitment to fund SACWIS, the federal government cannot assist state
child welfare agencies with this challenge.
the state officials to walk through the system's deficiencies and offer
guidance on how the state can move forward.
HHS facilitates the sharing of information between states developing
SACWIS through an automated system users' group that allows state and
federal officials to exchange information, ideas, and concerns. In
addition to the users' group, HHS officials also sponsor a Listserv-an
electronic mailing list-that allows state officials to exchange
information and a monthly conference call with state information
technology directors. 12 Technical assistance for SACWIS development is
also available to states through the National Resource Center for
Information Technology in Child Welfare (Resource Center), which opened in
1999. According to survey results, 9 states said they used the Resource
Center for assistance in developing SACWIS and 14 states reported using it
for help with SACWIS maintenance and improvements. According to Resource
Center officials, they assist states with SACWIS development by helping
states understand the technology that is available for use, providing
information on the automation of child welfare work and converting data,
and reviewing the APD documentation.
12In commenting on a draft of the July 2003 report, HHS indicated that a
Web resource is available to states interested in learning about other
states' efforts to develop human services-child welfare, food stamps,
Temporary Assistance to Needy Families, child care, and child support
enforcement-information systems at http://www.acf.hhs.gov/nhsitrc.
Several Factors Affect the States' Ability to Ensure Reliable Data on
Children's Experiences, and Some of HHS's Oversight and Assistance Is
Problematic
Several factors affect states' ability to collect and report reliable13
data on children served by state child welfare agencies, and some problems
exist, such as a lack of clear and documented guidance, with HHS's
oversight and technical assistance. Almost all of the states responding to
our survey reported that insufficient caseworker training and inaccurate
and incomplete data entry affect the quality of the data reported to
HHS.14 In addition, 36 of the 5015 states that responded to our survey
reported that technical challenges, such as matching their state data
element definitions to HHS's data categories, affected the quality of the
data that they report to the federal government. Despite the assistance
that HHS offers to states, such as testing state data quality and
providing the results to states to aid them in resubmitting data, states
report ongoing challenges receiving clear and documented guidance and
obtaining technical assistance.
Insufficient Caseworker Training and Inaccurate and Incomplete Data Entry
Are the Most Common Factors That Affect Data Reliability
Almost every state responding to our survey and all the states we visited
reported that insufficient training for caseworkers and inaccurate and
incomplete data entry affect the quality of the data reported to AFCARS
and NCANDS (see fig. 1). Although most states reported these as separate
factors, HHS and the states we visited found that insufficient training
and inaccurate and incomplete data entry are often linked. In official
reviews of states' information systems' capability to capture data and
report them to AFCARS, HHS advised states to offer additional training to
caseworkers on several AFCARS data elements, such as recording the reasons
for a child leaving foster care, to improve the accuracy of the data
submitted. However, state officials told us that training is typically one
of the first programs cut when states face tight budget restrictions. For
example, Iowa officials told us that training has been significantly
reduced in recent years because of budget cuts and new workers may wait 2
to 3 months before being trained how to enter data appropriately into
their SACWIS.
13Data are reliable when they are complete and accurate. A subcategory of
accuracy is consistency. Consistency refers to the need to obtain and use
data that are clear and well defined enough to yield similar results in
similar analysis. See U.S. General Accounting Office, Assessing the
Reliability of Computer-Processed Data, GAO-02-15G (Washington, D.C.:
Sept. 2002).
14States were asked the extent to which certain problems may decrease the
quality of the data submitted to AFCARS and NCANDS using the following
scale: very great, great, moderate, some, and no affect.
15The analysis of survey responses about reporting data to HHS is based on
responses from 49 states and the District of Columbia. All states,
regardless of SACWIS development, were asked to complete these questions.
Figure 1: Most Common Caseworker Issues That Affect Data Quality
States reporting at least some affect in data quality
50 49 49 46
40
30
20
10
0
Inaccurate data Insufficient Incomplete data entry by training for entry
by caseworkers caseworkers caseworkers
AFCARS
NCANDS
Source: GAO survey.
Notes: Based on responses from 50 states.
The results reported in the figure are a sum of the states that reported
the issue had a very great affect, great affect, moderate affect, or some
affect on the quality of state data submitted to HHS. Very great and great
affect responses are represented in the top section of each bar. Moderate
and some affect responses are represented in the bottom section of each
bar. States not included answered "no affect," "don't know," or "no
answer."
Inaccurate and incomplete data entry can also result from a number of
other factors, such as caseworkers' hesitation to ask families for
sensitive information. For example, caseworkers in Oklahoma reported that
they did not feel comfortable asking if a child's mother was married at
the time of birth or if a child is of Hispanic origin-both of which are
required AFCARS data elements. In commenting on a draft of this report,
Oklahoma added that caseworkers did not understand why the data elements
were required and how the federal government used the information. HHS
noted
similar issues in five states that have had an AFCARS review.16
Caseworkers were inaccurately recording a child's race as "unable to
determine" even though this option should be selected only if the child's
parents or relatives cannot provide the information, such as when a child
is abandoned.17
Caseworkers, supervisors, and managers in the 5 states we visited reported
that additional factors, such as difficulties balancing data entry with
the time that they spend with the families and children, contributed to
inaccurate or incomplete data entry. Supervisors in Iowa explained that
since caseworkers are responsible for ensuring that children and their
families receive the services they need, the caseworkers tend to initially
limit data entry to the information that is necessary to ensure timely
payment to foster care providers and complete all other data elements when
they have time. In addition, caseworkers in Colorado said that they are
between 30 and 60 days behind in their data entry, so the information in
the automated system may not accurately reflect the current circumstances
of children in care. HHS's Inspector General recently issued a report in
which more than two-thirds of the states reported that caseworkers'
workloads, turnover, a lack of training, and untimely and incomplete data
entry affected the reporting of AFCARS data.18
16For the July 2003 report, we reviewed AFCARS reports from six of the
eight states that had been assessed by HHS-Arkansas, Connecticut, New
Mexico, Texas, Vermont, and Wyoming. HHS conducted reviews in Delaware and
West Virginia after we completed our analysis. As of October 2003, HHS had
completed three additional reviews for North Dakota, Rhode Island, and
Washington.
17In commenting on a draft of the July 2003 report, ACF said that the
finding from the AFCARS reviews indicates that information is often
defaulted to the response "unable to determine" in order for the element
not to fail the missing data standard, not that workers are recording
"unknown"; however, the report findings we used in this analysis instruct
states to fix the defaults and address caseworker practice by enhancing
training on the correct use of "unable to determine" when noting a child's
race.
18Department of Health and Human Services, Office of Inspector General,
Adoption and Foster Care Analysis and Reporting System (AFCARS):
Challenges and Limitations,
OEI-07-01-00660 (Washington, D.C.: Mar. 2003).
Technical Challenges, such as Matching State Definitions to Federal
Definitions, Affect Data Reliability
In addition to data quality being affected by caseworker issues, many
states experienced technical challenges reporting their data to HHS. The
problems reported by states are typically a result of challenges
associated with data "mapping"-matching state data elements to the federal
data elements. For example, 36 states reported in our survey that matching
their state-defined data to HHS's definitions affected the quality of the
data reported to NCANDS and AFCARS. Similarly, 24 states reported that
matching the more detailed data options available in their states'
information systems to the federal data elements affected the quality of
the data reported to NCANDS. Twenty-nine states reported that this issue
created challenges in reporting data to AFCARS. For example, following an
AFCARS assessment, HHS instructed a state that collects detailed
information on children's disabilities, such as attention deficit disorder
and eating disorders, to map the information to the more limited options
in AFCARS, such as mental retardation and emotionally disturbed.
In many cases, states have to balance state policy with federal
requirements to ensure that they are reporting accurate data to AFCARS and
NCANDS, but are not contradicting their state policies. For example, Texas
officials reported that although the findings of their AFCARS review
instructed them to modify their SACWIS to collect, map, and extract data
on guardianship placements, the state does not support guardianship
arrangements.19 In addition, a recent report from the Child Welfare League
of America (CWLA) found that when reporting the number of times children
move from one foster care placement to another, states varied in the type
of placements included in that count.20 For example, 29 percent of the
states responding to CWLA's survey included respite,21 25 percent included
runaways, and 16 percent included trial home visits when reporting the
number of placements a child had during the AFCARS report period.
According to federal guidance, the "number of placements" element is meant
to gather information on the number of times the child welfare agency
found it necessary to move a child while in foster care and that by
including runaways or trial home visits, a state is inflating the number
of moves a child experienced.
19Guardianship arrangements occur when permanent legal custody of a child
is awarded to an individual, such as a relative, but the child is not
legally adopted.
20Child Welfare League of America. National Working Group Highlights,
"Placement Stability Measure and Diverse Out-of-Home Care Populations"
(Washington, D.C., Apr. 2002).
21Respite care provides temporary child care for children away from their
caretakers.
Although HHS Has Taken Steps to Help States Improve Their Data, Some
Problems with Its Efforts Exist
HHS provides technical assistance for AFCARS and NCANDS reporting through
a number of resources. HHS officials in the central office and NCANDS
contractor staff serve as the points of contact for states to ask
questions and seek guidance on reporting child welfare data. The officials
in three of the five states that we visited said that the one-on-one
focused technical assistance was useful when provided in a timely fashion.
Most state officials found the NCANDS data easier to report, in part
because more people were available for consultation and they were more
accessible and responsive. For example, states have access to four NCANDS
specialists and staff in the contractor's central office when they need
assistance reporting child abuse and neglect information. However, some of
the states we visited reported that only one or two staff in HHS's central
office are available to assist with AFCARS reporting.
In addition, the Resource Center offers states assistance with improving
data quality. However, Resource Center staff reported that the assistance
is geared more toward improving the limited data used in the federal
review process to monitor states' compliance with child welfare laws and
federal outcome measures-CFSR-rather than all the data reported to HHS.
The Resource Center also sponsors an annual information technology
conference during which sessions covering all data-related issues are
held, including practices for ensuring data quality and outcome evaluation
in child welfare. In conjunction with this conference, the HHS officials
and the contractors that operate NCANDS hold an annual technical
assistance meeting for states to share ideas with one another, discuss
data elements that pose difficulties, and explore ways to address these
problems. In addition, an NCANDS state advisory group meets annually to
talk with HHS officials about NCANDS data and their experiences reporting
data. From these meetings, the state advisory group proposes changes or
improvements to NCANDS. HHS and state officials reported that this
partnership has helped ease some of the challenges in reporting child
abuse and neglect data.
HHS has also made available to states the software it uses to examine
states' AFCARS and NCANDS submissions for inconsistencies and invalid
data. Officials in all the states we visited said that they regularly use
this software, and an HHS official said that nearly every state has used
the software at least once. When the data are submitted to HHS, they are
run through the same software, and HHS notifies the states of areas where
data are missing or inconsistent and allows the states to resubmit the
data after errors are corrected. HHS officials reported that these tests
help them to identify some data quality errors, such as missing data, and
said that they believe that, in general, data have improved in recent
years.
However, they indicated that the tests cannot pinpoint the underlying
problems contributing to these errors. Furthermore, one official reported
that no specific efforts have been conducted to track the individual data
elements, and therefore HHS cannot report on how data quality has changed
over time.
In an attempt to help states comply with the reporting standards and
address some of the factors that contribute to data quality problems, HHS
performs comprehensive reviews of state information systems' ability to
capture AFCARS data to identify problems associated with data collection
and reporting and to ensure that the information in the automated system
correctly reflects children's experiences in care. The assessments include
a technical review of the states' computer code, a comparison of the data
from selected cases available in the information system to the case files,
and an improvement plan to resolve any errors. In addition, HHS officials
offer guidance to the states on improvements that can be made to the
information system and changes to program code used to report the AFCARS
data. HHS conducted pilot reviews in eight states between 1996 and 2000.
By October 2003, HHS had conducted 11 official reviews-even though states
began reporting to AFCARS in 1995. According to results from 6 of the 11
official AFCARS assessments we reviewed, no state met the reporting
requirements for all AFCARS data elements. The problems noted in the
reviews are similar to those of states responding to our survey and those
we visited. For example, most states received ratings of 2 or 3,
indicating technical and/or data entry errors that affect the AFCARS data
quality.22 For the current placement setting data element,23 for instance,
4 states received a rating of 2, 1 state received a rating of 3, and 1
state received a rating of 4. In Connecticut, which received a rating of
2, HHS found that, among other things, workers were not consistently
entering placement information in a timely way. It also found that workers
entered
22HHS rates each data element using a four-point scale: (1) the AFCARS
requirement(s) has not been implemented in the information system; (2) the
technical system requirements for AFCARS reporting do not fully meet the
standards; (3) the technical system requirements for AFCARS reporting are
in place, but there are data entry problems affecting the quality of the
data; (4) all of the AFCARS requirements have been met. According to an
HHS official, data elements that have a combination of technical and data
entry problems are rated as 2 until the technical issues are resolved. HHS
will then rate the element as a 3 until the data entry practices are
changed.
23Current placement setting refers to a pre-adoptive home, foster family
home-relative, foster family home-nonrelative, group home, institution,
supervised independent living, runaway, or trial home visit.
placement data only into a narrative field, which resulted in placement
history gaps and incomplete AFCARS reports.
State officials in the six states for which we reviewed the HHS AFCARS
assessments reported that they found the reviews useful for improving
their AFCARS data submissions. In particular, they valued the thorough
review by HHS officials of the computer code states use to report the
data. Some of these officials reported that if all states were reviewed,
the quality of data available in AFCARS would improve tremendously.
However, HHS officials reported that they are not mandated to conduct the
AFCARS reviews and that priority is placed on other reviews, such as the
CFSR and SACWIS reviews. In addition, officials explained that the AFCARS
reviews are not conducted in states developing SACWIS until the systems
are operational. HHS expects to complete approximately four reviews each
year, depending on available resources, and has scheduled states through
2006. Similar to the SACWIS reviews, HHS officials offer recommendations
and technical assistance to states during the review on how they can
improve the quality of the data reported to AFCARS.
Although the states we visited appreciated some of HHS's efforts to assist
with improving state data quality, they and most states responding to our
survey agreed that the assistance is not always consistent or easily
accessible (see fig. 2). States reported similar information to the
Inspector General-AFCARS data elements were not clearly and consistently
defined and technical assistance is effective but difficult to access.
Figure 2: Federal Practices That Affect Data Quality
States reporting at least some affect in data quality
50
45
41
40
35
30
20
10
0
Imprecise Lack of clear, Difficulty definitions for documented accessing
required fields guidance technical
assistance
and guidance
AFCARS
NCANDS
Source: GAO survey.
Notes: Based on responses from 50 states.
The results reported in the figure are a sum of the states that reported
the issue had a very great affect, great affect, moderate affect, or some
affect on the quality of state data submitted to HHS. Very great and great
affect responses are represented in the top section of each bar. Moderate
and some affect responses are represented in the bottom section of each
bar. States not included answered "no affect," "don't know," or "no
answer."
The primary concerns reported by the states we visited were delays in
receiving clear written guidance on defining and reporting certain data
elements and the lack of state input in suggesting changes to AFCARS.
Despite the written guidance available to states in the form of
regulations and an online policy manual, states reported that the
variation in state policies and practices makes it difficult to interpret
how to apply the general guidance. As a result, states consult with HHS to
ensure they are applying the regulations appropriately. However, in
commenting on a draft of this report, officials in Oklahoma told us that a
common concern among the states is the lack of timely response from HHS
when seeking guidance
on how to report data. In addition, officials in New York explained they
have made it a practice to check the HHS Web site on a regular basis for
current guidance but have not found it a useful tool, and may turn to
other states for guidance on AFCARS reporting. In commenting on a draft of
this report, HHS explained that it first refers states to its Web site for
information and believes that the available guidance addresses states'
concerns in most instances. In addition, the states that have had an
AFCARS review experienced delays in obtaining guidance on how to proceed
following the on-site review. For example, Texas officials reported that
the state sought clarification on its improvement plan and submitted
additional questions to HHS following the review. However, when we spoke
with the state officials, they said that they had been waiting 3 months
for a response on how to proceed. An HHS official told us that since the
review process is relatively new, the agency is still developing a process
to respond to the states and recognizes that it has not been responsive to
the states already reviewed. In addition, HHS is taking steps to gather
feedback from states and other users of AFCARS data to determine how to
improve the system to make the data more accurate and usable. As a part of
these efforts, HHS has published a Federal Register notice soliciting
comments and held focus group meetings at national conferences. The
difficulties states face in receiving federal guidance and assistance, as
well as the other challenges they face in reporting data, may negatively
affect the reliability of the data available in AFCARS and NCANDS.
States Are Using Various Practices to Overcome System Development Challenges
and Improve Data on Children's Experiences
Some states are using a variety of practices to address the challenges
associated with developing SACWIS and improving data reliability, although
no formal evaluations of their effectiveness are available. To address the
challenge of developing a system to meet statewide needs, states relied on
caseworkers and supervisors from local offices to assist in the design and
testing of the system. Few states reported in our survey strategies to
overcome the other key challenges, such as limited funding and the
difficulty of securing knowledgeable contractors, but some states we
visited have devised some useful approaches. To improve data reliability,
the five states we visited routinely review their data to identify data
entry errors so that managers can ensure that the missing data are entered
appropriately.
States Are Primarily Relying on SACWIS Users to Overcome Some of the
Challenges to Completing Their Systems
To overcome development challenges, survey respondents emphasized the
importance of including system users in the various phases of completing
SACWIS-planning, design, development, testing, and implementation. Past
GAO work and other research efforts have determined similar approaches as
best practices in building information systems.24 Forty-four of the 46
states responding to our survey that they are developing or operating a
SACWIS indicated that they relied on internal users, such as caseworkers
and supervisors, in the development of their systems and 34 of these
states said that they were extremely helpful participants. The extent to
which the users were involved in development differed across the states.
For example, in Texas, caseworkers from all of their child welfare regions
were recruited to provide input on design and development, as well as
during initial testing, pilot testing, and implementation of the system.
Arkansas reported establishing a committee made up of users to review the
work plan and sign off on recommended changes.
Ten states noted that user input should not be limited to frontline
workers, such as caseworkers, but should include representatives from
other areas of the agency, such as the financial staff, and other agencies
that serve children, such as child support enforcement.25 While not one of
the most common challenges reported in our survey, New Hampshire reported
that one of its challenges with meeting its SACWIS timeframe was not
working collaboratively with other agencies, such as Temporary Assistance
for Needy Families (TANF)26 and child support enforcement, to develop the
payment component of SACWIS. To attempt to overcome this challenge, 26 of
the 46 states responding to our survey that they are developing or
operating a SACWIS indicated that they included external public agency
24See U.S. General Accounting Office, Executive Guide: Improving Mission
Performance Through Strategic Information Management and Technology,
GAO/AIMD-94-115 (Washington, D.C.: May 1, 1994); Center for Technology in
Government, University of Albany, SUNY. Tying a Sensible Knot: A Practical
Guide to State-Local Information Systems. Albany, N.Y., June 1997.
25The Child Support Enforcement Program is a joint federal, state, and
local partnership that was established in 1975 under Title IV-D of the
Social Security Act. Each state runs a child support program, which
provides four major services: locating noncustodial parents, establishing
paternity, establishing child support obligations, and collecting child
support for families.
26In 1996, the Congress created the block grant Temporary Assistance for
Needy Families program replacing the Aid to Families with Dependent
Children (AFDC) and related welfare programs. States were given increased
flexibility in designing the eligibility criteria and benefit rules, which
require work in exchange for time-limited benefits.
users, and 23 reported using representatives from other state agencies
that serve children in developing their SACWIS.
In addition to seeking input from caseworkers and other system users while
developing SACWIS, many states continue to include users as a part of the
implementation teams, to serve as contacts in the field and provide
ongoing assistance, and to provide input on system enhancements. Alabama
responded in our survey that the state had "mentors" in each county to
help caseworkers adjust to the new system. These mentors continue to
provide ongoing support now that the system is implemented. Oklahoma
recruits experienced child welfare field staff for its SACWIS help desk
because of their knowledge of the system and child welfare policy and
practice.
Although states faced other challenges in completing their SACWIS, few
reported implementing approaches to overcome the barriers. According to
survey results, a common problem states faced in developing SACWIS was
receiving insufficient state funding for development. States did not
report in our survey, however, approaches for obtaining more funding for
developing SACWIS, and few states reported developing strategies in an
attempt to overcome the challenges associated with tight budgets for
maintaining their systems. For example, Iowa officials engaged in careful
planning with system users to ensure that they addressed the highest
priorities when enhancing the system. In particular, the officials
reported that maintaining tight control over the development and
maintenance processes helps them avoid investing inordinate amounts of
resources to make corrections to the system. Similarly, few states
reported on approaches to overcome the challenge of finding contractors
with knowledge of child welfare practice. However, Iowa officials
explained that once the contract staff are hired, they are required to
attend the same training as new caseworkers to ensure that they are
familiar with the state's child welfare policies and to familiarize
themselves with casework practices.
States Use Strategies, such as Producing Reports That Identify Missing
Data, in an Attempt to Improve the Reliability of the Data Reported to HHS
Twenty-eight states reported using approaches to help caseworkers identify
the data elements that are required for federal reporting and to help them
better understand the importance of entering timely and accurate data. Ten
states responding to our survey reported reviewing the federal reporting
requirements in training sessions as a way to improve data quality. For
example, Tennessee reported that the state added a component about AFCARS
to the initial and ongoing training workers receive about using SACWIS.
The curriculum addresses the AFCARS
report in general and the individual data elements to help the caseworkers
better understand the purpose of collecting the information. In Nebraska,
a "desk aid" that explains the data elements and where and why to enter
them in the system is available on the caseworkers' computer desktops. In
addition, New York has developed a step-by-step guide explaining to
workers how NCANDS data should be entered, with references to the policy
or statute requiring the information.
To improve data reliability, some states have designed their information
systems with special features to encourage caseworkers to enter the
information. Four states responding to our survey and three states we
visited designed their SACWIS with color-coded fields to draw attention to
the data elements that caseworkers are required to enter. Colorado, Iowa,
New York, and Oklahoma have built into their systems alerts-also known as
"ticklers"-to remind caseworkers and supervisors of tasks that they need
to complete. For example, in Oklahoma, a stoplight icon on the
caseworker's computer desktop reminds the worker when tasks are due. A
green light indicates that nothing is due within 5 days; a yellow light
means that something is due within 5 days; and a red light means that
something is overdue. Caseworkers and supervisors in the states we visited
had mixed responses about the usefulness and effectiveness of the alerts.
Some caseworkers found them to be a nuisance, while other caseworkers and
supervisors found them to be useful tools in managing workloads and
prioritizing daily tasks.
Six states reported that the best way to improve data quality was to use
the data in published reports and hold the caseworkers and supervisors
accountable for the outcomes of the children in their care. In addition,
six states responding to our survey reported using the data available in
their information systems to measure state outcomes similar to the CFSR.
State officials reported that this approach is an effective way to get
local offices invested in the quality of the data. For example, North
Carolina publishes monthly reports for each county comparing their
performance on state data indicators, such as the length of time children
spend in care, to counties of similar size and the state as a whole.
County officials reported that these reports encourage workers to improve
the quality of the data collected and entered into the state system since
their performance is being widely published and compared with that of
other counties.
In addition, all the states we visited reported that frequent review of
their data, such as using software from HHS to test their AFCARS and
NCANDS data to pinpoint data entry errors prior to submitting them to HHS,
has helped improve data quality. When the states identify poor data, they
alert
Concluding Observations
the caseworkers and supervisors of needed corrections and data entry
improvements. For example, Colorado runs these reports about four to five
times a year, with one run occurring approximately 6 weeks before each
AFCARS submission. When the data specialists find errors, they notify the
caseworker to clean up the data.
While most states are developing statewide information systems, challenges
with data reliability remain. Although SACWIS development is delayed in
many states, state officials recognize the benefits of having a uniform
system that enhances the states' ability to monitor the services provided
and the outcomes for children in their care. Although states began
reporting to NCANDS in 1990 and were mandated to begin reporting to AFCARS
in 1995, most states continue to face challenges providing complete,
accurate, and consistent data to HHS. In addition, the results of more
recent HHS efforts, such as conducting AFCARS-related focus groups, are
unknown. Reliable data are essential to the federal government's
development of policies that address the needs of the children served by
state child welfare agencies and its ability to assist states in improving
child welfare system deficiencies. Without welldocumented, clearer
guidance and the completion of more comprehensive reviews of states'
AFCARS reporting capabilities, states are limited in overcoming challenges
that affect data reliability. Because these challenges still remain, HHS
may be using some questionable data as the foundation for national reports
and may not have a clear picture of how states meet the needs of children
in their care.
To improve the reliability of state-reported child welfare data, we
recommended in our July 2003 report that the Secretary of HHS consider, in
addition to HHS's recent efforts to improve AFCARS data, ways to enhance
the guidance and assistance offered to states to help them overcome the
key challenges in collecting and reporting child welfare data. These
efforts could include a stronger emphasis placed on conducting AFCARS
reviews and more timely follow-up to help states implement their
improvement plans or identifying a useful method to provide clear and
consistent guidance on AFCARS and NCANDS reporting. ACF generally agreed
with our findings and commented that the report provides a useful
perspective of the problems states face in collecting data and of ACF's
effort to provide ongoing technical assistance to improve the quality of
child welfare data. In response to our recommendation, ACF said that we
categorized its efforts as "recent" and did not recognize the long-term
efforts to provide AFCARS- and NCANDSrelated guidance to the states.
Although we did not discuss each effort in
depth, we did mention the agency's ongoing efforts in our report. ACF also
noted in its comments that the data definitions need to be updated and
revised and said it is currently in the process of revising the AFCARS
regulations to further standardize the information states are to report-
which we acknowledged in our report. ACF also commented that it is firmly
committed to continue to support the states and to provide technical
assistance and other guidance as its resources will permit. ACF commented
that it provided increased funding to the National Resource Centers in
fiscal year 2003, and it believed that this increase will improve ACF's
ability to provide assistance to the states. After receiving the draft
report for comment, HHS separately provided information on an additional
service the National Resource Center for Information Technology in Child
Welfare provides to states. More recently, HHS said that it would be
creating policy guidance that will delineate what will happen if a state
fails to complete its SACWIS within a reasonable time frame. For example,
funding may become contingent on successful completion of specific
milestones.
Mr. Chairman, this concludes my prepared statement. I would be pleased to
respond to any questions that you or other members of the Subcommittee may
have.
GAO Contact and For further contacts regarding this testimony, please
call Cornelia M. Ashby at (202) 512-8403. Individuals making key
contributions to this
Acknowledgments testimony include Diana Pietrowiak and Sara Schibanoff.
Related GAO Products
Child Welfare: Enhanced Federal Oversight of Title IV-B Could Provide
States Additional Information to Improve Services. GAO-03-956. Washington,
D.C.: September 12, 2003.
Child Welfare: Most States Are Developing Statewide Information Systems,
but the Reliability of Child Welfare Data Could Be Improved. GAO-03-809.
Washington, D.C.: July 31, 2003.
Child Welfare and Juvenile Justice: Several Factors Influence the
Placement of Children Solely to Obtain Mental Health Services.
GAO-03-865T. Washington, D.C.: July 17, 2003.
Child Welfare and Juvenile Justice: Federal Agencies Could Play a Stronger
Role in Helping States Reduce the Number of Children Placed Solely to
Obtain Mental Health Services. GAO-03-397. Washington, D.C.: April 21,
2003.
Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare
Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March 31,
2003.
Human Services: Federal Approval and Funding Processes for States'
Information Systems. GAO-02-347T. Washington, D.C.: July 9, 2002.
Foster Care: Recent Legislation Helps States Focus on Finding Permanent
Homes for Children, but Long-Standing Barriers Remain. GAO-02-585.
Washington, D.C.: June 28, 2002.
Human Services Integration: Results of a GAO Cosponsored Conference on
Modernizing Information Systems. GAO-02-121. Washington, D.C.: January 31,
2002.
District of Columbia Child Welfare: Long-Term Challenges to Ensuring
Children's Well-Being. GAO-01-191. Washington, D.C.: December 29, 2000.
Child Welfare: New Financing and Service Strategies Hold Promise, but
Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.
Welfare Reform: Improving State Automated Systems Requires Coordinated
Federal Effort. GAO/HEHS-00-48. Washington, D.C.: April 27, 2000.
Foster Care: States' Early Experiences Implementing the Adoption and Safe
Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.
Foster Care: HHS Could Better Facilitate the Interjurisdictional Adoption
Process. GAO/HEHS-00-12. Washington, D.C.: November 19, 1999.
Foster Care: Effectiveness of Independent Living Services Unknown.
GAO/HEHS-00-13. Washington, D.C.: November 5, 1999.
Foster Care: Kinship Care Quality and Permanency Issues. GAO/HEHS-99-32.
Washington, D.C.: May 6, 1999.
Juvenile Courts: Reforms Aim to Better Serve Maltreated Children.
GAO/HEHS-99-13. Washington, D.C.: January 11, 1999.
Child Welfare: Early Experiences Implementing a Managed Care Approach.
GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.
Foster Care: Agencies Face Challenges Securing Stable Homes for Children
of Substance Abusers. GAO/ HEHS-98-182. Washington, D.C.: September 30,
1998.
Managing Technology: Best Practices Can Improve Performance and Produce
Results. GAO/T-AIMD-97-38, January 31, 1997.
Child Welfare: HHS Begins to Assume Leadership to Implement National and
State Systems. GAO/AIMD-94-37. Washington, D.C.: June 8, 1994.
Executive Guide: Improving Mission Performance Through Strategic
Information Management and Technology. GAO/AIMD-94-115. Washington, D.C.:
May 1, 1994.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Contact:
To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***