[Federal Register Volume 79, Number 197 (Friday, October 10, 2014)]
[Rules and Regulations]
[Pages 61241-61261]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2014-24204]



[[Page 61241]]

=======================================================================
-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families

45 CFR Part 1355


Statewide Data Indicators and National Standards for Child and 
Family Services Reviews

AGENCY: Children's Bureau (CB), Administration for Children and 
Families (ACF), Administration on Children, Youth and Families (ACYF), 
Department of Health and Human Services (HHS).

ACTION: Final notice of statewide data indicators and national 
standards for Child and Family Services Reviews.

-----------------------------------------------------------------------

SUMMARY: On April 23, 2014, the Administration of Children and Families 
(ACF) published a document in the Federal Register (79 FR 22604). The 
document provided the Children's Bureau's plan to replace the statewide 
data indicators used to determine a state's substantial conformity with 
titles IV-B and IV-E of the Social Security Act through the Child and 
Family Services Reviews (CFSRs). After consideration of the public 
comments and additional Children's Bureau analysis, the Children's 
Bureau is now publishing its final plan. Where relevant, this document 
addresses key comments from the field in response to the April 23, 2014 
Federal Register document.

DATES: Effective October 10, 2014.

FOR FURTHER INFORMATION CONTACT: Miranda Lynch Thomas, Children's 
Bureau, 1250 Maryland Ave. SW., 8th Floor, Washington, DC 20024, (202) 
205-8138.

SUPPLEMENTARY INFORMATION:

Background

    The Children's Bureau (CB) implemented the CFSRs in 2001 in 
response to a mandate in the Social Security Amendments of 1994. The 
legislation required the Department of Health and Human Services to 
issue regulations for the review of state child and family services 
programs under titles IV-B and IV-E of the Social Security Act (see 
section 1123A of the Social Security Act). The reviews are required for 
CB to determine whether such programs are in substantial conformity 
with title IV-B and IV-E plan requirements. The review process, as 
regulated at 45 CFR 1355.31-37, grew out of extensive consultation with 
interested groups, individuals, and experts in the field of child 
welfare and related areas.
    The CFSRs enable CB to: (1) Ensure conformity with federal child 
welfare requirements; (2) determine what is actually happening to 
children and families as they are engaged in child welfare services; 
and (3) assist states to enhance their capacity to help children and 
families achieve positive outcomes. CB conducts the reviews in 
partnership with state child welfare agency staff and other partners 
and stakeholders involved in the provision of child welfare services. 
We have structured the reviews to help states identify strengths as 
well as areas needing improvement within their agencies and programs.
    We use the CFSR to assess state performance on seven outcomes and 
seven systemic factors. The seven outcomes focus on key items measuring 
safety, permanency, and well-being. The seven systemic factors focus on 
key state plan requirements of titles IV-B and IV-E that provide a 
foundation for child outcomes. If we determine that a state has not 
achieved substantial conformity in one or more of the areas assessed in 
the review, the state is required to develop and implement a program 
improvement plan within two years addressing the areas of 
nonconformity. CB supports the states with technical assistance and 
monitors implementation of their program improvement plans. We withhold 
a portion of the state's federal title IV-B and IV-E funds if the state 
is unable to complete its program improvement plan successfully.
    Most relevant to this document are the national standards for state 
performance on statewide data indicators CB uses to determine whether a 
state is in substantial conformity with certain child outcomes. We are 
authorized by the regulations at 45 CFR 1355.34(b)(4) and (5) to add, 
amend, or suspend any of the statewide data indicators and to adjust 
the national standards when appropriate. Statewide data indicators are 
aggregate measures and we calculate them using administrative data 
available from a state's submissions to the Adoption and Foster Care 
Analysis and Reporting System (AFCARS),\1\ the National Child Abuse and 
Neglect Data System (NCANDS),\2\ or a CB-approved alternate source for 
safety-related data. If a state is proposing to use alternative source 
data for NCANDS, such data must be child-level data and contain all of 
the data elements necessary for CB to calculate performance for an 
indicator. If we determine that a state is not in substantial 
conformity with a related outcome due to its performance on an 
indicator, the state will include that indicator in its program 
improvement plan. The improvement a state must achieve is relative to 
the state's baseline performance at the beginning of the program 
improvement plan period.
---------------------------------------------------------------------------

    \1\ AFCARS collects case-level information from state and Tribal 
title IV-E agencies on all children in foster care and those who 
have been adopted with title IV-E agency involvement. Title IV-E 
agencies must submit AFCARS data to the Children's Bureau twice a 
year.
    \2\ NCANDS collects child-level information on every child who 
receives a response from a child protective services agency due to 
an allegation of abuse or neglect. States report this data to the 
Children's Bureau voluntarily. In FFY 2013, all 50 states, the 
District of Columbia, and Puerto Rico submitted NCANDS data.
---------------------------------------------------------------------------

    In an April 23, 2014 Federal Register document (79 FR 22604) we 
provided a detailed review of the consultation with the field and 
information considered in developing the third round of the CFSRs. We 
also proposed a plan for using statewide data indicators and national 
standards that is different than those used in prior rounds including 
the method to calculating such indicators and standards and our 
rationale. During the 30-day public comment period following the 
Federal Register document, we received 52 unique responses from state 
and local child welfare agencies, national and local advocacy and human 
services organizations, researchers and other interested persons. CB's 
reviewed all public comments and questions before making final 
decisions regarding the statewide data indicators and the methodology. 
This public notice includes a summary of our response. The public 
comments and questions that were submitted are available in their 
original form on www.regulations.gov.

Summary of Final Statewide Data Indicators and Methods

    We have changed two indicators in response to the public comments. 
CB will measure the recurrence of maltreatment instead of repeat 
reports of maltreatment as we proposed in the April Federal Register 
document. We will also add a new indicator to measure permanency in 12 
months for children in foster care for 12 months to 23 months.
    Therefore our final plan is to use two statewide data indicators to 
measure maltreatment in foster care and recurrence of maltreatment in 
evaluating Safety Outcome 1: Children are, first and foremost, 
protected from abuse and neglect. We will use statewide data indicators 
to measure achievement of permanency in 12

[[Page 61242]]

months for children entering foster care, permanency in 12 months for 
children in foster care for 12 months to 23 months, permanency in 12 
months for children in foster care for 24 months or more, re-entry to 
foster care in 12 months, and placement stability. These five 
permanency indicators will be used in evaluating Permanency Outcome 1: 
Children have permanency and stability in their living situations.
    A description of each of the seven statewide data indicators, how 
we will calculate them, a summary of relevant public comments, and our 
rationale for the final indicators and response to the public comments 
follows. This document includes our approach to measuring a state's 
program improvement on the indicators should the state not meet a 
national standard. We also provide information on how we will share 
data and information related to state performance as well as data 
quality issues that may impact the indicators and methods.
    Attachment A provides a summary of each final statewide data 
indicator including the numerators, denominators, adjustments and data 
periods used to calculate the national standards. Attachment B provides 
a comparison of the data measures used during CFSR Round 2 with the 
statewide data indicators we will use during Round 3. Attachment C 
provides information on the AFCARS and NCANDS data elements that are 
used to calculate the indicators and national standards. Attachment D 
provides information on the data quality thresholds applied in 
determining whether to include state data for calculating the 
indicators.
    Finally we are issuing concurrent to this document, CFSR Technical 
Bulletin #8 that expands on this document with additional technical 
information and discussion relevant to the statewide data indicators, 
national standards and states' performance on them. The technical 
bulletin will be available on CB's Web site www.acf.hhs.gov/programs/cb.

Statewide Data Indicators for CFSR Safety Outcome 1: Children Are, 
First and Foremost, Protected From Abuse and Neglect

Safety Performance Area 1: Maltreatment in Foster Care

    Indicator Description: Of all children in foster care during a 12-
month period, what is the rate of victimization per day of foster care?
    Calculation: The denominator is of children in foster care during a 
12-month period, the total number of days these children were in foster 
care as of the end of the 12-month period. The denominator is drawn 
from AFCARS. The numerator is of children in the denominator, the total 
number of substantiated or indicated reports of maltreatment (by any 
perpetrator) during a foster care episode within the 12-month period. 
Rates are calculated per day of foster care. However, we will multiply 
the rate by 100,000 to produce larger and more readily understood 
numbers. This indicator is calculated using data that match children 
across AFCARS and NCANDS using the AFCARS record number.
    Some states provide incident dates in their NCANDS data 
submissions. If a state provides incident dates that are associated 
with the maltreatment report, those records with an incident date 
occurring outside of the removal episode will be excluded, even if the 
report dates fall within the episode. We will also exclude the 
following: Complete foster care episodes lasting less than 8 days, any 
report of maltreatment that occurs within the first 7 days of removal, 
victims who are age 18 or more and youth in foster care at age 18 or 
more. For those youth who at the beginning of an included report period 
are 17 years of age and turn age 18, any time spent in foster care 
beyond the young person's 18th birthday is not counted in the 
denominator.
    Justification for Inclusion: This indicator provides a measure of 
whether the state child welfare agency is able to ensure that children 
do not experience abuse or neglect while in the state's foster care 
system. The indicator holds states accountable for keeping children 
safe from harm while under the responsibility of the state, no matter 
who perpetrates the maltreatment while the child is in foster care.
    Public Comments and CB Response: Many commenters supported the 
statewide data indicator for maltreatment in foster care that we 
proposed originally. Such commenters endorsed how the rate will be 
calculated, the inclusion of all maltreatment types by any perpetrator 
(including parents), the exclusion of children in foster care less than 
eight days, and the use of incident dates.
    Regarding incident dates, some of the comments noted concern that 
not all states were consistently reporting incident dates and some 
states have difficulty identifying those dates. CB acknowledges that 
there is variation in states' capacity to report and actual reporting 
of incident dates. We are committed to continuing technical assistance 
to states so that they can improve their ability to report incident 
dates. Since the report of an actual incident date can clarify whether 
an occurrence of maltreatment is actually separate from another or 
whether there were multiple reports that refer to the same incident in 
the data, we are compelled to use this information where it exists. 
Additionally, to prevent potential over-counting of reports that are 
made when a child first enters foster care that reflect what may have 
occurred prior to the child's foster care entry, we will exclude all 
reports of maltreatment that occur within the first 7 days of a child's 
removal from home. We will apply this exclusion consistently for all 
states.
    Some commenters also expressed concern about the variation in how 
states decide to accept a report for investigation and define 
substantiated or indicated maltreatment to classify incidents of abuse 
or neglect. One commenter suggested that CB should have a consistent 
definition of substantiation or indication. We acknowledge that there 
is variation in how states screen in reports of maltreatment, define 
maltreatment, and substantiate maltreatment. This variation reflects 
the discretion that states have to define abuse and neglect and build a 
responsive child protective services system. CB does not have authority 
to mandate a singular definition or process. Further, doing so would 
result in skewing our understanding of how state child protective 
systems respond to alleged maltreatment. It may be helpful to think 
about this indicator as capturing how well the state is able to prevent 
child maltreatment, as it defines it, once the state has made a 
determination that a child needs the protection of the state's foster 
care system. How well the state is able to prevent child maltreatment 
in this circumstance is relative to a national standard based on how 
all states perform in preventing maltreatment in foster care as each 
state has defined maltreatment.
    A couple of commenters were concerned that this indicator did not 
seem to capture how the agency protects children from maltreatment if 
such children do not enter foster care. It is accurate that this 
indicator is focused on protection from subsequent maltreatment for 
children who are already in the state agency's custody. We have another 
indicator that looks at victims of abuse and neglect more broadly to 
address the recurrence of maltreatment. We believe it is important to 
emphasize, however, that the set of indicators that are used for CFSR 
purposes are limited. We encourage states to have a more comprehensive 
set

[[Page 61243]]

of indicators in their own CQI systems as measures of their performance 
for improvement and/or public accountability purposes. CB, through 
joint planning with states and the provision of technical assistance 
can assist states as they consider appropriate indicators and measures 
to be included in their Child and Family Services Plan.
    Two commenters questioned how trial home visits would impact the 
indicator. One commenter advocated for the inclusion of trial home 
visits in the denominator while the other suggested that it should be 
excluded since the public may consider children on trial home visits to 
be at home. Since this indicator is intentionally capturing the 
maltreatment of a child while in the placement and care responsibility 
of the state agency, including when the child is visited by his parent 
or on a trial home visit, we have factored in the entire length of the 
trial home visit (until discharge) in the indicator. As such we will 
not apply a trial home visit adjustment to this indicator.
    One commenter expressed concern that this indicator will make it 
more difficult for children in foster care to achieve normalcy in their 
lives. The concern was that a national measure of maltreatment in 
foster care may influence child welfare agencies to require all adults 
who a child comes into contact with to have criminal and child abuse 
background checks. CB is supportive of ensuring that children in foster 
care are afforded normalcy to the extent practicable. We would like to 
work with states that may have higher rates of maltreatment in foster 
care to analyze which populations appear at risk of such harm and the 
circumstances in which maltreatment is occurring. That way we can help 
states strategize how to address these issues programmatically while 
balancing the well-being and other needs of the children the state 
serves.
    Finally, a few commenters were concerned that the difficulty some 
states experience in using a common identifier in the AFCARS and NCANDS 
files could impact the accuracy of this measure. We have set data 
quality thresholds (see attachment D) to ensure that states' data 
quality issues do not affect the integrity of the standard. We have 
required states to have consistent identifiers of children used in the 
reporting of AFCARS data since it began (1993) and we have requested 
the AFCARS record number in the NCANDS child files since FY 2003. In 
the last round of CFSRs, we provided states with data profiles that 
indicated the percentage of records with AFCARS record numbers reported 
in the NCANDS child file. This was a means of improving state reporting 
and providing context to the data that was provided to states on 
maltreatment by parents in foster care. As such, we proposed this 
indicator noting that states had improved their reporting of AFCARS 
record numbers which made viable using an indicator with this link in 
this round of reviews. We have identified the states for which using a 
consistent identifier is an issue and will be engaging in discussions 
with them on how they can improve their reporting of AFCARS record 
numbers.

Safety Performance Area 2: Recurrence of Maltreatment

    Indicator Description: Of all children who were victims of a 
substantiated or indicated report of maltreatment during a 12-month 
reporting period, what percent were victims of another substantiated or 
indicated maltreatment allegation within 12 months of their initial 
report?
    Calculation: The denominator is the number of children with at 
least one substantiated or indicated report of maltreatment in a 12-
month period. The numerator is the number of children in the 
denominator that had another substantiated or indicated report of 
maltreatment within 12 months of their initial report. This indicator 
is calculated using data from NCANDS.
    We will use report dates as the primary data element to determine 
when the maltreatment occurred, and include only reports occurring in 
the 12-month period. Substantiated or indicated maltreatments reports 
with report dates in the 12-month period with disposition dates after 
the 12-month period are included, as well. If there is a subsequent 
report of maltreatment within 14 days of the earlier report we will not 
count it as recurrent maltreatment. If the state provides the incident 
date and it indicates that multiple reports refer to the same incident, 
we will also not count it as recurrent maltreatment. Youth who are age 
18 or more are excluded from the calculation of the indicator.
    Justification for Inclusion: This indicator provides an assessment 
of whether the agency was successful in preventing subsequent 
maltreatment for a child if the child is the subject of a substantiated 
or indicated report of maltreatment.
    Summary of Public Comments: We proposed originally an indicator of 
the percent of children with a screened-in report of alleged 
maltreatment that occurs within 12 months of an initial screened-in 
report. We justified the proposed indicator to replace the recurrence 
of maltreatment indicator used in prior CFSRs as we thought it could 
better assess the scope of the child welfare agency's protection 
response to incoming reports of maltreatment. We also believed the 
proposed indicator would address potential measurement problems of a 
substantiation-based indicator should a state change to a differential 
response approach during the course of a CFSR program improvement 
period.
    A couple of commenters supported the re-report of maltreatment 
indicator as we proposed it originally. However, the majority of 
commenters, particularly state child welfare agencies, expressed their 
concerns with the proposed indicator. Many commenters were concerned 
about several unintended consequences or challenges in messaging what 
the results of this indicator mean.
    One concern expressed by commenters was the potential for any state 
changes in the policy or program criteria for screening in reports to 
impact a state's performance on the indicator, either negatively or 
positively. Another concern was that the indicator was perceived as 
contrary to state and federal laws that encourage and support reporting 
of potential child maltreatment. Similarly, some commenters believed 
that the indicator, if constructed as a measure of safety, could be 
interpreted to mean that agencies that had high rates of screened-in 
reports of maltreatment were not ensuring child safety and that there 
were higher rates of actual recurrence of substantiated maltreatment. 
These commenters noted that some states screen in reports for children 
who are at little to no risk of maltreatment, such as for community or 
public service referrals. They noted that such referrals should not be 
thought of in the same way as actual allegations of maltreatment.
    Secondary concerns raised by commenters were around the variation 
in state responses to screened-in reports as a matter of practice that 
could make interpretation of the indicator challenging. For example, 
commenters identified challenges associated with the variation in state 
screening decisions and unsubstantiated report expunction requirements. 
Several commenters provided suggestions for retaining the re-report of 
maltreatment indicator including: Requiring a substantiated report to 
follow the initial screened-in report to qualify as a re-report of 
maltreatment; risk adjusting based on the state's screen-in rate; and

[[Page 61244]]

allowing for a defined period of time between a report and subsequent 
report.
    We believe that there is good reason for a revision to our 
approach. We are mindful that an indicator must be readily explainable 
to the field and the public in terms of what it tells us about a child 
welfare system's response to vulnerable children and families. We also 
were concerned about the potential for unintended consequences with the 
proposed measures. We considered some of the commenter's suggestions 
for improving a re-report indicator but each proposed solution raised 
some level of concern. Still, CB believes that this indicator does hold 
potential to shed light on how well states are providing services to 
the larger population of children at risk. As such, we will include the 
re-report indicator as originally proposed as a context measure in the 
state's data profile.
    CB will return to an indicator of recurrence of maltreatment, 
similar to that used in the prior two rounds. One of the modifications 
to this indicator over the one used in prior rounds will be to have an 
expanded timeframe--looking at substantiated or indicated reports in an 
initial 12-month period and whether there is a subsequent one within 12 
months. We are also using similar adjustments as used in the recurrence 
of maltreatment indicator. We will use incident dates where available, 
exclude reports made within 14 days of an earlier report, and exclude 
youth age 18 and older. With this indicator, however, we are not able 
to address one of our concerns about the potential impact of a state 
implementing differential or alternative response on the measure. Where 
states implement differential response during program improvement, we 
will consider on a case-by-case basis the situation and its 
implications for accurate depictions of compliance and/or meeting 
improvement goals.

CFSR Permanency Outcome 1: Children Have Permanency and Stability in 
Their Living Situations

Permanency Performance Area 1: Permanency in 12 Months for Children 
Entering Foster Care

    Indicator Description: Of all children who enter foster care in a 
12-month period, what percent discharged to permanency within 12 months 
of entering foster care?
    Calculation: The denominator is the number of children who enter 
foster care in a 12-month period. The numerator is the number of 
children in the denominator who discharged to permanency within 12 
months of entering foster care and before turning age 18. This 
indicator is calculated using data from AFCARS. For the purposes of 
this indicator, discharged to permanency includes the AFCARS foster 
care discharge reasons of: Reunification with parents or primary 
caretakers, living with other relative(s), adoption and guardianship. 
This indicator excludes youth who enter foster care at or after age 18 
and children who have a complete foster care episode lasting less than 
8 days. For children with multiple foster care episodes in the 12-month 
period, this indicator will use the first episode reported.
    We apply a trial home visit adjustment to this indicator. This 
means that if a child discharges from foster care during the 12-month 
period to reunification with parents or other caretakers after a 
placement setting of a trial home visit, any time in that trial home 
visit that exceeds 30 days is discounted from the length of stay in 
foster care. A similar trial home visit adjustment has been applied to 
permanency indicators in prior rounds of CFSRs. The adjustment is made 
to address variations in state policy regarding returning children to 
their families for a period of time before the state makes a formal 
discharge from foster care ending the agency's placement and care 
responsibility.
    Justification for Inclusion: This indicator provides a focus on the 
child welfare agency's responsibility to reunify or place children in 
safe and permanent homes as soon as possible after removal.
    Public Comments and CB Response: Many commenters expressed support 
for one or more aspects of the permanency in 12 months for children 
entering foster care indicator. In particular, commenters supported the 
inclusion of guardianship and adoption within the concept of permanency 
and the use of an entry cohort to assess the state's achievement of 
permanency for children. A few commenters requested clarification on 
whether we would apply the trial home visit adjustment to this 
indicator, which we have confirmed above.
    A significant number of commenters believed that this indicator, in 
combination with the permanency in 12 months indicator for children who 
have been in foster care for 24 months or more, left a significant gap 
in understanding the experiences of children who have been in foster 
care for 12 to 23 months. We are addressing these comments by adding an 
indicator. We provide details on the new indicator in the next section.
    Two commenters pointed out issues with our original description of 
the indicator as evaluating the first episode within the period for 
children who have multiple episodes during the same 12-month period. 
One commenter noted that we indicated in an attachment that we would 
rely on the ``date of most recent removal'' data element and questioned 
whether the description of capturing episodes was accurate. Another 
commenter pointed out that multiple episodes within a six-month period 
may be masked since you cannot duplicate children within a report 
period. Both commenters are accurate about the limits of the AFCARS 
data. Each six-month report period from AFCARS includes detail on the 
most recent foster care episode as of the end of the six-month period. 
We do not have information in AFCARS about any intervening foster care 
episodes. These `masked' episodes represent a very small percentage of 
all episodes reported to AFCARS. When we refer to using the first 
episode within the period, we mean we will use the episode provided in 
the first six-month report period of the year. We are using the 
earliest one available to us, given the structure of AFCARS. In the 
past, when we merged six-month submissions together we kept only the 
most recent reported episode for the 12-month period, so this 
represents a change from that practice.

Permanency Performance Area 2: Permanency in 12 Months for Children in 
Foster Care 12 to 23 Months

    Indicator Description: Of all children in foster care on the first 
day of a 12-month period who had been in foster care (in that episode) 
between 12 and 23 months, what percent discharged from foster care to 
permanency within 12 months of the first day of the 12-month period?
    Calculation: The denominator is the number of children in foster 
care on the first day of a 12-month period who had been in foster care 
(in that episode) between 12 and 23 months. The numerator is the number 
of children in the denominator who discharged from foster care to 
permanency within 12 months of the first day of the 12-month period and 
before turning 18. This indicator is calculated using data from AFCARS. 
For the purposes of this indicator, discharged to permanency includes 
AFCARS foster care discharge reasons of: Reunification with parents or 
primary caretakers, living with other relative(s), adoption and 
guardianship. Youth who are aged 18 years or more on the first day of 
the 12-month period are excluded from the calculation. We

[[Page 61245]]

apply the trial home visit adjustment, as defined earlier, to this 
indicator.
    Justification for Inclusion: This indicator provides a focus on the 
child welfare agency's responsibility to reunify or place children in 
safe and permanent homes timely if not achieved in the first 12 months 
of foster care.
    Public Comments and CB Response: As noted above a number of 
commenters were concerned about the potential for a significant gap in 
the understanding and measurement of performance for children who may 
achieve permanency between 12 and 23 months. Some of the concerns 
expressed noted that a significant portion of children who remain in 
care beyond a year achieve permanency within the next year and that 
could not be captured with the two originally proposed indicators. Some 
made a programmatic argument about the requirements in title IV-B and 
IV-E of the Social Security Act (primarily due to amendments made by 
the Adoption and Safe Families Act) that focus on procedural safeguards 
for children who remain in care beyond 12 months. These include 
requirements for permanency hearings every 12 months that focus on 
moving a child to permanency and requirements to file petitions for 
termination of parental rights once a child has been in foster care for 
15 out of the most recent 22 months, unless exceptions apply. 
Similarly, some commenters noted that guardianships and adoptions often 
take more than 12 months due to procedural and legal requirements, but 
could still be considered timely if occurred within 18 to 24 months. 
These commenters advocated for adding an indicator that incorporates 
the performance of the state in achieving permanency for children 
between their 1st and 3rd year of foster care. We found these arguments 
to be compelling and have added this 2nd indicator to be responsive to 
these points.
    Before adding this indicator, we considered whether to extend 
either the permanency achievement indicator for the entry cohort to 
include children who enter foster care in a 24-month period, or to 
expand the cohort of children in care 24 months or more to include 
children in care 12 months or more. With the former option, we believed 
that the longer cohort would weaken the focus on the large group of 
children who are likely to exit to permanency quickly. We also noted 
that by changing the cohort we could no longer pair it with a companion 
measure of re-entry to foster care within 12 months (discussed later). 
With the latter option we were similarly concerned that we would no 
longer be able to focus attention to the children who have been in care 
for long periods of time and are must likely to grow up in foster care. 
Thus we chose to add a new cohort rather than expand one of the 
originally proposed indicators.

Permanency Performance Area 3: Permanency in 12 Months for Children in 
Foster Care 24 Months or More

    Indicator Description: Of all children in foster care on the first 
day of a 12-month period, who had been in foster care (in that episode) 
for 24 months or more, what percent discharged to permanency within 12 
months of the first day of the 12-month period?
    Calculation: The denominator is the number of children in foster 
care on the first day of a 12-month period who had been in foster care 
(in that episode) for 24 months or more. The numerator is the number of 
children in the denominator who are discharged from foster care to 
permanency within 12 months of the first day of the 12-month period and 
before turning 18. This indicator is calculated using data from AFCARS. 
For the purposes of this indicator discharged to permanency includes 
AFCARS foster care discharge reasons of: Reunification with parents or 
primary caretakers, living with other relative(s), adoption, and 
guardianship. Young people who are aged 18 years or more on the first 
day of the 12-month period are excluded from the calculation. The trial 
home visit adjustment, as defined earlier, is applied to this 
indicator.
    Justification for Inclusion: This indicator monitors the 
effectiveness of the state child welfare agency in continuing to ensure 
permanency for children who have been in foster care for longer periods 
of time.
    Public Comments and CB Response: Several commenters expressed 
support for this indicator as a useful measure because we have a 
singular concept of permanency to include permanent placement with a 
relative, reunification, adoption and guardianship. Commenters agreed 
with using this measure in parallel with the permanency in 12 months 
for children entering foster care indicator. Commenters also 
appreciated the indicator's potential to maintain a focus on those 
children who experience long lengths of stay in foster care. The field 
expressed concerns similar to those for the permanency in 12 months for 
children entering foster care indicator. Some commenters also expressed 
a need to adjust for trial home visits. A few commenters raised 
concerns about how the experiences of children age 17 and older could 
impact the measure and if including this group of children in the 
statewide data indicator would disadvantage states that extend foster 
care beyond age 18.
    CB has specified that this indicator will include the trial home 
visit adjustment as do the other two permanency achievement indicators. 
We have also addressed the concern regarding the gap in cohorts by 
adding another indicator as explained previously. Although we have 
excluded from the calculation of this indicator young people age 18 or 
older on the first day of the 12-month period, we will not exclude from 
the denominator young people who turn age 18 during the 12-month 
period. Regardless of federal and state provisions that provide young 
people avenues to remain in foster care beyond 18 for care and services 
while they transition to adulthood, when young people do not achieve 
permanency by 18 they cannot be considered to have achieved permanency. 
While we can agree that providing such extended care can mean better 
well-being outcomes for youth based on existing research, extending 
care does not address the young person's need for permanency, which is 
the focus of this indicator.

Permanency Performance Area 4: Re-entry to Foster Care in 12 Months

    Indicator Description: Of all children who enter foster care in a 
12-month period who discharged within 12 months to reunification, 
living with a relative(s), or guardianship, what percent re-enter 
foster care within 12 months of their discharge?
    Calculation: The denominator is the number of children who entered 
foster care in a 12-month period and discharged within 12 months to 
reunification, living with a relative(s), or guardianship. The 
numerator is the number of children in the denominator who re-entered 
foster care within 12 months of their discharge from foster care. We 
exclude children in foster care for less than 8 days from this 
indicator and children who enter or exit foster care at age 18 or more. 
If a child re-enters foster care multiple times within 12 months of 
their discharge, only the first reported re-entry into foster care is 
selected. This indicator is calculated using data from AFCARS.
    Justification for Inclusion: This indicator enables CB to monitor 
the effectiveness of programs and practice that support reunification 
and other permanency goals so that children do not return to foster 
care.
    Public Comments and CB Response: Some commenters expressed support 
for the re-entry to foster care statewide data

[[Page 61246]]

indicator as its own measure and as a companion measure to permanency 
performance area 1 as we proposed. Companion measures are discussed in 
the program improvement plan section of this document. Several 
commenters shared concerns about the possibility that the indicator 
overlooks the re-entry to foster care for children who did not achieve 
permanency quickly. Comments in this area point out that, because the 
indicator focuses on children who achieve permanency within one year, 
children who leave foster care after a year are not considered. They 
argued that this creates a truncated view of re-entry to foster care. 
Some of these commenters noted that the indicator used in the prior 
round of reviews had this more expanded cohort of children included and 
provided the state with a better perspective of the children who 
returned to foster care. A number of alternative approaches to 
measuring re-entry to foster care were suggested including revising the 
cohort of focus or adding cohorts or indicators that looked at re-
entries into foster care more comprehensively.
    During CFSR Round 2, this performance area was evaluated using a 
similar measure as a part of a composite. For that measure, we 
calculated the percent of all children discharged from foster care to 
reunification or living with a relative in a 12-month period, who re-
entered foster care in less than 12 months from the date of discharge. 
The CFSR round 3 indicator differs from the measure used previously, in 
part, by limiting the children included in the indicator to the 12-
month entry cohort. We intentionally limited the indicator to focus on 
children that enter foster care within a 12-month period to better 
align it with the other cohorts. We also note again that since most 
children return to their homes or achieve permanency within the first 
year of entry into foster care, this indicator will capture the 
majority of the population that may re-enter foster care.

Proposed Permanency Performance Area 4: Placement Stability

    Indicator Description: Of all children who enter foster care in a 
12-month period, what is the rate of placement moves per day of foster 
care?
    Calculation: The denominator is of children who enter foster care 
in a 12-month period, the total number of days these children were in 
foster care as of the end of the 12-month period. The numerator is of 
children in the denominator, the total number of placement moves during 
the 12-month period. The days in care and moves during the placement 
episodes are cumulative across episodes reported in the same year. 
Rates are calculated per day of foster care. However, we will multiply 
the rate by 1,000 to produce larger numbers that are easier to 
understand. Only those placement settings that are required to be 
counted in the AFCARS file are used for this indicator. If the child is 
moved to a living arrangement or setting that would not result in the 
state increasing the number of placement settings reported in AFCARS 
such moves are not included in this indicator. Children in foster care 
for less than 8 days are excluded from the calculation. Youth who turn 
18 during the 12-month period will not have time in care beyond their 
18th birthday or moves after their 18th birthday counted.
    Justification for Inclusion: This indicator emphasizes states' 
responsibility to ensure that children whom the state removes from 
their homes experience stability while they are in foster care.
    Public Comment and CB Response: Several commenters expressed 
support for the placement stability data indicator citing it as an 
improvement over the previous measure and empirically-based. Some 
commenters agreed with the use of entry cohorts and the move to a rate 
of placements controlling for the length of stay. A few commenters 
asked for clarity on which moves in foster care are included in the 
indicator. In response, we have added to the description above. In 
general, there are placement settings that are reported in AFCARS but 
which are not `counted' in terms of a move. These include trial home 
visit episodes, runaway episodes, respite care and changes in a single 
foster family home's status, for example to reflect a licensing change 
from a foster care home to a home dually licensed for adoption. 
Additional information on AFCARS placement setting changes can be found 
in the CB's Child Welfare Policy Manual.\3\
---------------------------------------------------------------------------

    \3\ In particular, see the Child Welfare Policy Manual Section 
1.2B.7, AFCARS, Data Elements and Definitions, Foster Care Specific 
Elements, Placements found at http://www.acf.hhs.gov/cwpm/programs/cb/laws_policies/laws/cwpm/index.jsp.
---------------------------------------------------------------------------

    A few commenters voiced concerns about using only entry cohorts for 
placement stability, which overlooks children who have been in foster 
care for longer periods of time. Other commenters pointed out that 
states could track additional cohorts of children without it being a 
federal indicator for CFSR purposes. During CFSR Round 2, we evaluated 
placement stability through three individual measures that made up a 
composite. All three of the measures, differentiated by length of stay 
in foster care, looked at the percent of children with two or fewer 
placement settings. The new indicator controls for the length of time 
children spend in foster care so only one indicator is needed. Further, 
it looks at moves per day of foster care, rather than children as the 
unit of analysis, as was employed during CFSR Round 2. The measure used 
for CFSR Round 2 was unable to differentiate between children who moved 
twice from children who moved more. The new indicator does not count 
initial placements, but counts each subsequent move to capture 
accurately the rate of placement moves given the amount of time they 
were at risk of moving, rather than the number of children affected.
    CB believes that placement stability is important to the permanency 
and well-being of children in foster care regardless of how long they 
have been in foster care. Even so, our analysis of AFCARS data 
indicates that most placement moves occur within a child's first 12 
months of foster care, which is why we focused this indicator on that 
time period. With this refined focus, CB and states can monitor the 
period during which placement moves are most likely to occur and the 
state's most recent performance. Since the CFSR Round 2 measures will 
still be included as a context measures in the data profile, states can 
use such information to analyze their trends, practice and target areas 
for improvement.
    Some commenters questioned how to calculate the measure and whether 
the data were available to do so accurately. One concern was whether 
all placement days could be counted across all episodes in a year. 
Although the structure of AFCARS obscures some short-term episodes from 
view, we are using all available information to sum placement days and 
moves across episodes, to the extent practicable. The number of 
placement settings is always relevant to the reported episode, so this 
does not bias the results. Further, it is the same for all states, so 
we treat states equally methodologically.
    Another commenter asked for clarification on whether the indicator 
would track children for 12 months from entry date, or simply count 
placement days during the 12-month period for children entering during 
that period. The calculation is the latter; we will count only the care 
days used within the 12-month period. Even if the child entered late in 
the 12-month period, we will count only those days and moves within the 
12-month period. This measure allows for this because it controls for 
time in care.

[[Page 61247]]

    Some commenters were apprehensive about how the placement stability 
indicator might impact beneficial placement moves in foster care. 
Several commenters pointed out that there are circumstances when 
placement changes might produce better outcomes for children or best 
address their well-being needs such as when children may be moved to be 
with siblings or to meet the placement Indian Child Welfare Act's 
placement preferences. These commenters noted that the data generated 
by the placement stability indicator might not adequately explain these 
situations or create disincentives to move a child when such moves are 
appropriate.
    As we have noted in response to similar comments on the indicators 
of placement stability used in prior rounds of review, AFCARS does not 
have information about whether a placement change reflects a positive 
move that is made for the best interests of the child and/or towards 
the achievement of the child's permanency and well-being needs. The 
current administrative data collection does not capture all of the 
contextual information necessary for us to understand the dynamic needs 
of the child or the conditions of the child's placement. We have always 
used the onsite case review component of the CFSR to provide more 
evaluative information about a child's moves in foster care and 
continue to do so in this round of reviews.\4\ In so doing, we consider 
whether moves that legitimately support the child's best interests 
rather than an agency's resource limitations or other concerns justify 
the move. States past performance during the onsite case review in this 
area indicates that children experience many moves that are not for the 
purposes of meeting their needs.\5\
---------------------------------------------------------------------------

    \4\ See the CFSR Onsite Review Instrument, Stability of Foster 
Care Placement (item 4) at https://training.cfsrportal.org/resources/3044.
    \5\ U.S. Department of Health and Human Service, Child and 
Family Services Reviews Aggregate Report, Findings for Round 2 
Fiscal Years 2007-2010. December 16, 2011. Located online at http://www.acf.hhs.gov/sites/default/files/cb/fcfsr_report.pdf.
---------------------------------------------------------------------------

    Finally, a couple of commenters noted that state administrators 
might have difficulty in explaining this indicator to stakeholders or 
thinking through how it relates to practice since it is expressed as a 
rate as opposed to the prior placement stability measure. We understand 
that the new indicators, particularly those that are expressed as a 
rate, will require states to acquire new strategies to communicate with 
the field about how we measuring performance. We will work with states 
to do so in the data profiles and in the ongoing assistance we provide 
to states and their stakeholders around practice implications.

Additional Comments on Cross-Cutting Issues or Multiple Indicators

    Some commenters opined on cross-cutting issues or requested that CB 
address other issues in connection with the indicators that are 
relevant as general concerns or to multiple indicators. There were 
several additional comments that were outside the scope of this Federal 
Register document and relate to comments or perspectives on child 
welfare policy that are inappropriate for us to address in this 
document.
    Use of individual indicators and fewer indicators. Many commenters 
expressed strong support for our proposal to replace the composites 
used for permanency in round 2 with individual indicators of permanency 
in this round. Many appreciated our responsiveness to feedback from the 
field on their challenges with translating composite measures and noted 
that individual indicators had more promise for engaging their workers 
and partners in understanding performance and working together towards 
improvement. Similarly there were several commenters who supported 
using fewer indicators as part of the CFSR. Some noted that a limited 
number of indicators would also reduce challenges in the interpretation 
of multiple measures, which may sometimes appear to offer conflicting 
perspectives on performance.
    Greater reliance on entry cohorts. Commenters generally supported 
CB's intention to rely more on entry cohorts as a method for measuring 
performance and gauging state improvement. However, a commenter 
suggested that CB be more precise in its terminology, noting that the 
term ``entry cohort'' was overbroad to describe the cohorts of interest 
the indicators include. While we agree that this term is broad, we 
included the term to reflect our general change in approach to 
measurement in some areas. As we have described each indicator's cohort 
specifically in terms of which children and circumstances are included 
in the numerator and the denominator we do not believe it is necessary 
to go into greater detail in naming the type of cohorts used.
    Federal data elements and consistency of state practice. A few 
commenters requested that CB define terms that are referenced in the 
indicators or require states to have consistency in what is captured in 
AFCARS. One agency asked for CB to evaluate how to define ``foster care 
placement'' to ensure that the states report consistently who is in 
foster care across the country. In particular, the commenter noted that 
a child's placement outside of his or her own home and with a relative 
is not always included in the reporting population depending on the 
circumstances. Another requested that we provide more clarity regarding 
the discharge reason of `living with relatives' within AFCARS.
    CB is not defining those terms further in this document. However, 
we will consider how to provide additional technical assistance and 
guidance to states on how to report AFCARS data accurately consistent 
with existing policy and also consider whether additional policy is 
necessary. We note that in defining AFCARS data elements and guidance, 
CB has intentionally considered the range of states' child welfare 
practices, authorities and responsibilities. For example, the issue of 
whether a child `placed' with a relative is reported as in foster care 
to AFCARS depends in part on whether the state child welfare agency has 
placement and care responsibility of the child and not whether the 
child is residing in his own home. We want all states to understand and 
apply AFCARS reporting populations, data element definitions and other 
related guidance consistently. However, the application of that 
guidance will reflect the unique aspects of a state's foster care 
program and population.
    Well Being indicators. One organization recommended that CB improve 
well-being metrics used in the CFSR. Particular suggestions included 
tracking states' implementation of provisions of the Fostering 
Connections to Success and Increasing Adoptions Act of 2008 (Public Law 
112-34) related to health including that children in foster care 
receive health screenings, have up-to-date health information and 
records, and states have processes for health oversight plans including 
monitoring children's use of psychotropic medications. Another 
suggestion was for CB to work with the Centers for Medicare and 
Medicaid Services (CMS) and the National Collaborative for Innovation 
in Quality to develop effective well-being measures.
    CB focuses on how states are providing for children's well-being 
needs in the CFSR even though we do not have data elements in AFCARS or 
NCANDS that support the development of meaningful statewide data 
indicators relevant to child well-being at this time. Through the 
onsite review component

[[Page 61248]]

of the CFSR, CB examines whether the state has appropriately assessed a 
child's health (including dental) and mental health needs, and if 
applicable, whether the state also identified and managed any health 
and mental health issues by facilitating the provision of the necessary 
services for all children in foster care and applicable children 
receiving services in their own homes. In the evaluation, we consider 
whether the state conducted initial and periodic health/mental health 
screenings for the child, the presence or lack thereof of up to date 
health information and oversight of medications, if applicable. More 
information on the particular assessment questions in the onsite review 
can be found in the CFSR Onsite Review Instrument.\6\ CB has described 
some of our efforts to focus child well-being issues in an issuance in 
2012.\7\ CB will continue to work in collaboration with CMS and other 
appropriate partners to strengthen our ability to support states in 
measuring and ensuring positive outcomes in these areas.
---------------------------------------------------------------------------

    \6\ See the CFSR Onsite Review Instrument, Physical Health of 
the Child and Mental/Behavioral Health of the Child (items 17 and 
18). Available online at https://training.cfsrportal.org/resources/3044.
    \7\ See for example ACYF-CB-IM-12-04, Promoting Social and 
Emotional Well-Being for Children and Youth Receiving Child Welfare 
Services. April 17, 2012. Available at http://www.acf.hhs.gov/sites/default/files/cb/im1204.pdf.
---------------------------------------------------------------------------

    Framing indicators in a positive direction. There were several 
comments along the theme of reframing some of the indicators so that 
they were stated positively. For example, one commenter suggested that 
the indicator be renamed to `permanency maintained' and change the 
calculation of the indicator to be positively framed so that the 
denominator includes children exiting care to permanency and the 
numerator includes those that do not re-enter. Regarding placement 
stability, two commenters noted that although the indicator 
nomenclature is positively stated as placement stability, the 
description clarifies that the indicator itself is calculated 
negatively as placement instability. These commenters suggested 
switching the numerator and denominator so that the indicator could be 
expressed in a positive fashion. CB chose not to revise the indicators 
or their descriptions in this way. Communicating these indicators can 
be challenging, and reversing the direction of the indicator makes it 
less intuitive and more complicated to measure and communicate. Second, 
maintaining these indicators as described allows us to remain 
consistent with the concepts as measured during prior CFSR rounds, 
promoting greater ease of use. In other cases, the measures simply 
cannot be reversed. As such we are keeping the indicators framed as 
described.
    Applicability to particular populations. We received comments of 
concern about how the data indicators were perceived to apply to 
specific groups of children. One organization sought additional 
consultation with Indian tribes on the data indicators and revisions to 
round 3 overall to inform our thinking on applicability to Indian 
children. CB conducted in-person consultation with Indian tribes in 
2011 regarding improvements in the CFSR in the areas use of data and 
performance monitoring overall. We used this feedback, in conjunction 
with feedback from states and other stakeholders in revising round 3 
and the data indicators. However, we understand the need to further 
engage Indian tribes in meeting the needs of Indian children, 
particularly those in state custody. In addition to reinforcing with 
states the importance of engaging and collaborating Indian tribes 
throughout the CFSR process, CB will work directly with Indian tribes 
and organizations that advocate on behalf of Indian children to ensure 
that Indian tribes are informed about the CFSRs and the opportunities 
to participate in them.
    We also received comments of concern about how data indicators can 
miss how states are performing with regard to Native American children, 
LGBTQ populations and older youth. We also heard concerns that state 
results on such indicators could be used as justification for the state 
to focus their attention on other groups of children or avoid work in 
accordance with best practices for such populations. We understand that 
the data indicators are limited and provide generalized information 
about a state's performance. CB is committed to consulting with states 
to understand what their statewide performance is or is not revealing 
about its programs, practice and results for the particular populations 
of children served by the state. Although the assessment of the state's 
performance on national indicators is part of our monitoring efforts, 
it must be paired with a state analysis of cases reviewed during the 
onsite review and other data or information that the state has its 
disposal to better understand what is the experience of children 
involved in the child welfare system.

National Standards and State Performance

    We have set the national standard at the national observed 
performance for each of the seven indicators.
    For indicators in which the outcome for a child either occurred or 
did not occur the standard is calculated as the number of children in 
the nation experiencing the outcome divided by the number of children 
in the nation eligible for and therefore at risk of the outcome. This 
is the case for the indicators that measure permanency (for all 
cohorts) in 12 months, re-entry to foster care in 12 months and 
recurrence of maltreatment. The result of the calculation is a 
proportion. However, we present the standard as a percentage by 
multiplying the proportion by 100.
    For indicators in which the outcome for a child is a count per day 
in care the standard is calculated as the sum of counts for all 
children in the nation divided by the sum of days these children were 
in care. This is the case for the indicators for placement stability 
(moves per day in care) and maltreatment in foster care (number of 
victimizations per day in care). The result of the calculation is a 
rate. We are multiplying the rates to yield more understandable 
numbers: for placement stability by 1,000 to yield a rate of moves per 
1,000 days; and, for maltreatment in foster care by 100,000 to give a 
rate of victimizations per 100,000 days in care.
    The following table shows the national standards for each 
indicator.

[[Page 61249]]



                     Table 1--National Standards for CFSR Round 3 Statewide Data Indicators
----------------------------------------------------------------------------------------------------------------
  Statewide data indicators for safety
                outcome 1                                            National standard
----------------------------------------------------------------------------------------------------------------
Maltreatment in Foster Care.............  8.04 victimizations per 100,000 days in care.
Recurrence of Maltreatment..............  9.0%.
----------------------------------------------------------------------------------------------------------------
Statewide data indicators for permanency                             National standard
                outcome 1
----------------------------------------------------------------------------------------------------------------
Permanency in 12 Months for Children      40.4%.
 Entering Foster Care.
Permanency in 12 Months for Children in   43.7%.
 Foster Care 12 to 23 Months.
Permanency in 12 Months for Children in   30.3%.
 Foster Care 24 Months or More.
Re-Entry to Foster Care in 12 Months....  8.3%.
Placement Stability.....................  4.12 moves per 1,000 days in foster care.
----------------------------------------------------------------------------------------------------------------

    Public Comment and CB Response: Some commenters stated that using 
the national observed performance as the national standard for state 
performance was an improvement over CFSR round 2. A few others argued 
that the state should be held to higher standards believing that was 
consistent with legislative intent in requiring ``substantial 
conformity'' with federally mandated state plan requirements.
    As we considered how to set national standards, we attempted to 
balance the need for standards that were ambitious yet feasible. We 
also were mindful of the states' collective historical performance and 
our historical expectations of substantial conformity. As we noted in 
the prior document, we believe that the national observed performance 
is a reasonable benchmark and would appropriately challenge states to 
improve their performance.
    Some commenters urged us to allow states to be measured against 
their own performance rather than using a national comparison due to 
the disparate ways states across the country conduct child welfare 
activities. Although we acknowledge that there are disparities in child 
welfare activities in the states, we believe it is appropriate for CB 
to set consistent expectations for states' performance in its title IV-
B and IV-E programs. We also note that the regulation that governs 
CFSRs requires that we determine substantial conformity based in part 
on national standards versus state-specific benchmarks (45 CFR 
1355.31(a) and (b)). CB has, however, set improvement goals based on 
how each state has performed historically.
    Multi-level modeling approach. State performance on each statewide 
data indicator will be assessed using a multi-level (i.e., 
hierarchical) model appropriate for that indicator. A multi-level 
logistic regression model will be used for indicators in which the 
outcome for a child either occurred or did not occur. A multi-level 
Poisson regression model will be used for indicators in which the 
outcome is a count per unit of time. We chose multi-level modeling 
because it is a widely accepted statistical method that enables fair 
evaluation of relative performance among states with different case 
mixes. The multi-level model that we employ when assessing each state's 
performance takes into account: (1) The variation across states in the 
age distribution of children served for all indicators, and the state's 
entry rate for select indicators (risk adjustment); (2) the variation 
across states in the number of children they serve; and, (3) the 
variation in child outcomes between states. The result of this modeling 
is a performance value that is a more accurate and fair representation 
of each state's performance than can be obtained with simply using the 
state's observed performance.
    Public Comments and CB Response: No specific comments were received 
on using a multi-level approach.
    Risk Adjustment. We will risk adjust on child's age for each 
indicator (depending on the indicator it is the child's age at entry, 
exit, or on the first day). See appendix A for details on risk 
adjusters. We will also risk adjust on the state's foster care entry 
rate for two indicators: Permanency in 12 months for children entering 
foster care and re-entry to foster care in 12 months. Adjusting on age 
allows us to control statistically for the fact that children of 
different ages have different likelihoods of experiencing the outcome, 
regardless of the quality of care a state provides. Adjusting on foster 
care entry rate allows us to control for the impact of the states' case 
mixes as far as the overall risk children in that state have of 
experiencing the outcome. We use entry rate to account for the fact 
that states with lower entry rates tend to have children at greater 
risk for poor outcomes.
    We use a separate ``dummy'' variable for each age when calculating 
the risk adjustment for age. Use of dummy variables is a common 
strategy in regression models to measure the impact of a characteristic 
on an outcome. A dummy variable has a value of 1 or 0 to indicate the 
presence or absence of the characteristic. For example, a child who 
entered care at age 2 will have a ``1'' for the ``age 2'' variable and 
a ``0'' for all others. For all but the first day permanency 
indicators, 19 age dummy variables are used to represent the ages from 
birth to 3 months, four to 11 months, and each year from age 1 through 
17. The first day permanency measure for children in care 12 to 24 
months uses 17 age dummy variables (ages 1 through 17), and the first 
day permanency indicator for children in foster care 24 months of more 
uses 16 age dummy variables (ages 2 through 17). The method requires 
specifying a base or reference age group and for that we use the median 
age.
    We calculate the entry rate as the number of children entering 
foster care during the 12-month period divided by the number of 
children in the state's child population, multiplied by 1,000. We 
obtain the child population data from the population division of the 
U.S. Census Bureau.\8\ This Census data reflect population estimates as 
of July 1st of each year, whereas the 12-month periods CB uses to 
define children entering care are either October to September, or April 
to March. Therefore, we chose to use the Census year closest to the 12-
month period the child entered foster care as the denominator. For 
example, if the indicator follows children who entered care between 
April 1, 2011 and March 31, 2012 (an ``11B/12A'' file in AFCARS file 
conventions), we use child population estimates from the July 2011 
Census estimate. If the 12-month period spanned October 1, 2012 through

[[Page 61250]]

September 30, 2013, we would use population estimates as of July 1, 
2013.
---------------------------------------------------------------------------

    \8\ Population estimates can be downloaded from the U.S. Census 
Bureau's Web site at https://www.census.gov/popest/index.html.
---------------------------------------------------------------------------

    After we perform all the calculations in the model, the result will 
be the state's risk standardized performance. The risk standardized 
performance is the ratio of the number of predicted outcomes over the 
number of expected outcomes, multiplied by the national observed 
performance. For details on how the predicted and expected outcomes are 
calculated, please consult CFSR Technical Bulletin #8 for additional 
information.
    Public Comments and CB Response: Public comments expressed general 
support for risk adjustment, but many more requested more information, 
explanation, and transparency to understand and comment on the concept. 
We have provided more detail in this document to address the issues of 
transparency with precise methodology explanations in CFSR Technical 
Bulletin #8. Additionally, we understand that risk adjustment adds 
complexity to understanding state performance and so we decided as a 
matter of policy to employ it judiciously in this round of reviews and 
use only those variables that had wide support from the field and were 
statistically significant.
    Commenters offered numerous suggestions for possible risk 
adjustment variables, with the most frequently mentioned being child's 
age, foster care entry rate, and whether states included juvenile 
justice youth in their child welfare systems. Other variables the field 
proposed include: The length of time from the date of a report to the 
date of disposition, the state's screen-in rate, how child maltreatment 
is defined statutorily, the degree to which states serve mental health 
populations and adolescents with behavior problems, poverty, parent 
factors and children's individual risk factors such as sibling group or 
severe disabilities.
    CB considered and tested age as a risk adjuster for all indicators 
and found it to be statistically significant so we are including it as 
a variable for all indicators. We considered and tested whether the 
state's foster care entry rate should be used for permanency in 12 
months for children entering foster care, re-entry to foster care in 12 
months and placement stability. We found that the foster care entry 
rate was statistically significant for permanency in 12 months for 
children entering foster care and re-entry to foster care in 12 months 
and are using those. We found that foster care entry rates were not 
statistically significant for placement stability. We did not consider 
using foster care entry rate as an adjuster for the two permanency 
indicators for children in foster care on the first day. This is 
because children in foster care on the first day of the period will 
include children who entered in various years, and therefore an entry 
rate using data from a single year may not adequately reflect the 
experience with every child followed in the indicator. For a similar 
reason, entry rate was not considered for the maltreatment in foster 
care indicator. This indicator is based on children in foster care 
during a 12-month period. Although this indicator includes children who 
entered during the 12-month period, it also includes children who were 
in foster care on the first day of the period whose entry could have 
occurred at any point in the past.
    For the recurrence of maltreatment indicator, we considered as a 
risk adjuster the state's screen-in rate, defined as the number of 
referrals the state screens in per 1,000 children in the child 
population. However, we decided against using this adjustment because 
its impact on the outcome is unclear and may have unintended 
consequences. State's child protective services policies are still 
under considerable fluctuation, especially with the varied 
implementation of differential response and structured decision-making. 
These and other policies that states are implementing may affect 
screen-in rates in unclear ways, so it would be challenging to explain 
what the adjustment is doing. We believe more research on the impact of 
adjusting on screen-in rates is needed before implementing this into 
the CFSRs.
    Despite the call by some commenters to risk adjust for demographic 
variables, a few commenters argued that doing so could unintentionally 
relieve providers of their responsibility to work diligently to reunify 
vulnerable populations. Further, the commenters noted that child 
welfare agencies have a moderate degree of influence over the nature 
and adequacy of the services being provided to these populations and 
that adjusting for demographic variables could mask the disparate 
negative experiences of higher-risk populations. CB believes the 
limited use of risk adjustment at this time mitigate some of the 
concerns expressed in these comments. CB would also like to note that 
states are still encouraged to examine observed performance for 
children by age, sex, race and other demographic variables. This level 
of analysis will help uncover disparities in outcomes for certain 
populations based on their demographics.
    Many of the suggested risk adjustment variables related to the 
programmatic aspects of the state's child welfare program, such as 
whether the state child welfare agency serves youth who are involved in 
the juvenile justice system. Some commenters offered alternative 
approaches to risk adjustment including focusing on systemic and 
environmental variables at the state level. We note that state program 
features are not readily identifiable in the administrative data that 
states submit to CB at this time. However, risk adjusting on additional 
state-level variables is an important area of research, and CB 
encourages researchers to continue to explore the challenges and 
advantages of implementing such risk adjustment in child welfare.
    Some commenters offered alternative approaches to risk adjustment 
that involved dividing some of the data indicators by sub-populations. 
CB considered dividing the data indicators by sub-populations as 
stratifying performance by sub-populations is a useful strategy to see 
how outcomes vary for children from different backgrounds and 
experiences. However, in the context of the CFSR, we chose not to 
pursue this approach because of the unmanageable set of indicators it 
would produce. For example, if we grouped child age into five groups as 
is commonly done, and had separate indicators for each age group, the 
result would be 35 indicators (7 indicators by 5 age groups) based on 
age, and presumably 35 separate national standards, and so forth. 
Instead, we chose to implement a risk adjustment strategy that is 
widely practiced and can incorporate multiple risk adjustment variables 
into a single outcome.
    Some commenters questioned whether CB would provide risk adjusted 
information to local jurisdictions that would likely need to be 
responsible for implementing changes based on the states' performance 
on the indicators. We note that these same models could be implemented 
at the state level, using as the focus of analysis the county (instead 
of the state, as the CB is doing). Details about technical assistance 
available for states interested in performing similar analyses is 
forthcoming as are further details on the information that will be 
available to states in data profiles as we finalize them.
    A commenter requested clarity on the consequences for program 
improvement if a state's observed score meets the national standard, 
but the state's risk adjusted performance does not. In this situation 
CB will still require the state to enter into program improvement. This 
is because the state's observed performance is not the most precise

[[Page 61251]]

measure of the state's performance after considering its case mix and 
size in the context of the performance of other states with similar 
case mixes.
    Categorizing State Performance relative to the National Standards: 
A state's risk standardized performance can be compared directly to the 
national observed performance to determine if the state's risk 
standardized performance is statistically higher or lower than the 
national observed performance. To make this assessment, CB calculates 
approximate 95% interval estimates around each state's risk 
standardized performance. For details on how these interval estimates 
are calculated, see Technical Bulletin #8. CB will compare each state's 
interval estimate to the national observed performance, and assign each 
state to one of three groups:
     ``No different than national performance'' if the 95% 
interval estimate surrounding the state's risk standardized performance 
includes the national observed performance.
     ``Higher than national performance'' if the entire 95% 
interval estimate surrounding the state's risk standardized performance 
is higher than the national observed performance.
     ``Lower than national performance'' if the entire 95% 
interval estimate surrounding the state's risk standardized performance 
is lower than the national observed performance.
    Whether it is desirable for a state to be higher or lower than the 
national performance depends on the indicator. For the indicators 
assessing permanency in 12 months for the three cohorts, a higher value 
is desirable. For these indicators if the state's risk standardized 
performance is ``lower than national performance'' we will consider the 
state not to have met the national standard and will require program 
improvement. For the remaining indicators, a lower value is desirable. 
If a state's risk standardized performance is ``higher than the 
national performance'' for these indicators, we will consider the state 
not to have met the national standard and will require program 
improvement. For all indicators, we will consider states that are ``no 
different than national performance'' to have met the national standard 
and no program improvement will be required.
    Public Comments and CB Response: A commenter requested 
clarification on whether the national standards will remain fixed over 
the course of the round. The national standard will remain the fixed 
standard over round 3 of the CFSRs. However, there are situations in 
which a state's more recent data will be used to evaluate their 
performance relative to the standard. Due to the staggered schedule of 
CFSRs, some states will begin their onsite review one to three years 
after the establishment of the national standards and any initial 
assessment we provide of where states fall relative to the standards. 
Or a state may resubmit data for an earlier reporting period prior to 
its review. In preparation for these states' statewide assessments, CB 
will rerun the national model using the state's most current data 
applicable, but using the fixed data from the original reference 
population (i.e., the fixed data for all other states). This allows us 
to assess if the state, given its most recent performance, would now 
meet the national standard had it performed this way when we provide 
each state's performance initially.
    Sources and Data Periods: The datasets used for the national 
standard calculations depend on the indicator. Some indicators require 
more data periods than others. For example, the re-entry indicator 
requires six report periods of AFCARS data. This is because the cohort 
of children used requires a look at all children who enter foster care 
over a 12-month period; then they are followed for another 12-months to 
establish whether they have exited to permanency; then they are 
followed for a subsequent 12-months after their exit to see if they 
reenter foster care. Attachment A specifies the data periods that will 
be used for calculating the national standard for each indicator.

Monitoring Statewide Data Indicators in Program Improvement Plans

    CB will require states that do not meet the national standard for 
an indicator to include improvement on that indicator in its program 
improvement plan. If we are unable to determine a state's performance 
on an indicator due to data quality issues, we will also require the 
state to include that indicator in its program improvement plan. Data 
quality levels that prevent CB from identifying a state's performance 
are described in the next section and are specified in Attachment C. 
For two of the statewide data indicators, permanency in 12 months for 
children entering foster care and re-entry to foster care, CB will 
determine performance for program improvement purposes on one indicator 
in concert with the other as a companion measure. The key components 
for setting improvement goals and monitoring a state's progress over 
the course of a program improvement plan involve calculating baselines, 
setting improvement goals, and when companion measures are included in 
an improvement plan, also establishing thresholds. CB will set 
improvement goals and thresholds in part relative to each state's past 
performance.
    A state can complete its program improvement plan successfully with 
regard to the indicators by meeting its improvement goal and staying 
above the threshold for its companion measure, if applicable. The 
determination that the state has been successful can be made during the 
program improvement period or the non-overlapping data period. The non-
overlapping data period follows the end of the program improvement plan 
and is the period in which CB is evaluating the state's resulting 
performance as evidenced in the data. Alternatively, CB can relieve a 
state of any further obligation to improve for CFSR purposes if the 
state meets the national standard for an indicator prior to or during 
the course of program improvement monitoring.
    Companion Measures: If a state has a program improvement plan that 
includes improving on the indicator permanency in 12 months for 
children entering foster care, CB's determination of whether the state 
has improved successfully will take into consideration its performance 
on the re-entry to foster care indicator as a companion measure. 
Specifically, the state must not allow performance on the companion 
measure to get worse beyond a certain level from its baseline 
performance. Thresholds are established as the inverse of performance 
goals, to provide the bounds in which states should not worsen. For 
example, a state must stay below a threshold for the companion re-entry 
to foster care indicator as well as achieve its goal on the permanency 
in 12 months for children entering foster care indicator to 
successfully complete the program improvement plan. The reverse is also 
true. If a state must improve on the re-entry to foster care indicator 
in its program improvement plan, it must not get worse than the 
threshold established for permanency in 12 months for children entering 
foster care. For details about threshold calculations, please see the 
section below and CFSR Technical Bulletin #8.
    Public Comments and CB Response: Several commenters expressed 
strong support for the use of companion measures, but requested 
technical assistance to support states' work in translating these 
concepts and the calculations for thresholds. CB will work to provide 
states with clear explanations and visuals within their data profiles 
and technical materials of how the companion measures can be 
interpreted and are calculated. On the other hand, a commenter 
requested that

[[Page 61252]]

we acknowledge that there could be no evidence or justification that 
one indicator contributed to the result of the other. CB was careful to 
select the companion measures because of the close connection between 
the practices of one and the other. CB has no plans to demonstrate for 
program improvement purposes that when a state increases its exits to 
permanency within 12 months and there is a subsequent increase reentry 
that there is causal relationship between the two (or that decreased 
reentries was caused by decreased exits to permanency). However, the 
goal is not to show causality; the concept is that if a state is unable 
to keep from getting markedly worse on the companion measure it cannot 
be considered to have successfully improved on the primary indicator as 
it indicates that something in the state's practices was problematic 
for the related area of permanency. It will always be incumbent on the 
state, working in concert with CB, to drill down into the data and 
assess its practice to understand whether, where and how practices can 
be aligned to ensure that children's needs are met for permanency to be 
achieved timely and appears to be long lasting.
    State Baselines: CB will set the baseline for each statewide data 
indicator included in a program improvement plan at the state's 
observed performance on that indicator for the most recent year of 
available data at the beginning of the program improvement plan. 
However, just as there are multiple data periods used for the 
development of the national standards, multiple time periods are needed 
to evaluate the state's baseline performance at the time of the PIP and 
then subsequently throughout the program improvement period. Since the 
CFSR review schedule is staggered, the applicable year or data periods 
used in establishing the baseline will vary. For example, a state with 
an onsite review in April 2015 (FY 2015) and enters into a program 
improvement plan in September 2015 that includes the recurrence of 
maltreatment indicator would have its baseline calculated based on its 
performance in FY 2014. Since recurrence of maltreatment requires two 
years of NCANDS data, the applicable data periods would be FY 2013 and 
FY 2014.
    Public Comments and CB Response: No comments were received on the 
proposal in this area and no changes were made.
    State Improvement Goals and Thresholds: We will establish 
improvement factors for program improvement goals and thresholds (if 
applicable) for the data indicators based on the variability in a 
state's observed performance in the three most recent years of data. 
The improvement factor is multiplied to the state's observed 
performance for each statewide data indicator needing improvement in 
the most recent year available at the start of the improvement plan. 
Thresholds are calculated for companion measures and reflect levels of 
performance decline that the state cannot cross for us to consider the 
state to have successfully completed the primary statewide indicator. 
Thresholds are simply the inverse of the improvement goals.
    The resulting improvement goal or threshold may be limited or 
increased for a state based on minimum and maximum levels for 
improvement that we have set for each indicator. We will set the 
minimum and maximum improvement levels so that no states are required 
to improve by more than the amount of improvement at the 50th 
percentile, and all states engaged in a program improvement plan are to 
improve by at least the amount of improvement at the 20th percentile 
(or 80th percentile, depending on whether higher or lower performance 
is preferable on the indicator). We will then use these values to 
replace the otherwise resulting improvement goal/threshold. The 
technical detail of the several steps we will take for these 
calculations are presented in CFSR Technical Bulletin #8 as well as a 
full discussion about the methods chosen and our rationales for doing 
so.
    Table 2 provides the range of improvement factors for each 
statewide data indicator. If the state is required to improve for an 
indicator, the state will use their most recent year of observed 
performance as their baseline in determining the applicable improvement 
factor. For example, for the permanency in 12 months for children 
entering foster care indicator, improvement factors will be no lower 
than 1.035 and no higher than 1.057. If the value generated by a 
state's own prior performance generates a value within that range, they 
would use that value. For example, if the baseline was 40% and the 
state has to show the most improvement, they would simply multiply 
1.057 with the baseline and obtain a goal of 42.28%.

     Table 2--Minimum and Maximum Improvement on the Statewide Data
                               Indicators
------------------------------------------------------------------------
  Statewide data indicators for safety outcome 1     Minimum    Maximum
------------------------------------------------------------------------
Maltreatment in Foster Care.......................      0.922      0.849
Recurrence of Maltreatment........................      0.953      0.910
------------------------------------------------------------------------
Statewide data indicators for permanency outcome 1   Minimum    Maximum
------------------------------------------------------------------------
Permanency in 12 Months for Children Entering           1.035      1.057
 Foster care......................................
Permanency in 12 Months for Children in Foster          1.040      1.074
 Care 12 to 23 months.............................
Permanency in 12 Months for Children in Foster          1.034      1.080
 Care 24 Months or More...........................
Re-Entry to Foster Care in 12 Months..............      0.912      0.867
Placement Stability...............................      0.953      0.912
------------------------------------------------------------------------

    Public Comment and CB Response: Some commenters expressed support 
for the program improvement methodology related to statewide data 
indicators as an overall concept. Such comments included support for 
the use of companion measures and thresholds as well as the use of 
historical performance as the basis for performance improvement 
targets. However, others commented that they were confused about the 
methods we proposed and that they would have difficulty explaining them 
to stakeholders. Commenters requested more explicit descriptions on how 
we will establish goals and threshold and on the consequences for 
states that have performance that drops below a threshold during 
program improvement.
    Further, a number of commenters stated that there was not enough 
information in the original document to inform further comments and 
challenged a number of our methods chosen as technically inaccurate. 
These commenters noted concerns with

[[Page 61253]]

establishing states performance improvement goals based on only three 
data points; using four standard deviations as the distance required 
for improvement; employing the Chebyshev's theorem; and how the 
application of these techniques could lead to states failing to meet 
the minimal level of improvement. As alternatives, commenters suggested 
the use of two standard deviations; relying upon available data, such 
as historical AFCARS and NCANDS data; applying the Empirical Rule 
rather than using the Chebyshev theorem; and allowing performance goals 
to be mutually negotiated between states and ACF.
    We made several changes in response to these comments. First, we 
have provided a more thorough explanation of our methods and rationales 
for those methods in CFSR Technical Bulletin #8 as we believe it is 
important for states to see the full detail of our methods. We also 
took another look at the application of four standard deviations in 
developing the improvement factors given the concerns about setting 
goals that were too large. After we conducted additional analysis of 
the resulting improvement factors we agree with commenters that in some 
circumstances employing the 4 standard deviations would result in more 
aggressive improvement factors than round 2 even when also setting 
minimum and maximum improvement expectations at the 80th and 20th 
percentiles. In response, we have adjusted the approach to use 2 
standard deviations and also to set the maximum improvement of all 
states' expectations to the 50th percentile of all states' original 
improvement factors, when calculated for every state and ordered from 
highest to lowest.
    Another commenter requested additional information on whether 
improvement goals and thresholds for the statewide data indicators can 
be negotiated. As was the case in the prior round, we have standardized 
the approach to establish improvement factors that are applied to the 
state's baseline and are not negotiating the amount of improvement on 
the indicators. However, we will negotiate with a state how to design 
its program improvement approaches to attain the improvement goals. We 
will also still allow a state the opportunity during a program 
improvement plan to provide data that can be verified, reproduced and 
otherwise approved by ACF, as evidence that the state has met the 
requirement for attaining the required improvement.
    A commenter requested clarification on whether the same multi-level 
modeling and risk adjustment will be utilized in assessing a state's 
performance over time to account for fluctuations in the state's 
population. When assessing a state's performance over time to determine 
whether or not states meet program improvement plan goals, we will not 
be using the same multi-level modeling and risk adjustment approach. We 
will be using the state's own observed performance on the indicators, 
regardless of changes in the state's population to make these 
determination.
    Successful completion of program improvement relative to the 
indicators: Although not specifically outlined in our original 
proposal, we wanted to clarify that a state can complete its program 
improvement plan successfully with regard to the indicators in a couple 
of ways. One is by meeting its improvement goal and not exceeding the 
threshold for its companion measure, if applicable, at some point 
before the end of the program improvement monitoring. Alternatively, CB 
can relieve a state of any further obligation to improve for CFSR 
purposes if the state meets the national standard for an indicator 
prior to the approval of a program improvement plan or during the 
course of program improvement monitoring. This latter provision also 
means that a state need not meet a program improvement goal (by 
application of the improvement factor or the minimum or maximum 
improvement level) for an indicator if the state first meets the 
national standard for that indicator.

Data

    Data Profiles: We will provide data profiles of state performance 
to each state before the state's CFSR on all seven of the statewide 
data indicators and other contextual data available from AFCARS and 
NCANDS. This data profile will assist the state to develop its 
statewide assessment and begin planning for program improvement, if 
appropriate. In addition, we will provide data profiles semi-annually 
to assist states in measuring progress toward the goals identified in 
the program improvement plan.
    Public Comment and CB Response: Several commenters appreciated our 
commitment to providing data semi-annually, recognizing their 
importance in preparing for CFSRs and improving practice on a more 
general basis. Several commenters requested specific categories of 
information that would be beneficial for continuous quality improvement 
activities. Requested information included disaggregated data for the 
statewide data indicators, a rate of placement that is not tied to 
federal performance standards, and indicators of juvenile justice case 
type and a child's Indian Child Welfare Act (ICWA) eligibility and 
status.
    In CFSR Technical Bulletin #8 we have outlined the content of the 
data profiles that we will send to states so that they can evaluate 
their performance in completing the statewide assessment. We have also 
outlined our plans for data profile content that will be sent to states 
during program improvement, if necessary. We welcome continued input 
from states on the content of program improvement profiles that will 
support their analysis in developing strategies for improvement. 
However, we also encourage states to conduct analysis on any data 
available to the state, including data that is not submitted to CB such 
as juvenile justice case type and ICWA status, to inform their 
understanding of their performance and measure progress.

Data Quality: Excluding States From National Standards or State 
Performance

    Setting national standards and measuring state performance on 
statewide data indicators for CFSR purposes relies upon the states 
submitting high-quality data to AFCARS and NCANDS. Therefore we will 
exclude states that have data quality issues that exceed the data 
quality limits established from the model we use to calculate the 
national standard (i.e., the national observed performance) and 
estimate states' risk adjusted performance.
    Because errors in the data can misrepresent state performance, we 
made the decision to remove a state from the analysis entirely if they 
exceed certain limits on the data quality checks. We reviewed state-by-
state performance on each data quality item before establishing these 
limits. Because we do not want to be too strict and exclude a great 
number of states, we were conservative and set the limits high for 
common issues (e.g. 10% for dropped cases). However, some checks are 
critical to calculations (such as a count of placements for the 
placement stability measure), and we set the limits a bit lower (5%) in 
order to not misrepresent state performance.

Data Quality: Case-Level Exclusions

    For those states that do not exceed the data quality thresholds but 
still have identified data quality problems, we will include the state 
in national standards calculations and measure state performance but we 
will exclude child-level records with missing or

[[Page 61254]]

invalid data on elements needed to determine the child's outcome and 
perform the risk adjustment. For example, if the risk adjustment for an 
indicator includes age at entry, a child whose age at entry cannot be 
determined (due to a missing date of birth) will not be included in the 
analysis. For each indicator, we will provide each state with a list of 
records that were excluded from the analyses.
    Public Comments and CB Response: Two commenters expressed support 
for our approach to addressing data quality issues in estimating 
national standards and a state's risk adjusted performance. One 
commenter urged us to hold states responsible for producing ``high-
quality, consistent, and complete data'' pointing out that we have not 
found any state in the past 13 years, to be in full compliance with the 
AFCARS standards through ACF's AFCARS Assessment Reviews. The other 
commenter commended us for recognizing that quality data is critical to 
assessing performance. Another commenter was concerned that the 
thresholds meant that the standards could not be considered national; 
while another wanted the thresholds raised to allow more states to 
either participate in the national standard calculations or have their 
state performance evaluated.
    We concur with those commenters that believe that data quality 
standards are necessary to ensure the integrity of our performance 
assessment. We believe we have maintained an appropriate balance in 
setting data quality thresholds so as not to exclude states 
unreasonably. In terms of the national standards, the number of states 
excluded was relatively few. For the indicators permanency by 12 months 
for the 12 to 23 month and 2 or more years first day cohorts, one state 
was excluded from the national standard calculation. For the permanency 
by 12 months entry cohort indicator, three states were excluded. For 
the reentry to foster care, recurrence of maltreatment and maltreatment 
in foster care indicators, four states were excluded. Six states were 
excluded from the calculation of the national standard for the 
placement stability indicator. We will continue to work with states 
that have their data excluded from the national standards or evaluation 
of state performance and advise on how they can address the data 
quality issues in their systems.

    (Authority: 42 U.S.C. 1320a-1a; 45 CFR 1355.31-37.)

Mark Greenberg,
Acting Commissioner, Administration on Children, Youth and Families.

Attachment A: Statewide Data Indicators

--------------------------------------------------------------------------------------------------------------------------------------------------------
       Category            Measure title      Measure description       Denominator            Numerator        Exclusions & notes     Risk adjustment
--------------------------------------------------------------------------------------------------------------------------------------------------------
Safety...............  Maltreatment in       Of all children in    Of children in        Of children in the    --If a state          --Age at entry (for
                        Foster Care.          foster care during    foster care during    denominator, the      provides incident     children entering)
                                              a 12-month period,    a 12-month period,    total number of       dates, records with   or age on first
                                              what is the rate of   the total number of   substantiated or      an incident date      day of the 12-
                                              victimization per     days these children   indicated reports     occurring outside     month period (for
                                              day of foster care?   were in foster care   of maltreatment (by   of the removal        children already
                                             For national           as of the end of      any perpetrator)      episode will be       in care).
                                              standard              the 12-month period   during a foster       excluded, even if
                                              calculation, uses     \a\.                  care episode within   report dates fall
                                              AFCARS periods                              the 12-month period   within the episode.
                                              2013A and 2013B and                         \b\.                 --Complete foster
                                              NCANDS FY2013 Child                                               care episodes
                                              File..                                                            lasting <8 days are
                                                                                                                excluded..
                                                                                                               --Any report that
                                                                                                                occurs within the
                                                                                                                first 7 days of
                                                                                                                removal is
                                                                                                                excluded..
                                                                                                               --Victims age 18 or
                                                                                                                more are excluded,
                                                                                                                as well as youth in
                                                                                                                foster care at 18
                                                                                                                or more. For youth
                                                                                                                who start out as 17
                                                                                                                years of age and
                                                                                                                turn 18 during the
                                                                                                                period, any time in
                                                                                                                foster care beyond
                                                                                                                his/her 18th
                                                                                                                birthday is not
                                                                                                                counted in the
                                                                                                                denominator.
                                                                                                               --Cases are matched
                                                                                                                across AFCARS and
                                                                                                                NCANDS using AFCARS
                                                                                                                ID.
--------------------------------------------------------------------------------------------------------------------------------------------------------

[[Page 61255]]

 
Safety...............  Recurrence of         Of all children who   Number of children    Number of children    Relies primarily on   --Age at initial
                        Maltreatment.         were victims of a     with at least one     in the denominator    the report date to    victimization.
                                              substantiated or      substantiated or      that had another      determine whether
                                              indicated report of   indicated report of   substantiated or      the maltreatment
                                              maltreatment during   maltreatment in a     indicated report of   occurred in the
                                              a 12-month period,    12-month period.      maltreatment within   first 12-month
                                              what percent were                           12 months of their    period; therefore,
                                              victims of another                          initial report.       if a case does not
                                              substantiated or                                                  reach disposition
                                              indicated report of                                               until the following
                                              maltreatment within                                               12-month period but
                                              12 months of their                                                has a report date
                                              initial report?                                                   in the first, we
                                             For national                                                       include it.
                                              standard                                                         --If subsequent
                                              calculation, uses                                                 report is within 14
                                              NCANDS FY 2012 and                                                days, we do not
                                              FY 2013 Child                                                     count it..
                                              Files..                                                          --If incident date
                                                                                                                indicates that two
                                                                                                                reports refer to
                                                                                                                the same incident,
                                                                                                                we do not count it..
                                                                                                               --If report date is
                                                                                                                prior to the first
                                                                                                                12 months, we
                                                                                                                exclude it..
                                                                                                               --Youth age 18 or
                                                                                                                more are excluded
                                                                                                                from the measure..
--------------------------------------------------------------------------------------------------------------------------------------------------------
Permanency...........  Permanency in 12      Of all children who   Number of children    Number of children    --Children in foster  --Age at entry.
                        Months for Children   enter foster care     who enter foster      in the denominator    care <8 days are     --State's foster
                        Entering Foster       in a 12-month         care in a 12-month    who discharged to     excluded.             care entry rate.
                        Care.                 period, what          period.               permanency within    --Children who enter
                                              percent discharged                          12 months of          foster care at age
                                              to permanency                               entering foster       18 or more are
                                              within 12 months of                         care and before       excluded..
                                              entering foster                             turning 18.          --Trial home visit
                                              care? \c\                                                         adjustment is
                                             For national                                                       applied..
                                              standard
                                              calculation, uses
                                              AFCARS periods
                                              2011B through
                                              2013A..
--------------------------------------------------------------------------------------------------------------------------------------------------------

[[Page 61256]]

 
Permanency...........  Permanency in 12      Of all children in    Number of children    Number of children    --Children age 18 or  --Age on first day.
                        Months for Children   foster care on the    in foster care on     in the denominator    more on the first
                        in Foster Care 12-    first day of a 12-    the first day of a    who discharged from   day of the 12-month
                        23 Months.            month period who      12-month period,      foster care to        period are excluded.
                                              had been in foster    who had been in       permanency within    --Trial home visit
                                              care (in that         foster care (in       12 months of the      adjustment is
                                              episode) between 12   that episode)         first day of the 12-  applied..
                                              and 23 months, what   between 12 and 23     month period and
                                              percent discharged    months.               before turning 18.
                                              from foster care to
                                              permanency within
                                              12 months of the
                                              first day of the 12-
                                              month period?
                                             For national
                                              standard
                                              calculation, uses
                                              AFCARS periods
                                              2013B and 2014A..
--------------------------------------------------------------------------------------------------------------------------------------------------------
Permanency...........  Permanency in 12      Of all children in    Number of children    Number of children    --Children age 18 or  --Age on first day.
                        Months for Children   foster care on the    in foster care on     in the denominator    more on the first
                        in Foster Care 24     first day of a 12-    the first day of a    who discharged from   day of the 12-month
                        Months or More.       month period, who     12-month period,      foster care to        period are
                                              had been in foster    who had been in       permanency within     excluded.
                                              care (in that         foster care (in       12 months of the     --Trial home visit
                                              episode) for 24       that episode) for     first day of the 12-  adjustment is
                                              months or more,       24 months or more.    month period and      applied..
                                              what percent                                before turning 18.
                                              discharged to
                                              permanency within
                                              12 months of the
                                              first day of the 12-
                                              month period?
                                             For national
                                              standard
                                              calculation, uses
                                              AFCARS periods
                                              2013B and 2014A..
--------------------------------------------------------------------------------------------------------------------------------------------------------
Permanency...........  Re-Entry to Foster    Of all children who   Number of children    Number of children    --Children in foster  --Age at exit.
                        Care in 12 Months.    enter foster care     who enter foster      in the denominator    care <8 days are     --State's foster
                                              in a 12-month         care in a 12-month    who re-enter foster   excluded.             care entry rate.
                                              period, who           period and            care within 12       --Children who enter
                                              discharged within     discharged within     months of their       or exit foster care
                                              12 months to          12 months to          discharge.            at age 18 or more
                                              reunification, live   reunification, live                         are excluded,.
                                              with relative, or     with relative(s),                          --If a child has
                                              guardianship, what    or guardianship.                            multiple re-entries
                                              percent re-enter                                                  within 12 months of
                                              foster care within                                                their discharge,
                                              12 months of their                                                only his first re-
                                              discharge? \a\                                                    entry is selected..
                                             For national
                                              standard
                                              calculation, uses
                                              AFCARS periods
                                              2011B through
                                              2014A..
--------------------------------------------------------------------------------------------------------------------------------------------------------

[[Page 61257]]

 
Permanency...........  Placement Stability.  Of all children who   Of children who       Of children in the    --Children in foster  --Age at entry.
                                              enter foster care     enter foster care     denominator, the      care <8 days are
                                              in a 12-month         in a 12-month         total number of       excluded.
                                              period, what is the   period, the total     placement moves      --Children who enter
                                              rate of placement     number of days        during the 12-month   foster care at age
                                              moves per day of      these children were   period \e\.           18 or more are
                                              foster care?          in foster care as                           excluded. For youth
                                             For national           of the end of the                           who enter at 17
                                              standard              12-month period \d\.                        years of age and
                                              calculation, uses                                                 turn 18 during the
                                              AFCARS periods                                                    period, any time in
                                              2013B and 2014A..                                                 foster care beyond
                                                                                                                his/her 18th
                                                                                                                birthday or
                                                                                                                placement changes
                                                                                                                after that date are
                                                                                                                not counted..
                                                                                                               --The initial
                                                                                                                removal from home
                                                                                                                (and into care) is
                                                                                                                not counted as a
                                                                                                                placement move.
--------------------------------------------------------------------------------------------------------------------------------------------------------
Notes: The letters `A' and `B' are shorthand for the six-month AFCARS reporting periods. The `A' period spans October 1st-March 31st, and the `B' period
  spans April 1st-September 30th of any given year. The year always refers to the year in which the six-month period ends. For example, 2014A refers to
  the six month period of 10/1/2013 through 3/31/2014.
\a\ For example, if during the 12-month period there were two children in foster care, one child for 10 days (1st episode), the same child for 40 days
  (2nd episode), and the other child for 100 days (his only episode), the denominator would = 150 days (10+40+100).
\b\ For example, if during the 12-month period there were two children in foster care, and one child had 3 substantiated or indicated reports and the
  other had 1 such report, the numerator would = 4 reports (3+1).
\c\ If a child has multiple entries during the 12-month period, only the first entry in the 12-month period is selected.
\d\ For example, if during the 12-month period two children entered care, one child for 10 days and the other child for 100 days, the denominator would
  be 110 days (10+100).
\e\ For example, if during the 12-month period two children entered care, and one child had 3 moves and the other had 1 move, the numerator would = 4
  moves (3+1).

Attachment B: Comparison of Data Measures--CFSR Round 2 and Round 3

----------------------------------------------------------------------------------------------------------------
                                                  CFSR round 3      Comparable CFSR round     How and why it's
      Category            Measure title            indicator              2 measure               changed
----------------------------------------------------------------------------------------------------------------
Safety..............  Maltreatment in        Of all children in     Of all children in     In the CFSR 2
                       foster care.           foster care during a   foster care during     measure, counts of
                                              12-month period,       the reporting          children not
                                              what is the rate of    period, what percent   maltreated in foster
                                              victimization per      were not victims of    care are derived by
                                              day \a\ of foster      substantiated or       subtracting the
                                              care?                  indicated              NCANDS count of
                                                                     maltreatment by a      children maltreated
                                                                     foster parent or       by foster care
                                                                     facility staff         providers from the
                                                                     member?                total count of all
                                                                                            children placed in
                                                                                            foster care, as
                                                                                            reported in AFCARS.
                                                                                            Because of improved
                                                                                            reporting by states,
                                                                                            we now link AFCARS
                                                                                            and NCANDS data
                                                                                            using the child ID
                                                                                            and determine if
                                                                                            maltreatment
                                                                                            occurred during a
                                                                                            foster care episode,
                                                                                            improving accuracy
                                                                                            on the indicator.

[[Page 61258]]

 
                                                                                           This also allows us
                                                                                            to expand the
                                                                                            measure to include
                                                                                            all types of
                                                                                            perpetrators
                                                                                            (including, for
                                                                                            example, parents)
                                                                                            under the assumption
                                                                                            that states should
                                                                                            be held accountable
                                                                                            for keeping children
                                                                                            safe from harm while
                                                                                            in the care of the
                                                                                            state, no matter who
                                                                                            the perpetrator is.
----------------------------------------------------------------------------------------------------------------
Safety..............  Recurrence of          Of all children who    Of all children who    We will use a full 12-
                       maltreatment.          were victims of        were victims of        month period rather
                                              substantiated or       substantiated or       than only 6 months
                                              indicated              indicated              to capture the
                                              maltreatment           maltreatment           denominator, to
                                              allegation during a    allegation during      create more stable
                                              12 month period,       the first 6 months     estimates. We will
                                              what percent were      of the reporting       also track them for
                                              victims of another     period, what percent   another full 12
                                              substantiated or       were not victims of    month to see if
                                              indicated              another                there is a recurring
                                              maltreatment           substantiated or       maltreatment.
                                              allegation within      indicated
                                              the next 12 months?    maltreatment
                                                                     allegation within a
                                                                     6-month period?
                                                                                           The indicator also
                                                                                            includes these
                                                                                            changes: If the
                                                                                            subsequent report is
                                                                                            within 14 days, we
                                                                                            will not count it.
                                                                                            While the measure
                                                                                            relies on report
                                                                                            date, we will also
                                                                                            make use of the
                                                                                            incident data, when
                                                                                            available. If the
                                                                                            incident date
                                                                                            indicates that two
                                                                                            reports refer to the
                                                                                            same incident, we
                                                                                            will not count it.
                                                                                           Finally, youth age 18
                                                                                            or more are excluded
                                                                                            from the measure.
----------------------------------------------------------------------------------------------------------------
Permanency..........  Permanency in 12       Of all children who    Composite 1.3: Of all  We now count all
                       months for children    enter foster care in   children entering      types of permanency
                       entering foster care.  a 12-month period,     foster care for the    (reunification, live
                                              what percent           first time in a 6-     with relative,
                                              discharged to          month period, what     adoption or
                                              permanency within 12   percent discharged     guardianship) as
                                              months of entering     to reunification (or   having `met' the
                                              foster care?           live with relative)    indicator.
                                                                     within 12 months of   We also expanded the
                                                                     entering foster care   measure to include
                                                                     or by the time they    all children who
                                                                     reached 18?            entered foster care
                                                                                            that year; not just
                                                                                            those on their first
                                                                                            removal episode.
                                                                                           We also expanded the
                                                                                            window of time for
                                                                                            the entry cohort to
                                                                                            a full year instead
                                                                                            of 6 months; this
                                                                                            will yield more
                                                                                            stable estimates.
----------------------------------------------------------------------------------------------------------------
Permanency..........  Permanency in 12       Of all children in     In CFSR Round 2, we    We add this cohort to
                       months for children    foster care on the     looked at              allow for children
                       in foster care         first day of a 12-     reunifications         and youth in foster
                       between 12 and 23      month period who had   within 12 months as    care who have
                       months.                been in foster care    part of a measure      already been in
                                              (in that episode)      within Composite 1,    foster care between
                                              between 12 and 23      and we looked at       1 and 2 years to be
                                              months, what percent   adoptions in 24        a focus for
                                              discharged to          months as part of      permanency, as well.
                                              permanency within 12   Composite 2.
                                              months of the first
                                              day?
                                                                                           We expect this
                                                                                            population to have a
                                                                                            higher percentage of
                                                                                            exits to adoption or
                                                                                            guardianship than
                                                                                            those entering care
                                                                                            during the year.
----------------------------------------------------------------------------------------------------------------

[[Page 61259]]

 
Permanency..........  Permanency in 12       Of all children in     Composite 3.1: Of all  Same measure; no
                       months for children    foster care on the     children in foster     change. The
                       in foster care for     first day of a 12-     care on the first      difference is that
                       24 months or longer.   month period who had   day of a 12-month      it is now evaluated
                                              been in foster care    period who had been    on its own, rather
                                              (in that episode)      in foster care (in     than as just one
                                              for 24 months or       that episode) for 2    part of a composite
                                              longer, what percent   or more years, what    measure.
                                              discharged to          percent discharged    We believe it is
                                              permanency within 12   to permanency within   important to hold
                                              months of the first    12 months of the       states accountable
                                              day?                   first day or by the    for getting those
                                                                     time they reached      children and youth
                                                                     18?                    who have been in
                                                                                            foster care for long
                                                                                            periods of time to
                                                                                            permanent homes.
----------------------------------------------------------------------------------------------------------------
Permanency..........  Re-entry in 12 months  Of all children who    Composite 1.4: Of all  The new indicator is
                                              enter foster care in   children discharged    limited to those
                                              a 12-month period      from foster care to    children who entered
                                              and discharged         reunification or       foster care during
                                              within 12 months to    live with a relative   the year, whereas
                                              reunification, live    in a 12-month          the CFSR Round 2
                                              with relative, or      period, what percent   measure counted all
                                              guardianship, what     re-entered foster      children who
                                              percent re-entered     care in less than 12   discharged to
                                              foster care within     months from the date   reunification or
                                              12 months of their     of discharge?          live with relative,
                                              date of discharge?                            regardless of when
                                                                                            they entered foster
                                                                                            care. The purpose of
                                                                                            this focus is in
                                                                                            keeping with the
                                                                                            rationale that new
                                                                                            interventions may
                                                                                            best be monitored in
                                                                                            an entry cohort.
                                                                                            This indicator will
                                                                                            also be used as a
                                                                                            companion measure
                                                                                            with permanency in
                                                                                            12 months, to ensure
                                                                                            that states working
                                                                                            to improve
                                                                                            permanency rates in
                                                                                            their entry cohort
                                                                                            do not see worsening
                                                                                            performance on rates
                                                                                            of re-entry to
                                                                                            foster care.
                                                                                           We also expanded the
                                                                                            denominator to allow
                                                                                            discharges to
                                                                                            guardianship, in an
                                                                                            effort to capture
                                                                                            more discharges to
                                                                                            permanency. Exits to
                                                                                            adoption are not
                                                                                            included because
                                                                                            they cannot be
                                                                                            tracked reliably, as
                                                                                            some states issue
                                                                                            new child
                                                                                            identifiers if a
                                                                                            child who was
                                                                                            previously adopted
                                                                                            enters foster care.
----------------------------------------------------------------------------------------------------------------
Permanency..........  Placement stability..  Of all children who    Composite 4.1: Of all  The proposed
                                              enter foster care in   children served in     indicator controls
                                              a 12-month period,     foster care during     for length of time
                                              what is the rate of    the 12-month period,   in foster care, so
                                              placement moves per    what percent had two   we are looking at
                                              day \b\ of foster      or fewer placement     moves per day of
                                              care?                  settings?              foster care, rather
                                                                                            than children as the
                                                                                            unit of analysis.
                                                                                           The rationale for
                                                                                            using an entry
                                                                                            cohort rather than
                                                                                            all children served
                                                                                            is that our analysis
                                                                                            shows children
                                                                                            entering foster care
                                                                                            tend to move much
                                                                                            more than those
                                                                                            children/youth in
                                                                                            foster care for
                                                                                            longer periods of
                                                                                            time, whose
                                                                                            placements may have
                                                                                            stabilized.
                                                                                           In CFSR Round 2
                                                                                            measure, moves that
                                                                                            took place prior to
                                                                                            the monitoring
                                                                                            period were counted.
                                                                                            Now we only count
                                                                                            those moves that
                                                                                            occur during the
                                                                                            monitoring period.
                                                                                            The initial
                                                                                            placement is not
                                                                                            counted.

[[Page 61260]]

 
                                                                                           The CFSR Round 2
                                                                                            measure treated
                                                                                            children who moved 2
                                                                                            times in an episode
                                                                                            the same as children
                                                                                            who moved 15 times;
                                                                                            both were a failure
                                                                                            to meet the measure.
                                                                                            The new indicator
                                                                                            counts each move, so
                                                                                            it continues to hold
                                                                                            states accountable
                                                                                            for those children/
                                                                                            youth who have
                                                                                            already moved
                                                                                            several times.
----------------------------------------------------------------------------------------------------------------
\a\ The rate may be expressed per 100,000 days because it is such a rare event. Using this metric gives us
  larger numbers that are easier to communicate.
\b\ The rate is expressed per 1,000 days to convert the rate to a metric that gives us larger numbers.

Attachment C: Data Elements Used for Statewide Data Indicators

    For information regarding AFCARS data elements, refer to http://www.acf.hhs.gov/programs/cb/resource/afcars-tb1.
    For information regarding NCANDS data elements, refer to http://www.ndacan.cornell.edu/datasets/pdfs_user_guides/178-NCANDS-child2012v1-User-Guide-and-Codebook.pdf.

--------------------------------------------------------------------------------------------------------------------------------------------------------
                                                       Permanency in  12
   Primary data elements required for calculation       months  (all 3    Re-entry to foster       Placement         Recurrence of      Maltreatment in
                                                          indicators)      care in 12 months       stability         maltreatment         foster care
--------------------------------------------------------------------------------------------------------------------------------------------------------
AFCARS FC Element #1 \1\: Title IV-E Agency.........                  X                   X                   X                  NA                   X
AFCARS FC Element #4: Record Number.................                  X                   X                   X                  NA                   X
AFCARS FC Element #21: Date of Latest Removal.......                  X                   X                   X                  NA                   X
AFCARS FC Element #23: Date of Placement in Current                  NA                  NA                   X                  NA                  NA
 Foster Care Setting................................
AFCARS FC Element #24: Number of Placement Settings                  NA                  NA                   X                  NA                  NA
 during this Removal Episode........................
AFCARS FC Element #56: Date of Discharge from FC....                  X                   X                   X                  NA                   X
AFCARS FC Element #58: Reason for Discharge.........                  X                   X                  NA                  NA                  NA
NCANDS CF Element #4: Child ID......................                 NA                  NA                  NA                   X                  NA
NCANDS CF Element #6: Report Date...................                 NA                  NA                  NA                   X                   X
NCANDS CF Element #27: Child Maltreatment 1--                        NA                  NA                  NA                   X                   X
 Disposition Level \2\..............................
NCANDS CF Element #29: Child Maltreatment 2--                        NA                  NA                  NA                   X                   X
 Disposition Level..................................
NCANDS CF Element #31: Child Maltreatment 3--                        NA                  NA                  NA                   X                   X
 Disposition Level..................................
NCANDS CF Element #33: Child Maltreatment 4--                        NA                  NA                  NA                   X                   X
 Disposition Level..................................
NCANDS CF Element #34: Maltreatment death...........                 NA                  NA                  NA                   X                   X
NCANDS CF Element #145: AFCARS ID...................                 NA                  NA                  NA                  NA                   X
Optional Data Elements:
    AFCARS FC Element #41: Current Placement Setting                  X                  NA                  NA                  NA                  NA
    NCANDS CF #146 Incident Date....................                 NA                  NA                  NA                   X                   X
Additional Data Elements Required for Risk-Adjusted
 Analysis:
    AFCARS FC Element #6: Child's Date of Birth.....                  X                   X                   X                  NA                   X
    NCANDS CF Element #14: Child Age................                 NA                  NA                  NA                   X                  NA

[[Page 61261]]

 
    U.S. Census Bureau: Child Population, by State                X \3\                   X                  NA                   X                   X
     (Used to derive state foster care entry rates).
--------------------------------------------------------------------------------------------------------------------------------------------------------
\1\ The elements are numbered by their position in the flat ASCII files submitted by states to these reporting systems. These numbering schema are
  specific to the files utilized by ACYF. Files obtained through the National Data Archive on Child Abuse and Neglect (NDACAN) may have a slightly
  different order.
\2\ Definition of `victim' includes all children with a disposition level (for any of up to four maltreatments per child) of: a) Substantiated, or b)
  Indicated. These do not propose including differential response victims. Victims also include children who died as a result of maltreatment.
\3\ Relevant to Permanency by 12 months for the entry cohort only.

Attachment D: Data Quality Items, Limits, and Applicable Measures

--------------------------------------------------------------------------------------------------------------------------------------------------------
                                                                                                                   Permanency in 12
                                                                                                                     months (all 3
                  Data quality item                   Data quality limit    Maltreatment in      Recurrence of     indicators) & re-       Placement
                                                                              foster care        maltreatment       entry to foster        stability
                                                                                                                   care in 12 months
--------------------------------------------------------------------------------------------------------------------------------------------------------
AFCARS--Cross File Checks:
    Dropped cases...................................               >10%                   X                 n/a                   X                   X
    AFCARS IDs don't match from one period to next..               >40%                   X                 n/a                   X                   X
AFCARS--Within-file checks:
    Missing date of birth...........................                >5%                   X                 n/a                   X                   X
    Missing date of latest removal..................                >5%                   X                 n/a                   X                   X
    Missing # of placement settings.................                >5%                 n/a                 n/a                 n/a                   X
    Date of birth after date of entry...............                >5%                   X                 n/a                   X                   X
    Date of birth after date of exit................                >5%                   X                 n/a                   X                   X
    Age at entry greater than 21....................                >5%                   X                 n/a                   X                   X
    Age at discharge greater than 21................                >5%                   X                 n/a                   X                   X
    In foster care more than 21 years...............                >5%                   X                 n/a                   X                   X
    Enters and exits care the same day..............                >5%                   X                 n/a                   X                   X
    Exit date is prior to removal date..............                >5%                   X                 n/a                   X                   X
    Missing discharge reason (exit date exists).....                >5%                 n/a                 n/a                   X                 n/a
    Percent of children on 1st removal..............               <95%                   X                 n/a                   X                   X
NCANDS Data--Cross File Checks:
    Child IDs don't match across years..............                <1%                 n/a                   X                 n/a                 n/a
    Child IDs match across years, but dates of birth                >5%                   X                   X                 n/a                 n/a
     and sex do not match...........................
    Some victims with AFCARS IDs should match IDs in                Y/N                   X                 n/a                 n/a                 n/a
     AFCARS files...................................
    Some victims have AFCARS IDs....................                <1%                   X                 n/a                 n/a                 n/a
NCANDS Within file checks:
    Missing age.....................................                >5%                   X                   X                 n/a                 n/a
--------------------------------------------------------------------------------------------------------------------------------------------------------
Note. If a state exceeds these specified limits, we will not calculate performance for the state on the indicator.

[FR Doc. 2014-24204 Filed 10-9-14; 8:45 am]
BILLING CODE 4184-25-P