[Federal Register Volume 83, Number 228 (Tuesday, November 27, 2018)]
[Notices]
[Pages 60874-60876]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2018-25835]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Administration for Children and Families


Statewide Data Indicators for Child and Family Services Reviews: 
Request for Public Comment

AGENCY: Children's Bureau (CB), Administration for Children and 
Families (ACF), Administration on Children, Youth and Families (ACYF), 
Department of Health and Human Services (HHS).

ACTION: Request for public comment on revised syntax used to calculate 
Statewide Data Indicators for Child and Family Services Reviews 
(CFSRs).

-----------------------------------------------------------------------

SUMMARY: In October 2014, the Administration for Children and Families 
(ACF) published a notice in the Federal Register (79 FR 61241) with the 
final plan to replace the statewide data indicators used to determine a 
state's substantial conformity with titles IV-B and IV-E of the Social 
Security Act through the Child and Family Services Reviews (CFSRs). In 
May 2015, ACF published a notice in the Federal Register (80 FR 27263) 
to correct some of the calculations and language used in the October 
2014 notice. After the May 2015 notice, additional technical errors in 
the syntax and formulation of the statewide data indicators were 
identified. Some of those errors were discovered by states and other 
interested parties that operationalized the indicators for continuous 
quality improvement. Based on the amount of time required to complete a 
comprehensive review of the syntax, make revisions and validate the 
accuracy of the calculations; the Children's Bureau (CB) decided to 
suspend use of the indicators in determinations of substantial 
conformity during the third round of CFSRs. In October 2016, CB 
published CFSR Technical Bulletin #9 to inform states of the decision 
to limit use of the CFSR statewide data indicators and national 
performance to context information for round three of CFSRs. Since 
then, CB has thoroughly reviewed, revised, tested, and obtained 
independent review and validation of the syntax used to perform data 
quality checks and calculate state performance on the statewide data 
indicators. Before CB finalizes the revised syntax, and publishes 
revised performance results for all states and revised national 
standards, we invite state child welfare agencies, partner 
organizations, and the public to review, test, and provide comments on 
the revised syntax used to perform data quality checks and calculate 
observed performance on the statewide data indicators.

DATES: Comments are due by February 25, 2019.

ADDRESSES: You may send comments, identified by Docket Number ACF-2018-
0007, by one of the following methods:

[[Page 60875]]

     Federal eRulemaking Portal: https://www.regulations.gov. 
Follow the instructions for submitting comments.
     Email: [email protected]. Include Docket Number ACF-
2018-0007 in subject line of the message.

FOR FURTHER INFORMATION CONTACT: James Gregory, (202) 260-4668, 
[email protected].

SUPPLEMENTARY INFORMATION: This Federal Register announcement has four 
sections: Background describes the purpose, history, and authority for 
the CFSRs and the use of statewide data indicators; Overview of Review 
and Validation of Syntax to Calculate CFSR 3 Statewide Data Indicator 
Performance describes the internal and independent review and 
validation process and findings; Supporting Documents provides a brief 
description of five resources available to aid interested parties in 
the syntax review process; and Invitation to Comment wherein we solicit 
public comment on the revised syntax that is pending finalization.

Background

    CB implemented the CFSRs in 2001 in response to a mandate in the 
Social Security Amendments of 1994. The reviews are required for CB to 
determine if such programs are in substantial conformity with title IV-
B and IV-E plan requirements. The review process, as regulated at 45 
CFR 1355.31-37, grew out of extensive consultation with interested 
groups, individuals, and experts in the field of child welfare and 
related areas.
    The CFSRs enable CB to: (1) Ensure conformity with federal child 
welfare requirements; (2) determine what is actually happening to 
children and families as they are engaged in child welfare services; 
and (3) assist states to enhance their capacity to help children and 
families achieve positive outcomes. CB conducts the reviews in 
partnership with state child welfare agency staff and other partners 
and stakeholders involved in the provision of child welfare services. 
We have structured the reviews to help states identify strengths as 
well as areas needing improvement within their agencies and programs.
    We use the CFSR to assess state performance on seven outcomes and 
seven systemic factors. The seven outcomes focus on key items measuring 
safety, permanency, and well-being. The seven systemic factors focus on 
key state plan requirements of titles IV-B and IV-E that provide a 
foundation for child outcomes. If we determine a state has not achieved 
substantial conformity in one or more of the areas assessed in the 
review, the state is required to develop and implement a program 
improvement plan addressing the areas of nonconformity within 2 years. 
CB supports the states with technical assistance and monitors 
implementation of their program improvement plans. If the state is 
unable to complete its program improvement plan successfully, a portion 
of the state's federal title IV-B and IV-E funds is withheld.
    In the April 23, 2014, Federal Register (79 FR 22604), we provided 
a detailed review of the consultation with the field and information 
considered in developing the third round of CFSRs, and proposed a set 
of statewide data indicators for public comment. We considered all 
public comments and issued a final plan in the October 10, 2014, 
Federal Register (79 FR 61241) that included national standards for 
state performance on statewide data indicators and criteria CB would 
use to determine whether a state is in substantial conformity with 
certain child welfare outcomes. Simultaneously, CB released CFSR 
Technical Bulletin #8 which provided more details on calculation 
methods, and a workbook that showed individual state performance on the 
indicators and preliminary findings of whether the state met national 
standards. In responding to state and other stakeholder questions after 
the release of those publications, we discovered errors in our 
descriptions and calculations. As a result, on May 13, 2015, we 
released a notice in the Federal Register (80 FR 27263) to correct some 
calculations and language used in the October 2014 final notice. We 
also amended Technical Bulletin #8 that provided technical detail on 
calculations, how CB determined whether a state met national standards, 
and how program improvement goals relative to the statewide data 
indicators would be established for states not meeting national 
standards.
    After the May 2015 notice, we discovered additional technical 
errors in the syntax and formulation of the statewide data indicators. 
Some of the changes were identified by states and other interested 
parties that operationalized the indicators for continuous quality 
improvement. CB determined additional time was necessary to review, 
correct, and thoroughly test revised syntax used to calculate national 
and state performance on the data indicators. Recognizing challenges 
associated with the delay and implementation of revised indicators, 
especially for states already involved in the CFSR and program 
improvement planning process, CB decided to suspend use of the 
indicators in determinations of substantial conformity and as a basis 
for potentially imposing financial penalties during the third round of 
CFSRs. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and 
(5) to add, amend, or suspend any of the statewide data indicators and 
to adjust the national standards when appropriate. CB issued Technical 
Bulletin #9 on October 11, 2016, to notify states of the change in the 
use of the statewide data indicators. Technical Bulletin #9, which 
rescinded Technical Bulletin #8A, provided guidance on how state 
performance on the CFSR 3 statewide data indicators would be used as 
context, revised related program improvement plan requirements, and re-
issued the description of program improvement measurement methods.
    We recently completed the comprehensive syntax review, revision, 
and validation process for calculating statewide data indicator 
performance information. The revisions do not change the statewide data 
indicator measures as defined in prior Federal Register notices. 
Revisions are limited to operationalizing the measures and calculation 
methods contained in the statistical syntax used to generate 
performance information.
    Following the commencement of this Federal Register Announcement, 
we will review public comments to determine readiness to finalize the 
revised syntax or the potential need for additional revisions. We will 
issue a Federal Register to provide notice when the revised syntax is 
finalized. The notification will address applicable revisions and 
clarifications necessary to the Federal Register document from May 
2015, and include a summary of, and our responses to, public comments 
we received. We will also publish revised performance results for all 
states, revised national standards, and the final revised syntax and 
supporting files to calculate data quality, observed performance, and 
risk-standardized performance for the CFSR 3 statewide data indicators.

Overview of Review and Validation of Syntax To Calculate CFSR 3 
Statewide Data Indicator Performance

    After the discovery of technical errors in the syntax and 
formulation of the statewide data indicators, federal staff completed a 
comprehensive internal review to identify and make syntax revisions. 
The internal in-depth line-level review of all syntax related to 
computing performance on the statewide indicators was completed in May 
2017. Staff verified each step of

[[Page 60876]]

indicator computation to determine whether the syntax was functioning 
accurately and as designed. Additional data quality tests were 
performed to confirm the integrity of the population of child records 
eligible for analysis, and syntax functioned correctly across a variety 
of scenarios, including outlier cases with high error risk. The 
comprehensive review resulted in the identification of additional 
syntax revisions, changes to how measurement methods are 
operationalized, and new methods to screen and address data quality 
problems in AFCARS and NCANDS submissions.
    In addition to our internal review, we contracted for an 
independent review and validation of the syntax used to complete data 
quality checks and calculate observed and risk-standardized performance 
on the seven statewide data indicators. The independent reviewers used 
the following multi-step process to systematically review and verify 
all of the syntax:

1. Reviewed syntax for completeness, functionality on systems external 
to the Children's Bureau
2. Performed reliability checks against performance indicators and 
state data profiles
3. Performed line-level validity checks of SPSS and STATA syntax files 
for accurate task execution, data quality verification
    a. Reviewed the syntax and assessed its function/purpose
    b. Selected and applied the appropriate validity check(s)
    c. Ran all syntax and inspected dataset(s) and output
    d. Debriefed on similar code segments across syntax files

    The independent reviewers determined the syntax can validly and 
reliably develop source data files, perform data quality checks, and 
calculate state observed performance and risk-standardized performance 
on the seven statewide data indicators according to the definitions set 
forth in the Federal Register Notices and associated CB guidance 
documents. The reviewers also verified there are no errors in the 
revised syntax for observed performance and data quality checks, or the 
original syntax for risk-standardized performance that would cause 
inaccuracies in the calculations. The independent review also helped 
demonstrate the feasibility for states and partner agencies to 
replicate the syntax with several additional steps to calculate data 
quality and observed performance on their own. Additional information 
about the independent review and verification process and associated 
findings are detailed in the 2018 report, Review and Validation of the 
Analytic Syntax Used to Produce CFSR Statewide Data Indicators (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-validation).

Supporting Documents

    The documents identified below are provided to assist states and 
other interested parties with the independent syntax review process:
    A. List of CFSR Round 3 Statewide Data Indicator Syntax Revisions: 
Provides a summary of errors, corrections, and changes made to the 
syntax used to calculate performance on the statewide data indicator(s) 
and data quality checks. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax).
    B. CFSR Round 3 Statewide Data Indicator Data Dictionary: Provides 
a description of each statewide data indicator and data quality check, 
including the numerators, denominators, risk adjustments, exclusions, 
and corresponding data notes. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-data-dictionary).
    C. Statewide Data Indicators--PDF Version of Syntax: Provides the 
syntax used to perform data quality checks, calculate observed 
performance, and calculate risk-standardized performance on the seven 
statewide data indicators. This document may help individuals develop a 
better understanding of the statistical calculations or provide 
assistance for individuals who do not have SPSS and/or STATA 
statistical software to run the syntax files. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-pdf).
    D. CFSR Round 3 Statewide Data Indicator Syntax Zip File: Provides 
syntax and supporting files in SPSS format that are used to calculate 
data quality and observed performance. Note: Syntax to calculate risk-
standardized performance (RSP) for each state uses a national, risk-
adjusted model that requires child-level data from all states (i.e., 
national datasets) and thus cannot be replicated by states and 
interested parties. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-zip).
    E. Instructions to Run Data Quality (DQ) Checks and Observed 
Performance Syntax for Statewide Data Indicators: Provides an overview 
and brief description of the folders, files, and syntax included in the 
CFSR round 3 zip file; considerations and steps for running the syntax; 
and software requirements. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-dq-instructions).

Invitation To Comment

    We are inviting state child welfare agencies, partner 
organizations, and the public to review, test, and provide comments on 
the revised syntax to perform data quality checks and calculate 
observed performance on the statewide data indicators. The purpose of 
this invitation is to provide an opportunity for additional independent 
review prior to finalizing the revised syntax. Interested parties may 
choose to review a written version of the statistical syntax and 
supporting documents, and/or follow the instructions to replicate the 
syntax files to generate CFSR data indicator performance information 
using state-generated AFCARS and NCANDS files.
    We are interested in receiving comments on the revised syntax and 
related considerations and recommendations to strengthen our 
statistical calculations for generating CFSR statewide data indicator 
performance information. We encourage states and interested parties to 
share the outcome of their review and/or results from running the 
revised syntax files. Please be specific in identifying the 
corresponding statewide data indicator or data quality check being 
referenced in the comment; the syntax revision, question, or 
recommendation; and the rationale for considerations or 
recommendation(s), as applicable.
    Comments containing suggested revisions should be limited to 
operationalizing the measures and calculation methods contained in the 
statistical syntax used to generate performance data. We are not making 
changes or requesting comments at this time pertaining to the statewide 
data indicator measures as defined in prior Federal Register notices 
and the Data Dictionary available with this invitation.

Lynn A. Johnson,
Assistant Secretary for Children and Families.
[FR Doc. 2018-25835 Filed 11-26-18; 8:45 am]
 BILLING CODE 4184-25-P