[Federal Register Volume 83, Number 228 (Tuesday, November 27, 2018)]
[Notices]
[Pages 60874-60876]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2018-25835]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Statewide Data Indicators for Child and Family Services Reviews:
Request for Public Comment
AGENCY: Children's Bureau (CB), Administration for Children and
Families (ACF), Administration on Children, Youth and Families (ACYF),
Department of Health and Human Services (HHS).
ACTION: Request for public comment on revised syntax used to calculate
Statewide Data Indicators for Child and Family Services Reviews
(CFSRs).
-----------------------------------------------------------------------
SUMMARY: In October 2014, the Administration for Children and Families
(ACF) published a notice in the Federal Register (79 FR 61241) with the
final plan to replace the statewide data indicators used to determine a
state's substantial conformity with titles IV-B and IV-E of the Social
Security Act through the Child and Family Services Reviews (CFSRs). In
May 2015, ACF published a notice in the Federal Register (80 FR 27263)
to correct some of the calculations and language used in the October
2014 notice. After the May 2015 notice, additional technical errors in
the syntax and formulation of the statewide data indicators were
identified. Some of those errors were discovered by states and other
interested parties that operationalized the indicators for continuous
quality improvement. Based on the amount of time required to complete a
comprehensive review of the syntax, make revisions and validate the
accuracy of the calculations; the Children's Bureau (CB) decided to
suspend use of the indicators in determinations of substantial
conformity during the third round of CFSRs. In October 2016, CB
published CFSR Technical Bulletin #9 to inform states of the decision
to limit use of the CFSR statewide data indicators and national
performance to context information for round three of CFSRs. Since
then, CB has thoroughly reviewed, revised, tested, and obtained
independent review and validation of the syntax used to perform data
quality checks and calculate state performance on the statewide data
indicators. Before CB finalizes the revised syntax, and publishes
revised performance results for all states and revised national
standards, we invite state child welfare agencies, partner
organizations, and the public to review, test, and provide comments on
the revised syntax used to perform data quality checks and calculate
observed performance on the statewide data indicators.
DATES: Comments are due by February 25, 2019.
ADDRESSES: You may send comments, identified by Docket Number ACF-2018-
0007, by one of the following methods:
[[Page 60875]]
Federal eRulemaking Portal: https://www.regulations.gov.
Follow the instructions for submitting comments.
Email: [email protected]. Include Docket Number ACF-
2018-0007 in subject line of the message.
FOR FURTHER INFORMATION CONTACT: James Gregory, (202) 260-4668,
[email protected].
SUPPLEMENTARY INFORMATION: This Federal Register announcement has four
sections: Background describes the purpose, history, and authority for
the CFSRs and the use of statewide data indicators; Overview of Review
and Validation of Syntax to Calculate CFSR 3 Statewide Data Indicator
Performance describes the internal and independent review and
validation process and findings; Supporting Documents provides a brief
description of five resources available to aid interested parties in
the syntax review process; and Invitation to Comment wherein we solicit
public comment on the revised syntax that is pending finalization.
Background
CB implemented the CFSRs in 2001 in response to a mandate in the
Social Security Amendments of 1994. The reviews are required for CB to
determine if such programs are in substantial conformity with title IV-
B and IV-E plan requirements. The review process, as regulated at 45
CFR 1355.31-37, grew out of extensive consultation with interested
groups, individuals, and experts in the field of child welfare and
related areas.
The CFSRs enable CB to: (1) Ensure conformity with federal child
welfare requirements; (2) determine what is actually happening to
children and families as they are engaged in child welfare services;
and (3) assist states to enhance their capacity to help children and
families achieve positive outcomes. CB conducts the reviews in
partnership with state child welfare agency staff and other partners
and stakeholders involved in the provision of child welfare services.
We have structured the reviews to help states identify strengths as
well as areas needing improvement within their agencies and programs.
We use the CFSR to assess state performance on seven outcomes and
seven systemic factors. The seven outcomes focus on key items measuring
safety, permanency, and well-being. The seven systemic factors focus on
key state plan requirements of titles IV-B and IV-E that provide a
foundation for child outcomes. If we determine a state has not achieved
substantial conformity in one or more of the areas assessed in the
review, the state is required to develop and implement a program
improvement plan addressing the areas of nonconformity within 2 years.
CB supports the states with technical assistance and monitors
implementation of their program improvement plans. If the state is
unable to complete its program improvement plan successfully, a portion
of the state's federal title IV-B and IV-E funds is withheld.
In the April 23, 2014, Federal Register (79 FR 22604), we provided
a detailed review of the consultation with the field and information
considered in developing the third round of CFSRs, and proposed a set
of statewide data indicators for public comment. We considered all
public comments and issued a final plan in the October 10, 2014,
Federal Register (79 FR 61241) that included national standards for
state performance on statewide data indicators and criteria CB would
use to determine whether a state is in substantial conformity with
certain child welfare outcomes. Simultaneously, CB released CFSR
Technical Bulletin #8 which provided more details on calculation
methods, and a workbook that showed individual state performance on the
indicators and preliminary findings of whether the state met national
standards. In responding to state and other stakeholder questions after
the release of those publications, we discovered errors in our
descriptions and calculations. As a result, on May 13, 2015, we
released a notice in the Federal Register (80 FR 27263) to correct some
calculations and language used in the October 2014 final notice. We
also amended Technical Bulletin #8 that provided technical detail on
calculations, how CB determined whether a state met national standards,
and how program improvement goals relative to the statewide data
indicators would be established for states not meeting national
standards.
After the May 2015 notice, we discovered additional technical
errors in the syntax and formulation of the statewide data indicators.
Some of the changes were identified by states and other interested
parties that operationalized the indicators for continuous quality
improvement. CB determined additional time was necessary to review,
correct, and thoroughly test revised syntax used to calculate national
and state performance on the data indicators. Recognizing challenges
associated with the delay and implementation of revised indicators,
especially for states already involved in the CFSR and program
improvement planning process, CB decided to suspend use of the
indicators in determinations of substantial conformity and as a basis
for potentially imposing financial penalties during the third round of
CFSRs. We are authorized by the regulations at 45 CFR 1355.34(b)(4) and
(5) to add, amend, or suspend any of the statewide data indicators and
to adjust the national standards when appropriate. CB issued Technical
Bulletin #9 on October 11, 2016, to notify states of the change in the
use of the statewide data indicators. Technical Bulletin #9, which
rescinded Technical Bulletin #8A, provided guidance on how state
performance on the CFSR 3 statewide data indicators would be used as
context, revised related program improvement plan requirements, and re-
issued the description of program improvement measurement methods.
We recently completed the comprehensive syntax review, revision,
and validation process for calculating statewide data indicator
performance information. The revisions do not change the statewide data
indicator measures as defined in prior Federal Register notices.
Revisions are limited to operationalizing the measures and calculation
methods contained in the statistical syntax used to generate
performance information.
Following the commencement of this Federal Register Announcement,
we will review public comments to determine readiness to finalize the
revised syntax or the potential need for additional revisions. We will
issue a Federal Register to provide notice when the revised syntax is
finalized. The notification will address applicable revisions and
clarifications necessary to the Federal Register document from May
2015, and include a summary of, and our responses to, public comments
we received. We will also publish revised performance results for all
states, revised national standards, and the final revised syntax and
supporting files to calculate data quality, observed performance, and
risk-standardized performance for the CFSR 3 statewide data indicators.
Overview of Review and Validation of Syntax To Calculate CFSR 3
Statewide Data Indicator Performance
After the discovery of technical errors in the syntax and
formulation of the statewide data indicators, federal staff completed a
comprehensive internal review to identify and make syntax revisions.
The internal in-depth line-level review of all syntax related to
computing performance on the statewide indicators was completed in May
2017. Staff verified each step of
[[Page 60876]]
indicator computation to determine whether the syntax was functioning
accurately and as designed. Additional data quality tests were
performed to confirm the integrity of the population of child records
eligible for analysis, and syntax functioned correctly across a variety
of scenarios, including outlier cases with high error risk. The
comprehensive review resulted in the identification of additional
syntax revisions, changes to how measurement methods are
operationalized, and new methods to screen and address data quality
problems in AFCARS and NCANDS submissions.
In addition to our internal review, we contracted for an
independent review and validation of the syntax used to complete data
quality checks and calculate observed and risk-standardized performance
on the seven statewide data indicators. The independent reviewers used
the following multi-step process to systematically review and verify
all of the syntax:
1. Reviewed syntax for completeness, functionality on systems external
to the Children's Bureau
2. Performed reliability checks against performance indicators and
state data profiles
3. Performed line-level validity checks of SPSS and STATA syntax files
for accurate task execution, data quality verification
a. Reviewed the syntax and assessed its function/purpose
b. Selected and applied the appropriate validity check(s)
c. Ran all syntax and inspected dataset(s) and output
d. Debriefed on similar code segments across syntax files
The independent reviewers determined the syntax can validly and
reliably develop source data files, perform data quality checks, and
calculate state observed performance and risk-standardized performance
on the seven statewide data indicators according to the definitions set
forth in the Federal Register Notices and associated CB guidance
documents. The reviewers also verified there are no errors in the
revised syntax for observed performance and data quality checks, or the
original syntax for risk-standardized performance that would cause
inaccuracies in the calculations. The independent review also helped
demonstrate the feasibility for states and partner agencies to
replicate the syntax with several additional steps to calculate data
quality and observed performance on their own. Additional information
about the independent review and verification process and associated
findings are detailed in the 2018 report, Review and Validation of the
Analytic Syntax Used to Produce CFSR Statewide Data Indicators (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-validation).
Supporting Documents
The documents identified below are provided to assist states and
other interested parties with the independent syntax review process:
A. List of CFSR Round 3 Statewide Data Indicator Syntax Revisions:
Provides a summary of errors, corrections, and changes made to the
syntax used to calculate performance on the statewide data indicator(s)
and data quality checks. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax).
B. CFSR Round 3 Statewide Data Indicator Data Dictionary: Provides
a description of each statewide data indicator and data quality check,
including the numerators, denominators, risk adjustments, exclusions,
and corresponding data notes. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-data-dictionary).
C. Statewide Data Indicators--PDF Version of Syntax: Provides the
syntax used to perform data quality checks, calculate observed
performance, and calculate risk-standardized performance on the seven
statewide data indicators. This document may help individuals develop a
better understanding of the statistical calculations or provide
assistance for individuals who do not have SPSS and/or STATA
statistical software to run the syntax files. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-pdf).
D. CFSR Round 3 Statewide Data Indicator Syntax Zip File: Provides
syntax and supporting files in SPSS format that are used to calculate
data quality and observed performance. Note: Syntax to calculate risk-
standardized performance (RSP) for each state uses a national, risk-
adjusted model that requires child-level data from all states (i.e.,
national datasets) and thus cannot be replicated by states and
interested parties. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-syntax-zip).
E. Instructions to Run Data Quality (DQ) Checks and Observed
Performance Syntax for Statewide Data Indicators: Provides an overview
and brief description of the folders, files, and syntax included in the
CFSR round 3 zip file; considerations and steps for running the syntax;
and software requirements. (https://www.acf.hhs.gov/cb/resource/cfsr-round3-sdi-dq-instructions).
Invitation To Comment
We are inviting state child welfare agencies, partner
organizations, and the public to review, test, and provide comments on
the revised syntax to perform data quality checks and calculate
observed performance on the statewide data indicators. The purpose of
this invitation is to provide an opportunity for additional independent
review prior to finalizing the revised syntax. Interested parties may
choose to review a written version of the statistical syntax and
supporting documents, and/or follow the instructions to replicate the
syntax files to generate CFSR data indicator performance information
using state-generated AFCARS and NCANDS files.
We are interested in receiving comments on the revised syntax and
related considerations and recommendations to strengthen our
statistical calculations for generating CFSR statewide data indicator
performance information. We encourage states and interested parties to
share the outcome of their review and/or results from running the
revised syntax files. Please be specific in identifying the
corresponding statewide data indicator or data quality check being
referenced in the comment; the syntax revision, question, or
recommendation; and the rationale for considerations or
recommendation(s), as applicable.
Comments containing suggested revisions should be limited to
operationalizing the measures and calculation methods contained in the
statistical syntax used to generate performance data. We are not making
changes or requesting comments at this time pertaining to the statewide
data indicator measures as defined in prior Federal Register notices
and the Data Dictionary available with this invitation.
Lynn A. Johnson,
Assistant Secretary for Children and Families.
[FR Doc. 2018-25835 Filed 11-26-18; 8:45 am]
BILLING CODE 4184-25-P