[Federal Register Volume 70, Number 165 (Friday, August 26, 2005)]
[Notices]
[Pages 50381-50390]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 05-17034]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Substance Abuse and Mental Health Services Administration (SAMHSA)
Notice: Request for Comments; National Registry of Evidence-Based
Programs and Practices (NREPP)
Authority: Sec. 501, Pub. L. 106-310
SUMMARY: The Substance Abuse and Mental Health Services Administration
(SAMHSA) is committed to preventing the onset and reducing the
progression of mental illness, substance abuse and substance related
problems among all individuals, including youth. As part of this
effort, SAMHSA is expanding and refining the agency's National Registry
of Evidence-based Programs and Practices (NREPP) so that the system
serves as a leading national resource for contemporary and reliable
information on the scientific basis and practicality of interventions
to prevent and/or treat mental illness and substance use and abuse.
NREPP represents a major agency activity within SAMHSA's Science to
Service initiative. The initiative seeks to accelerate the translation
of research into practice by promoting the implementation of effective,
evidence-based interventions for preventing and/or treating mental
disorders and substance use and abuse. Of equal measure, the initiative
emphasizes the essential role of the services community in providing
input and feedback to influence and better frame the research questions
and activities pursued by researchers in these areas.
Through SAMHSA's Science to Service initiative, the agency
ultimately seeks to develop a range of tools that will facilitate
evidence-based decision-making in substance abuse prevention, mental
health promotion, and the treatment of mental and substance use
disorders. In addition to NREPP, SAMHSA is developing an informational
guide of web-based
[[Page 50382]]
resources on evidence-based interventions that will be available in
2006. SAMHSA also is exploring the feasibility of supporting a
searchable web database of evidence-based information (e.g., systematic
reviews, meta-analyses, clinical guidelines) for mental health and
substance abuse prevention and treatment providers. Such a system could
reduce the lag time between the initial development and broader
application of research knowledge by serving as a real-time resource to
providers for ``keeping current'' in ways that will enhance their
delivery of high quality, effective services. In combination, these
three tools--NREPP, guide to web-based resources, and database of
evidence-based information--would provide valuable information that can
be used in a variety of ways by a range of interested stakeholders.
With regard to NREPP, during the past two years, SAMHSA convened a
series of scientific/stakeholder panels to inform the agency's
expansion of the system to include interventions in all substance abuse
and mental health treatment and prevention domains. These panels
thoroughly assessed the existing NREPP review process and review
criteria and provided comments and suggestions for refining and
enhancing NREPP. As part of this expansion effort, SAMHSA also engaged
a contractor to assess the NREPP process and review criteria, including
how the system and criteria compare to other, similar evidence review
and rating systems in the behavioral and social sciences. The
cumulative results of these activities have guided efforts to refine
the NREPP review process and review criteria, as well as inform the
agency's plans for how such a system may be used to promote greater
adoption of evidence-based interventions within typical community-based
settings.
This Federal Register Notice (FRN) provides an opportunity for
interested parties to become familiar with and comment on SAMHSA's
plans for expansion and use of NREPP.
DATES: Submit comments on or before October 25, 2005.
ADDRESSES: Address all comments concerning this notice to: SAMHSA c/o
NREPP Notice, 1 Choke Cherry Road, Rockville, MD 20857. See
SUPPLEMENTARY INFORMATION for information about electronic filing.
FOR FURTHER INFORMATION CONTACT: Kevin D. Hennessy, PhD, Science to
Service Coordinator/SAMHSA, 1 Choke Cherry Road, Room 8-1017,
Rockville, Maryland 20857. Dr. Hennessy may be reached at (240) 276-
2234.
SUPPLEMENTARY INFORMATION:
Electronic Access and Filing Addresses
You may submit comments by sending electronic mail (e-mail) to
[email protected].
Dated: August 18, 2005.
Charles G. Curie,
Administrator.
Overview
Increasingly, individuals and organizations responsible for
purchasing, providing and receiving services to prevent substance abuse
and/or treat mental and substance use disorders are considering the
extent to which these services are ``evidence-based''--that there
exists some degree of documented scientific support for the outcomes
obtained by these services. As the Federal agency responsible for
promoting the delivery of substance abuse and mental health services,
SAMHSA is particularly interested in supporting and advancing
activities that encourage greater adoption of effective, evidence-based
interventions to prevent and/or treat mental and substance use
disorders. With this in mind, SAMHSA proposes to refine and expand its
National Registry of Evidence-based Programs and Practices (NREPP).
SAMHSA believes that the growth and evolution of NREPP can serve as an
important mechanism for promoting greater adoption of evidence-based
substance abuse and mental health services,--one that can do so in
conjunction with an ever-growing array of scientific knowledge,
clinical expertise and judgment, and patient/recipient values and
perspectives. By clearly identifying and assessing the scientific basis
and disseminability of a range of behavioral interventions, NREPP is
likely to prove an important resource to both individuals and systems
seeking information on the effectiveness of various services to prevent
and/or treat mental and substance use disorders.
Background and Need
As SAMHSA promotes the identification and greater use of effective,
evidence-based interventions for individual-, population-, policy-, and
system-level changes, the agency seeks to build upon the strong
foundation provided by the precursor to the National Registry of
Evidence-based Programs and Practices--namely, the National Registry of
Effective Prevention Programs. The previous system provides an
important building block in the agency's efforts to develop a SAMHSA-
wide registry.
The National Registry of Effective Prevention Programs developed in
SAMHSA's Center for Substance Abuse Prevention (CSAP) beginning in 1997
as a way to help professionals in the field become better consumers of
prevention programs. Between 1997 and 2004, NREPP reviewed and rated
more than 1,100 prevention programs, with more than 150 obtaining
designation as a Model, Effective, or Promising Program.
Information on all current NREPP programs is available through the
Model Programs Web site at http://www.modelprograms.samhsa.gov.
Additional details about the review process, review criteria, and
rating system for the National Registry of Effective Prevention
Programs are available in the SAMHSA publication ``Science-Based
Prevention programs and Principles 2002,'' which can be downloaded from
SAMHSA's Model Programs Web site http://www.modelprograms.samhsa.gov)
by clicking on ``Publications'' on the tool bar on the left side of the
page; or by requesting the publication through SAMHSA's National
Clearinghouse for Alcohol and Drug Information (NCADI) at 1-800-729-
6686 (or by visiting the NCADI Web site at http://www.health.org).
As SAMHSA expands the NREPP system, one area of potential
improvement is in the efficient screening and triage of applications.
Given the historical applications trends among substance abuse
prevention programs, combined with the increased demands on the system
through expansion to other SAMHSA domains, it is essential that the
agency develop a transparent and scientifically defensible process for
screening and triaging applications.
Moreover, as SAMHSA engaged NREPP scientific/stakeholder panels
over the past 2 years, concerns about the existing review process and
review criteria emerged. In particular, a range of scientific experts
voiced concerns regarding specific review criteria and other elements
of the review process.
In addition, systematic efforts to examine and compare the current
NREPP review criteria with other evidence-grading systems in the social
and behavioral sciences has revealed both areas of relative strength
and relative weakness. At a minimum, this comparison has affirmed for
SAMHSA the importance and value of reexamining and refining the NREPP
review process and review criteria in ways that reflect to the public
SAMHSA's commitment to identifying
[[Page 50383]]
and promoting interventions that have shown to be effective through
prevailing scientific standards. One important element of this process
is providing support for the re-review of existing NREPP programs
against these prevailing scientific standards (see below), while
another component is identifying both SAMHSA and other mechanisms and
resources for supporting efforts to evaluate and document the evidence-
base of innovative interventions in ways that will maximize their
opportunity for entry into NREPP.
Further, SAMHSA's experience with NREPP to date suggests that the
system is limited in its ability to identify and rate interventions
designed to promote population-, policy-, and system-level outcomes,
such as those promoted by community prevention coalitions. SAMHSA's
plans for NREPP include an expansion of the system in this area. As
part of this expansion, SAMHSA proposes a second set of review criteria
for these interventions, with the recognition that some interventions
may be designed to affect a community over time, and that the
prevailing scientific standards for assessing the effectiveness of
these interventions may indeed be different than those for
interventions seeking to change individual-level outcomes. Finally,
input into the NREPP process to date suggests the need for SAMHSA to
provide greater policy guidance on how best to use the system to
appropriately select specific interventions, as well as contextual
guidance on how NREPP might be used in conjunction with other important
information--such as clinical expertise, patient values, and
administrative and policy perspectives and data--in making decisions
regarding the financing and delivery of substance abuse and mental
health services.
Proposal
After extensive consultation with both scientific experts and a
range of stakeholders. SAMHSA is seeking your comments on a proposal to
advance a voluntary rating and classification system for mental health
and substance abuse prevention and treatment interventions--a system
designed to categorize and disseminate information about programs and
practices that meet established evidentiary criteria. This proposal
presents in detail the new NREPP system, including refinements to the
review process and review criteria for programs and practices in
substance abuse and mental health, as well as an expansion of the
system to include successful community coalition efforts to achieve
population-, policy-, or system-level outcomes. The proposal also
describes SAMHSA's plans for a new Web site that will highlight the
scientific evidence-base of interventions, as well as provide a range
of practical information needed by those who are considering
implementation of programs or practices on the Registry.
SAMHSA further anticipates that additional revisions and
refinements to the NREPP system may be needed on a periodic basis, and
proposes the formation of an external advisory panel to regularly
assist the agency in assessing proposed suggestions for improvements to
the system (see Question 10 below).
Initial Input From the Field
Upon determining that SAMHSA would expand NREPP to include
interventions in all agency domains, three expert panel meetings of
both scientist and nonscientist stakeholders were convened to provide
feedback on the current review system and review criteria, as well as
solicit suggestions about redesigning the system to promote the goals
noted above. Each meeting was conducted over a 2-day period, and
included invited participants representing a range of relevant
organizations, expertise, and perspectives. All meetings took place in
Washington, DC, in 2003, with mental health experts meeting in April,
substance abuse prevention and mental health promotion experts meeting
in September, and substance abuse treatment experts meeting in
December. Transcripts of these meetings are available on-line at the
NREPP Web page accessible through the ``Quick Picks'' section on
SAMHSA's Home page (http://www.samhsa.gov).
SAMHSA also convened a meeting in May 2005 to solicit
recommendations for integrating evidence-based findings from community
coalitions into NREPP. The 2-day meeting brought together prominent
researchers and practitioners who reaffirmed the importance of
including prevention coalitions within NREPP, and offered suggestions
as to the types of outcomes and evidence criteria appropriate to the
assessment of community coalitions. A summary of this meeting is
available on-line at the NREPP web page accessible through the ``Quick
Picks'' section on SAMHSA's Home page (http://www.samhsa.gov).
Review Process for Determining Individual-Level Outcome Ratings for
Interventions
A primary goal of the Registry is to provide the public with
reliable information about the evidence quality and strength of
scientific support for specific interventions. The strength of
scientific support includes: the quality of evaluation design (e.g.,
experimental or quasi-experimental designs); fidelity to predetermined
intervention components; confidence in the link between intervention
components and specific outcome(s) achieved; freedom from internal and
external sources of bias and error; and the statistical significance
and practical magnitude (e.g., effect size) of outcomes achieved.
An additional goal is to provide key information about the
transferability of these programs and practices to real-world
prevention and treatment settings. NREPP utility descriptors provide
information about the appropriate settings and populations for
implementation of specific interventions, the availability of training
and dissemination materials, and their practicality and costs.
This section describes the NREPP review process, including the
evidence rating criteria and utility descriptors that will form the
basis for Web-based information about programs and practices.
Based on important feedback from scientists and practitioners in
the prevention and treatment fields, the NREPP review process has been
enhanced in several important respects:
--Programs and practices will be rated on the strength of evidence for
specific outcomes achieved, rather than on global assessments of the
effectiveness of intervention(s). In addition, indicators of strength
of association or magnitude of outcome effects, such as effect size
statistics, will be utilized in NREPP to complement traditional,
statistical significance (null-hypothesis) testing.
--There will be multiple, outcome-specific ratings of evidence quality
strength. All programs and/or practices listed on the Registry will be
considered ``Effective'' for demonstrating specific outcomes having
varying levels of evidence quality and confidence resulting from
independent (or applicant) replication(s).
--Evidence rating criteria have been refined and now emphasize
intervention impacts, evaluation design and fidelity, quality of
comparison conditions, and replications.
The section below is an overview of the NREPP process for obtaining
expert reviewers' ratings of the evidence quality for outcome-specific
program and practice interventions. The process includes an internal
screening and triage process conducted by qualified
[[Page 50384]]
NREPP contractor staff serving as review coordinators, as well as an
independent, external scientific review process conducted by qualified
and trained external scientists--working independently of one another--
to assess the evidence quality of candidate interventions.
I. Submitted Materials
Applicants will submit a variety of documents that allow a panel of
expert reviewers to rate objectively the evidence for an intervention's
impact on substance abuse or mental health outcomes. Materials
submitted may include:
Descriptive program summaries, including the specific
outcomes targeted;
Research reports and published and unpublished articles to
provide scientific evidence (experimental or quasi-experimental
studies) about the effectiveness of the intervention for improving
specific outcomes;
Documents to verify that participants were assured privacy
and freedom from negative consequences regardless of participating
(used to assess participant response bias, not human subjects
protections per se); and
Documents to provide evidence that outcomes and analytic
approaches were developed through a theory-driven or hypothesis-based
approach.
These materials will provide SAMHSA and potential reviewers with
objective evidence to support conclusions about the validity and impact
of the program or practice. Reviewers must be assured that program
investigators did not capitalize on chance findings or excessive
postintervention data analyses to find effects that were not components
of the intervention design or theory.
II. Internal Review and Triage
Upon receipt, each set of materials will be assigned to an NREPP
review coordinator (contractor staff), who will inventory and document
the contents of the submission. The review coordinator will contact the
applicant by phone and/or e-mail confirming receipt and notifying the
applicant if additional application materials are required.
When all materials for a program have been received by the review
coordinator, an internal review is conducted to eliminate those
programs or practices that do not meet NREPP minimum evidence-based
standards. These minimum standards are (1) An intervention that is
consistent with SAMHSA's matrix of program priority areas; (2) one or
more evaluations of the intervention using an experimental or quasi-
experimental design; and (3) statistically significant intervention
outcome(s) related to either the prevention or treatment of mental or
substance use disorders. Non-reviewed programs will receive written
notification of this decision, including problematic or missing
components of their application, and may be considered for re-review at
a later date.
SAMHSA will maintain oversight over the entire NREPP application
and selection process. Moreover, SAMHSA's Administrator and Center
Directors may establish specific program and practice areas for
priority review.
III. Ratings by Reviewers
For all NREPP applications determined appropriate for review, three
(3) independent, external reviewers will evaluate and rate the
intervention. Reviewers are doctoral-level researchers and
practitioners who have been selected based on their training and
expertise in the fields of mental health promotion or treatment and/or
substance abuse prevention or treatment. Moreover, reviewers will be
thoroughly trained in the NREPP review process, and will be monitored
and provided feedback periodically on their performance. Of note,
interventions targeting individuals or populations with co-occurring
mental health and substance use disorders (or other cross-domain
initiatives) will be assigned to reviewers across these categorical
domains.
To maintain the objectivity and autonomy of the peer review
process, SAMHSA will not disclose to the public or applicants the
identities of individual reviewers assigned to specific reviews. On a
periodic basis, SAMHSA may post listings of reviewer panels within
specific SAMHSA domains as an informational resource to the public.
Reviewers will be selected based on their qualifications and
expertise related to specific interventions and/or SAMHSA priority
areas. In addition to reviewers identified by SAMHSA, NREPP will
consider third-party and self-nominations to become part of the
reviewer pool. All reviewers will provide written assurance, to be
maintained on file with the NREPP contractor, that they do not have a
current or previous conflict of interest (e.g., financial or business
interest) that might impact a fair and impartial review of specific
programs or practices applying for NREPP review.
Reviewers provide independent assessments of the evidence quality
and provide numerical summary scores across the 16 outcome-specific
evidence quality criteria. Each criterion is scored on a 0 to 4 scale.
The 16 evidence quality criteria are presented below.
Individual-Level Outcome Evidence Rating Criteria
1. Theory-Driven Measure Selection
Outcome measures for a study should be selected before data are
collected and should be based on a priori theories of hypotheses.
0 = The applicant selected the measure after data collection for the
apparent purpose of obtaining more favorable results than would be
expected from using the measures originally planned, OR there is no
documentation of selection prior to data analysis.
4 = Documentation shows that the applicant selected the measure prior
to study implementation, OR the measure was selected after study
inception, but before data analysis, and is supported by a peer review
panel or literature related to study theories or hypotheses.
2. Reliability
Outcome measures should have acceptable reliability to be
interpretable. ``Acceptable'' here means reliability at a level that is
conventionally accepted by experts in the field.
0 = No evidence of measure realibility.
1 = Reliability coefficients indicate that some but not all relevant
types of reliability (e.g., test-retest, inter-rater, inter-item) are
acceptable.
3 = All relevant types of realibility have been documented to be at
acceptable levels in studies by the applicant.
4 = All relevant types of reliability have been documented to be
acceptable levels in studies by independent investigators.
3. Validity
Outcome measures should have acceptable validity to be
interpretable. ``Acceptable'' here means validity at a level that is
conventionally accepted by experts in the field.
0 = No evidence of measure validity, or some evidence that the measure
is not valid.
1 = Measure has face validity.
3 = Studies by applicant show that measure has one or more acceptable
forms of criterion-related validity that are correlated with
appropriate, validated measures or objective criteria; OR, as an
objective measure of response, there are procedural checks to confirm
data validity, but they have not been adequately documented.
4 = Studies by independent investigators show that measure has
[[Page 50385]]
one or more acceptable forms of criterion-related validity that are
correlated with appropriate, validated measures or objective criteria;
OR, as an objective measure of response, there are adequately
documented procedural checks that confirm data validity.
4. Intervention Fidelity
The ``experimental'' intervention implemented in a study should
have fidelity to the intervention proposed by the applicant.
Instruments that have tested acceptable psychometric properties (e.g.,
interrator reliability, validity as shown by positive association with
outcomes) provides the highest level of evidence.
0 = There is evidence the intervention implemented was substantially
different from the one proposed.
1 = There is only narrative evidence that the applicant or provider
believes the intervention was implemented with acceptable fidelity.
2 = There is evidence of acceptable fidelity in the form of judgment(s)
by experts, based on limited on-site evaluation and data collection.
3 = There is evidence of acceptable fidelity, based on the systematic
collection of data on factors such as dosage, time spent in training,
adherence to guidelines or a manual, or a fidelity measure with
unspecified or unknown psychometric properties.
4 = There is evidence of acceptable fidelity from a tested fidelity
instrument shown to have reliability and validity.
5. Comparison Fidelity
A study's comparison condition should be implemented with fidelity
to the comparison condition proposed by the applicant. Instruments for
measuring fidelity that have tested acceptable psychometric properties
(e.g., interrater reliability, validity as shown by predicted
association with outcomes) provide the highest level of evidence.
0 = There is evidence that the comparison condition implemented was
substantially different from one proposed.
1 = There is only narrative evidence that the applicant or provider
believes the comparison condition was implemented with fidelity.
2 = Researchers report observational or administrative data that the
comparison condition was implemented with fidelity.
3 = Documentation confirms that comparison group participants did not
receive interventions that were very similar or identical to
intervention participants, AND there is documentation of degree of
participation in any comparison conditions such as lectures or
treatment.
4 = There is evidence from a tested instrument suggesting that the
comparison condition was implemented with fidelity.
6. Nature of Comparison Condition
The quality of evidence for an intervention depends in part on the
nature of the comparison condition(s), including assessments of their
active components and overall effectiveness. Interventions have the
potential to cause more harm than good; therefore, an active comparison
intervention should be shown to be better than no treatment.
0 = There was no comparison condition.
1 = The comparison condition is an active intervention that has not
been proven to better than no treatment.
2 = The comparison condition is no service or wait-list, or an active
intervention shown to be as effective as or better than no treatment.
3 = The comparison condition is an attention control.
4 = The comparison condition was shown to be as safe or safer and more
effective than an attention control.
7. Assurances to Participants
Study participants should always be assured that their responses
will be kept confidential and not affect their care or services. When
these procedures are in place, participants are more likely to disclose
valid data.
0 = There was no effort to encourage and reassure subjects about
privacy and that consent or participation would have no effect on
services.
1 = Data collector was the service provider, AND there were documented
assurances to participants about privacy and that consent or
participation would have no effect on care or services.
2 = Data collector was not the service provider. There were
indications, but no documentation, that participants were assured about
their privacy and that consent or participation would have no effect on
care or services.
4 = Data collector was not the service provider, AND there were
documented assurances to participants about privacy and that consent or
participation would have no effect on care or services; OR, data were
not collected directly from participants.
8. Participant Expectations
Participants can be biased by how an intervention is introduced to
them and by an awareness of their study condition. Information used to
recruit and inform study participants should be carefully crafted to
equalize expectations. Masking treatment conditions during
implementation of the study provides the strongest control for
participant expectancies.
0 = Investigators did not make adequate attempts to mask study
conditions or equalize expectations among participants in the
experimental and comparison conditions, or differential participant
expectations were measured and found to be too great to control for
statistically.
2 = Investigators attempted to mask study conditions or equalize
expectations among participants in the experimental and comparison
conditions. Some participants appeared likely to have known their study
condition assignment (experimental or comparison).
3 = Investigators attempted to mask study conditions or equalize
expectations among participants in the experimental and comparison
conditions. Some participants appeared likely to have known their study
condition assignment (experimental or comparison), but these
differential participant expectations were measured and appropriately
controlled for statistically.
4 = Investigators adequately masked study conditions. Participants did
not appear likely to have known their study condition assignment.
9. Standardized Data Collection
All outcome data should be collected in a standardized manner. Data
collectors trained and monitored for adherence to standardized
protocols provide the highest quality evidence of standardized data
collection.
0 = Applicant did not use standardized data collection protocols.
2 = Data was collected using standardized protocol and trained data
collectors.
3 = Data was collected using standardized protocol and trained data
collectors, with evidence of good initial adherence by data collectors
to the standardized protocol.
4 = Data was collected using standardized protocol and trained data
collectors, with evidence of good initial adherence to data collectors
to the standardized protocol and evidence of data collector retraining
[[Page 50386]]
when necessary to control for rater ``drift.''
10. Data Collector Bias
Data collector bias is most strongly controlled when data
collectors are not aware of the conditions to which study participants
have been assigned. When data collectors are aware of specific study
conditions, their expectations should be controlled for through
training and/or statistical methods.
0 = Data collectors were not masked to participants' conditions, and
nothing was done to control for possible bias, OR collector bias was
measured and found to be too great to control for statistically.
2 = Data collectors were not masked to participants' conditions, but
data collectors received training to reduce possible bias.
3 = Data collectors were not masked to participants' conditions;
possible bias was appropriately controlled for statistically.
4 = Data collectors were masked to participants' conditions.
11. Selection Bias
Concealed random assignment of participants provides the strongest
evidence of control for selection bias. When participants are not
randomly assigned, covariates and confounding variables should be
controlled as indicated by theory and research.
0 = There was no comparison condition, OR participants or providers
selected conditions.
3 = Participants were not assigned randomly, but researchers controlled
for theoretically relevant confounding variables, OR participants were
assigned with non-concealed randomization.
4 = Selection bias was controlled with concealed random assignment.
12. Attrition
Study results can be biased by participant attrition. Statistical
methods as supported by theory and research can be employed to control
for attrition that would bias results, but studies with no attrition
needing adjustment provide the strongest evidence that results are not
biased.
0 = Attrition was taken into account inadequately, OR there was too
much attrition to control for bias.
1 = No significant differences were found between participants lost to
attrition and remaining participants.
2 = Attrition was taken into account by simpler methods that crudely
estimate data for missing observations.
3 = Attrition was taken into account by more sophisticated methods that
model missing data, observations, or participants.
4 = There was no attrition, OR there was no attrition needing
adjustment.
13. Missing Data
Study results can be biased by missing data. Statistical methods as
supported by theory and research can be employed to control for missing
data that would bias results, but studies with no missing data needing
adjustment provide the strongest evidence.
0 = Missing data were an issue and were taken into account
inadequately, OR levels of missing data were too high to control for
bias.
1= Missing data were an issue and were taken into account, but high
quantity makes the control for bias suspect.
2= Missing data were an issue and were taken into account by simpler
methods (mean replacement, last point carried forward) that
simplistically estimate missing data; control for missing data is
plausible.
3= Missing data were an issue and were taken into account by more
sophisticated methods that model missing data; control for missing data
very plausible.
4= Missing data were not an issue.
14. Analysis Meets Data Assumptions
The appropriateness of statistical analyses is a function of the
properties of the data being analyzed and the degree to which meet
statistical assumptions.
0= Analyses were clearly inappropriate to the data collected; severe
violation(s) of assumptions make analysis uninterpretable.
1= Some data were analyzed appropriately, but for other analyses
important violation(s) of assumptions cast doubt on interpretation.
2= There were minor violations of assumptions for most or all analyses,
making interpretation of results arguable.
3= There were minor violations of assumptions for only a few analyses;
results were generally interpretable.
4= There were no violations of assumptions for any analysis.
15. Theory-Driven Selection of Analytic Methods
Analytic methods should be selected for a study based on a priori
theories or hypotheses underlying the intervention. Changes to analytic
methods after initial data analysis (e.g., to ``dredge'' for
significant results) decrease the confidence that can be placed in the
findings.
0= Analysis selected appears inconsistent with the intervention theory
or hypotheses; insufficient rational provided by investigator.
1= Analysis selected appears inconsistent with the intervention theory
or hypotheses, but applicant provides a potentially viable rationale.
3= Analysis is widely accepted by the field as the most consistent with
study theory or hypotheses; no documentation showing methods were
selected prior to data analysis.
4= Analysis is widely accepted by the field as the most consistent with
study theory or hypotheses; documentation shows that methods were
selected prior to data analysis.
16. Anomalous Findings
Findings that contradict the theories and hypotheses underlying an
intervention suggest the possibility of confounding causal variables
and limit the validity of study findings.
0 = There were anomalous findings suggesting alternate explanations for
outcomes reported.
4 = There were no anomalous findings, OR researchers explained
anomalous findings in a way that preserves the validity of results
reported.
Based upon the independent reviewer assessments, review coordinators
will compute average evidence quality ratings for specific outcome
measures (based on the 16 evidence quality criteria), and then ask
reviewers to determine the overall intervention outcome evidence
ratings according to two components: quality of evidence and
intervention replications. Average evidence quality ratings scores
below 2.0 will be considered ``insufficient current evidence'' for the
effectiveness of a given outcome, and will not be included in the
Registry. Evidence quality rating scores of 2.0 to 2.5 will be
considered ``emerging evidence'' for effectiveness, and scores of 2.5
and higher (4.0 is the maximum) will be considered ``strong evidence.''
Specific rating category labels for effective outcomes remain to be
finalized, but might include categories such as: (1) Strong evidence
with independent replication(s); (2) Strong evidence with developer
replication(s); (3) Strong evidence without replication; (4) Emerging
evidence with independent replication(s); (5) Emerging evidence with
developer replication(s); and (6) Emerging evidence without
replication.
IV. Applicant Notification
Applicants will be notified in writing of their final NREPP rating
category(s) by the SAMHSA Administrator or his/
[[Page 50387]]
her designee within 2 weeks following the completion of the review.
This notification will include the summary comments of reviewers as
well as the consensus ratings on each review criteria. Where relevant,
the notification letter will provide applicants with the effective date
of the program status designation.
Applicants will have the opportunity to appeal any review decision
by submitting a written request to the NREPP contractor within 30 days
of notification. Appeals will be resolved through the assignment of two
(2) additional reviewers to conduct focused reviews of the evidence
quality for specific, disputed outcome ratings. Reviews conducted as
part of a formal appeal process will be independent and reviewers will
be unaware of previous ratings and decisions. The numeric evidence
ratings will be averaged across the five (i.e., 3 original; 2 appeal)
reviews for a final determination of intervention outcome rating(s).
V. Utility Descriptors
The NREPP utility descriptors will provide information to program
purchasers and planners, service providers, consumers and the general
public about the transferability of intervention technologies to
different (including non-research-based) settings. These descriptors
complement NREPP's scientific evidence-based program and practice
ratings with information pertaining to the following dimensions:
1. Implementation. What kinds of materials are available to support
implementation and what audiences are they appropriate for? What kinds
of training and training resources are available to support
implementation?
2. Quality Monitoring. What tools, procedures, and documentation
are available to support quality monitoring and quality improvement as
the program or practice is implemented?
3. Unintended or Adverse Events. What procedures, systems and data
have been identified to indicate whether the intervention has ever
resulted in unintended or adverse events?
4. Population Coverage. Were the study samples recruited
representative of the persons identified to receive the intervention in
the theory/framework?
5. Cultural Relevance and Cultural Competence. Were the outcomes
demonstrated to be effective and applicable to specific demographic and
culturally defined groups? If so, are training and other implementation
materials available to promote culturally competent implementation of
the intervention?
6. Staffing. What is aggregate level of staffing (e.g., FTEs)
required to implement the intervention effectively? What are the
individual skills, expertise, and training required of staff to deliver
the intervention?
7. Cost. What are the estimated start-up and annual costs per
person served and unit of service for the intervention at full
operation?
Interventions with outcomes achieving any one of the effective
statuses will be asked to provide descriptive information about the
intervention's readiness for implementation, appropriateness for
different populations, freedom from unintended or adverse effects, and
staffing and cost requirements. These utility descriptors will be
featured on the NREPP Web site to help assure a proper match between
specific prevention and treatment interventions and the settings and
populations to which they are most appropriate.
In light of SAMHSA's commitment to consumer and family involvement,
the agency is seeking ways to ensure that these groups provide input
into the assessment of interventions that achieve NREPP status. SAMHSA
seeks suggestions regarding the most useful and efficient way to
conduct this process (see Question 7 below).
Review Process for Determining Population-, Policy-, and System-Level
Outcome Ratings for Interventions
The NREPP Evidence Rating Criteria for Population-, Policy-, and
System-Level Outcomes are proposed as the basis for reviewer ratings of
outcomes generated by community prevention coalitions and other
environmental interventions to promote resiliency and recovery at the
community level. SAMHSA's rationale for use of these separate criteria
comes through a recognition that some interventions may be designed to
affect a community over time, and that the prevailing scientific
standards for assessing the effectiveness of these interventions may
indeed be different than those for interventions seeking to change
individual-level outcomes.
An outcome of a prevention or treatment intervention qualifies for
review under these 12 criteria only when it can be included in one of
the following three categories:
1. Population-Level Outcome--measures the effect of an intervention
of an existing, predefined population. Examples of such existing,
predefined populations include ``all youth residing in a
neighborhood,'' ``all female employees of a manufacturing plant,'' or
``all Native Americans receiving social services from a tribal
government.'' ``All patients receiving a specific treatment,'' in
contrast, cannot be defined as an existing, predefined population
because that population would have come into existence as a direct
response to the intervention.
2. Policy-Level Outcome--measures the effect of an intervention on
enactment, maintenance, or enforcement of policies that are assumed to
have a positive aggregate impact on resiliency or recovery. Examples of
such outcomes include ``the rate of passage of legislation restricting
access to alcoholic beverages'' or ``the percentage of arrests for
illicit drug manufacturing that result in convictions.''
3. System-Level Outcome--measures the effect of an intervention on
prevention and treatment capacity, efficiency, or effectiveness in an
existing system or community. Examples of such outcomes include
``increased capacity of a State government to quantify alcohol- or
drug-related problems'' or ``increased effectiveness of a community
treatment system to respond to the comprehensive needs of individuals
with Axis I mental health diagnoses.''
To support the transparency of the review process, SAMHSA wants
stakeholders to understand clearly the NREPP procedures and decision-
making processes. All community coalition interventions included in
NREPP will have demonstrated evidence of effectiveness at the
population, policy, or system level. The ratings will indicate the
strength of the supporting evidence, and may be as follows:
(1) Strong evidence with replication;
(2) Strong evidence without replication;
(3) Emerging evidence with replication; and
(4) Emerging evidence without replication.
All NREPP evidence ratings are defined at the level of specific
outcomes. The 12 evidence rating criteria used for population-, policy-
and system-level outcomes, summarized as an average Evidence Quality
Score (EQS) for each outcome, allow independent expert reviewers to
score along dimensions of outcome measurement, intervention fidelity,
comparison conditions, participant and data collector biases, design
and analysis, and anomalous findings. Each of the 12 criteria is
assessed by independent reviewers on a 0 to 2 scale, in which a ``1''
indicates that methodological rigor may have been acceptable and a
``2'' indicates that adequate methodological rigor was achieved for
this type of outcome.
[[Page 50388]]
Preliminary discussions of classifications have suggested that ``Strong
evidence'' be defined as an average EQS of 1.75 or above (out of a
possible 2.0), and that ``Emerging evidence'' be defined as an average
EQS between 1.50 and 1.74 (out of a possible 2.0).
The 12 criteria applied to each population-, policy-, or system-
level outcome measures are:
1. Logic-Driven Selection of Measures
2. Reliability
3. Validity
4. Intervention Fidelity
5. Nature of Comparison Condition
6. Standardized Data Collection
7. Data Collector Bias
8. Population Studied
9. Missing Data
10. Analysis Meets Data Assumptions
11. Theory-Driven Selection of Analytic Methods
12. Anomalous Findings
Outcome Measurement Criteria
1. Logic-Driven Selection of Measures
Outcome measures should be based on a theory or logic model that
associates them with the intervention.
0 = The applicant appears to have selected outcome measures for the
purpose of identifying favorable results rather than from a logic-based
rationale.
1 = There is no explicit description of a guiding logic model or theory
for measures, although a rationale for the inclusion of most measures
can be inferred.
2 = Measures are supported by a theory or logic model that associates
the intervention with the outcome.
2. Reliability
Outcome measures should have acceptable reliability to be
interpretable. ``Acceptable'' here means reliability at a level that is
conventionally accepted by experts in the field.
0 = No evidence of reliability of measures is presented.
1 = Relevant reliability measures are in the marginal range.
2 = Relevant reliability measures are in clearly acceptable ranges.
3. Validity
Outcome measures should have acceptable validity to be
interpretable.
0 = No evidence of validity of measures is presented or evidence that
is presented suggests measures are not valid.
1 = Measures has face validity.
2 = Relevant validity has been documented to be at acceptable levels in
independent studies.
4. Intervention Fidelity
The intervention should be well defined and its implementation
should be described in sufficient detail to assess whether
implementation affected outcomes.
0 = The intervention and/or its implementation are not described in
sufficient detail to verify that the intervention was implemented as
intended.
1 = The intervention and its implementation are described in adequate
detail, including justification for significant variation during
implementation.
2 = The intervention and its implementation are described in adequate
detail, reflecting variation during implementation with little or no
plausible impact on outcomes.
5. Nature of Comparison Condition
The quality of evidence for an intervention depends in part on the
nature of the comparison condition(s).
0 = Research design either lacks a comparison condition, or employs a
before/after comparison.
1 = Comparison condition was no service or wait-list (including
baseline comparison for a multipoint time series), or an active
intervention that has not been shown to be safer or more effective than
no service.
2 = Comparison condition was an active intervention shown to be as safe
as, or safer and more effective than, no service.
6. Standardized Data Collection
All outcome data should be collected in a standardized manner. Data
collectors trained and monitored for adherence to standardized
protocols provide the highest quality evidence of standardized data
collection.
0 = Data collection or archival sources used by the evaluation to
assess outcome did not use standardized data collection protocol(s).
1 = All outcome data were collected using standardized protocol(s).
2 = All outcome data were collected using standardized protocol(s) and
trained data collectors.
7. Data Collector Bias
Data collector bias is most strongly controlled when data
collectors are not aware of the interventions to which populations have
been exposed. When data collectors are aware of specific interventions,
their expectations should be controlled for through training and/or
statistical analysis methods on resultant data.
0 = Data collectors were not masked to the population's condition, and
nothing was done to control for possible bias, OR collector bias was
identified and not controlled for statistically.
1 = Data collectors were not masked to the population's condition;
possible bias was appropriately controlled for statistically or through
training.
2 = Data collectors were masked to the population's condition, or only
archival data was employed.
8. Population Studied
0 = A single group pre/posttest design was applied without a comparison
group, OR the alleged comparison group is significantly different from
the population receiving the intervention.
1 = Population(s) were studied using time trend analysis, multiple
baseline design, or a regression-discontinuity design that uses within-
group differences as a substitute for comparison groups.
2 = Population matching or similar techniques were used to compare
outcomes of population that received the intervention with the outcomes
of a valid comparison group.
9. Missing Data
Study results can be biased by missing data. Statistical methods as
supported by theory and research can be employed to control for missing
data that would bias results, but studies with no missing data needing
adjustment provide the strongest evidence.
0 = Missing data were an issue and were taken into account
inadequately, OR levels of missing data were too high to control for
bias.
1 = Missing data were an issue and were taken into account, but high
quality makes the control for bias suspect.
2 = Missing data were not an issue or were taken into account by
methods that estimate missing data.
10. Analysis Meets Data Assumptions
The appropriateness of statistical analysis is a function of the
properties of the data being analyzed and the degree to which data meet
statistical assumptions.
0 = Analyses were clearly inappropriate to the data collected; severe
violation(s) of assumptions make analysis uninterpretable.
1 = There were minor violations of assumptions, making interpretation
of results arguable.
2 = There were no or only very minor violations of assumptions; result
were generally interpretable.
[[Page 50389]]
11. Theory-Driven Selection of Analytic Methods
In addition to the properties of the data, analytic methods should
be based on a logic model or theory underlying the intervention.
Changes to analytic methods after initial data analysis (e.g., to
dredge for significant results) decrease the confidence that can be
placed in the findings.
0 = Analysis selected appears inconsistent with the intervention theory
or hypotheses; insufficient rationale was provided by the investigator.
1 = Analysis selected appears inconsistent with the intervention logic
model or hypotheses, but the investigator provides a potentially viable
rationale.
2 = Analysis is widely accepted by the field as consistent with the
intervention logic model or hypotheses.
12. Anomalous Findings
Findings that contradict the theories and hypotheses underlying an
intervention suggest the possibility of confounding causal variables
and limit the validity of study findings.
0 = There were anomalous findings suggesting alternate explanations for
outcomes reported that were not acknowledged by the applicant.
1 = There were a few anomalous findings, but additional analysis or
previous literature cited by the applicant provide a reasonable
explanation.
2 = There were no anomalous findings, OR researchers explained
anomalous findings in a way that preserves the validity of results
reported.
Re-Review of Existing NREPP Programs
As noted above, SAMHSA believes it is important to ensure that both
current and future NREPP interventions meet consistent scientific
standards so that the public and other interested stakeholders can be
confident in the effectiveness of these interventions. With this in
mind, SAMHSA is committed to expeditiously re-reviewing all existing
NREPP programs under the new process. As part of this effort, SAMHSA
plans to provide--directly or indirectly--sufficient resources to each
existing NREPP program to cover costs associated with a re-review. In
addition, programs already received by NREPP and pending review will be
reviewed under the new process. If additional support is needed by
these pending programs regarding the new review processes, these
resources will also be provided.
In order to accomplish these re-reviews efficiently and
expeditiously, NREPP review coordinators will work with each program to
obtain any additional documentation that might be needed for re-review.
These review coordinators will then conduct a re-review of each program
against the new review criteria. Programs with favorable re-reviews
will be included in the new NREPP system when it is launched in 2006.
Programs not receiving favorable re-reviews will have the opportunity
to appeal the re-review decision, and will be eligible for re-review by
independent, external reviewers. However, the schedule for re-reviews
of appealed programs will be subject to SAMHSA Administrator and SAMHSA
Center Director review priorities.
New Web Site
The primary goal of the revised NREPP Web site--http://www.nationalregistry.samhsa.gov--will be to provide the public with
contemporary and reliable information about the scientific basis and
practicality of interventions to prevent and treat mental and substance
use disorders. All interventions achieving NREPP status will be listed
on the Web site. Average ratings and evaluate scores from scientific
peer reviewers, as well as information on the utility and
transferability of these interventions, will be posted on the site.
In addition, a searchable outcomes database of evidence-based
interventions will be a key feature. The Web site will also contain a
variety of learning and self-assessment tools for prospective and
current NREPP interventions to continuously improve their scientific
evidence base. Features of the new Web site will include:
Evidence rating criteria and utility descriptors
Detailed review guidelines
Self-assessment tool to assist interventions in
determining if they are ready to submit an application to NREPP
Detailed information on how to apply
Links to technical assistance resources available to
potential applicants
Relevant resources, including publications, presentations,
links, and other supplemental materials
Frequently Asked Questions (FAQs)
Information on how to contact a representative from the
NREPP team
Glossary of terms
Support for Innovative Interventions
SAMHSA recognizes that the long term utility and value of NREPP
rests, in part, on the ability of SAMHSA and others to support efforts
to evaluate and document the evidence-base of innovative interventions
in ways that will maximize their opportunity for entry into NREPP.
SAMHSA is considering potential options for both the direct and
indirect provision of such support, and will seek to clarify its
intentions in this area sometime in Fiscal Year 2006.
Questions To Consider in Making Your Comments
Responders should feel free to comment on any, or all, questions,
as well as provide relevant suggestions not included in the specific
questions. In order to facilitate the compilation and analysis of
comments, responders are asked to be explicit about the questions to
which they are responding.
1. SAMHSA is seeking to establish an objective, transparent,
efficient, and scientifically defensible process for identifying
effective, evidence-based interventions to prevent and/or treat mental
and substance use disorders. Is the proposed NREPP system--including
the suggested provisions for screening and triage of applications, as
well as potential appeals by applicants--likely to accomplish these
goals?
2. SAMHSA's NREPP priorities are reflected in the agency's matrix
of program priority areas. How might SAMHSA engage interested
stakeholders on a periodic basis in helping the agency determine
intervention priority areas for review by NREPP?
3. There has been considerable discussion in the scientific
literature on how to use statistical significance and various measures
of effect size in assessing the effectiveness of interventions based
upon both single and multiple studies (Schmidt & Hunter, 1995;
Rosenthal, 1996; Mason, Schott, Chapman, & Tu, 2000: Rutledge & Loh,
2004). How should SAMHSA use statistical significance and measures of
effect size in NREPP? Note that SAMHSA would appreciate receiving
citations for published materials elaborating upon responders
suggestions in this area.
4. SAMHSA's proposal for NREPP would recognize as effective several
categories of interventions, ranging from those with high-quality
evidence and more replication to those with lower quality evidence and
fewer replications. This would allow for the recognition of emerging as
well as fully evidence-based interventions. Some view this as a
desirable feature that reflects the continuous nature of evidence;
provides important options for interventions
[[Page 50390]]
recipients, providers, and funders when no or few fully evidence-based
interventions are available; and helps promote continued innovation in
the development of evidence-based interventions. Others have argued
that several distinct categories will confuse NREPP users. Please
comment on SAMHSA's proposal in this area.
5. SAMHSA recognizes the importance of considering the extent to
which interventions have been tested with diverse populations and in
diverse settings. Therefore, the agency anticipates incorporating this
information into the web site descriptions of interventions listed on
NREPP. This may allow NREPP users to learn if interventions are
applicable to their specific needs and situations, and may also help to
identify areas where additional studies are needed to address the
effectiveness of interventions with diverse populations and in diverse
locations.
SAMHSA is aware that more evidence is needed on these topics.
Please comment on SAMHSA's approach in this area.
6. To promote consistent, reliable, and transparent standards to
the public, SAMHSA proposes that all existing programs on NREPP meet
the prevailing scientific criteria described in this proposal, and that
this be accomplished through required re-reviews of all programs
currently on NREPP. SAMHSA has considered an alternative approach that
would ``grandfather'' all existing NREPP programs under the new system,
but would provide clear communication that these existing programs have
not been assessed against the new NREPP scientific standards. Please
comment on which approach you believe to be in the best interests of
SAMHSA stakeholders.
7. What types of guidance, resources, and/or specific technical
assistance activities are needed to promote greater adoption of NREPP
interventions, and what direct and indirect methods should SAMHSA
consider in advancing this goal?
8. SAMHSA is committed to consumer, family, and other nonscientist
involvement in the NREPP process. The panels convened by SAMHSA and
described earlier in this notice suggested that these stakeholders be
included specifically to address issues of intervention utility and
practicality. Please comment on how consumer, family, and other
nonscientist stakeholders could be involved in NREPP.
9. SAMHSA has identified NREPP as one source of evidence-based
interventions for selection by potential agency grantees in meeting the
requirements related to some of SAMHSA's discretionary grants. What
guidance, if any, should SAMHSA provide related to NREPP as a source of
evidence-based interventions for use under the agency's substance abuse
and mental health block grants?
10. SAMHSA believes that NREPP should serve as an important, but
not exclusive source, of evidence-based interventions to prevent and/or
treat mental and substance use disorders. What steps should SAMHSA take
to promote consideration of other sources (e.g., clinical expertise,
consumer or recipient values) in stakeholders' decisions regarding the
selection, delivery and financing of mental health and substance abuse
prevention and treatment services?
11. SAMHSA anticipates that once NREPP is in operation, various
stakeholders will make suggestions for improving the system. To
consider this input in a respectful, deliberate, and orderly manner,
SAMHSA anticipates annually reviewing these suggestions. These reviews
would be conducted by a group of scientist and nonscientist
stakeholders knowledgeable about evidence in behavioral health and the
social sciences. Please comment on SAMHSA's proposal in this area.
References
Mason, CA, Scott, KG, Chapman, DA, Tu, S. A review of some
individual and community level effect size indices for the study of
risk factors for child and adolescent development. Ed. Psychol.
Measurement. 2000; 60(3): 385-410.
Rosenthal, JA. Qualitative descriptors of strength of
association and effect size. J. of Soc. Serv. Research. 1996; 21(4):
37-59.
Rutledge, T, Loh, C. Effect size and statistical testing in the
determination of clinical significance in behavioral medicine
research. Annals of Beh. Med. 2004; 27(2): 138-145.
Schmidt, F, Hunter, JE. The impact of data-analysis methods on
cumulative research knowledge: statistical significance testing,
confidence intervals, and meta-analysis. Eval. Health Prof. 1995
Dec; 18(4): 408-27.
[FR Doc. 05-17034 Filed 8-25-05; 8:45 am]
BILLING CODE 4160-01-M