[Federal Register Volume 86, Number 154 (Friday, August 13, 2021)]
[Notices]
[Pages 44738-44740]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 2021-17339]


=======================================================================
-----------------------------------------------------------------------

DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT

[Docket No. FR-6278-N-01]


HUD Program Evaluation Policy--Policy Statement

AGENCY: Office of Policy Development and Research, HUD.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: This policy statement articulates the core principles and 
practices of the Department of Housing and Urban Development's 
evaluation and research activities. This policy reaffirms HUD's 
commitment to conducting rigorous, relevant evaluations and to using 
evidence from evaluations to inform policy and practice.

DATES: August 13, 2021.

FOR FURTHER INFORMATION CONTACT: For information about this notice, 
contact Todd M. Richardson, Evaluation Officer, Office of Policy 
Development and Research, U.S. Department of Housing and Urban 
Development, 451 7th Street SW, Washington, DC 20410, telephone (202) 
402-5922. The listed telephone number is not a toll-free number. 
Persons with hearing- or speech-impairments may access this number 
through TTY by calling Federal Relay Service at 1-800-877-8339 (this is 
a toll-free number).

SUPPLEMENTARY INFORMATION:

I. Background

    Evaluation activity occurs in several offices at HUD, but the 
special mission of HUD's Office of Policy Development and Research 
(PD&R) is to inform HUD policy development and implementation to 
improve life in American communities through conducting, supporting, 
and sharing research, surveys, demonstrations, program evaluations, and 
best practices. Within HUD, PD&R is responsible for most, but not all, 
program evaluations. The office provides reliable and objective data 
and analysis to help inform policy decisions.
    In July 2016, GAO issued a report entitled ``Department of Housing 
and Urban Development: Actions Needed to Incorporate Key Practices into 
Management Functions and Program Oversight,'' (GAO 16-497) in which GAO 
presented a broad assessment of HUD's management of its operations and 
programs.\1\ In the report, GAO examined HUD efforts to: (1) Meet 
Federal requirements and implement key practices for management 
functions, including performance planning and reporting and human 
capital, financial, acquisition, and information technology (IT) 
management; and (2) oversee and evaluate programs.
---------------------------------------------------------------------------

    \1\ See http://www.gao.gov/assets/680/678551.pdf.
---------------------------------------------------------------------------

    PD&R is the primary office within HUD responsible for data 
analysis, research, program evaluations, and policy studies that inform 
the development and implementation of programs and policies across HUD 
offices. PD&R undertakes program evaluations, often by using a process 
that includes convening expert panels. However, GAO found that PD&R had 
neither developed agency-wide, written policies for its program 
evaluations, nor documented the criteria used to select the expert 
panels and review the quality of program evaluations.
    On December 6, 2016 (81 FR 87949), HUD issued a policy statement in 
the Federal Register \2\ responding to the GAO report by setting out 
the core principles and practices of PD&R's evaluation and research 
activities. This statement incorporated some language from a policy 
statement by the Office of Policy, Research, and Evaluation of the 
Administration for Children and Families of the U.S. Department of 
Health and Human Services.
---------------------------------------------------------------------------

    \2\ https://www.federalregister.gov/d/2016-29215.
---------------------------------------------------------------------------

    On January 14, 2019, the Foundations for Evidence-Based 
Policymaking Act of 2018 (``Evidence Act''), Public Law 115-435, was 
enacted.\3\ Section 101 of the Evidence Act created 5 U.S.C. 311-315 
and mandated that the head of each agency appoint an Evaluation 
Officer, including at HUD. This officer must establish common standards 
for all HUD evaluations, whether performed by PD&R or another office. 
This issuance articulates department-wide evaluation standards and 
states other new principles based on PD&R's experience since the 
November 2016 publication.
---------------------------------------------------------------------------

    \3\ https://www.congress.gov/115/plaws/publ435/PLAW-115publ435.pdf. See also the website, at https://www.congress.gov/bill/115th-congress/house-bill/4174, on the bill, H.R. 4174 of the 
115th Congress, that became the Act.
---------------------------------------------------------------------------

II. HUD Program Evaluation Policy

    Section 101 of the Evidence Act defines ``evaluation'' to mean ``an 
assessment using systematic data collection and analysis of one or more 
programs, policies, and organizations intended to assess their 
effectiveness and efficiency.''
    HUD has identified the following core principles and practices as 
fundamental to ensuring high-quality and consistent evaluation results: 
Rigor, relevance, transparency, independence, ethics, and technical 
innovation. This policy applies to all HUD-sponsored

[[Page 44739]]

evaluations and regulatory impact analyses; they also apply to the 
selection of projects, contractors, and HUD staff that are involved in 
evaluations.

Rigor

    HUD is committed to using the most rigorous methods that are 
appropriate to the evaluation questions and feasible within budget and 
other constraints. The need for rigor is not restricted to impact 
evaluations; rigor is also necessary in implementation or process 
evaluations, descriptive studies, outcome evaluations, and formative 
evaluations; in both qualitative and quantitative approaches. Rigor 
requires ensuring that inferences about cause and effect are well 
founded (internal validity); requires clarity about the populations, 
settings, or circumstances to which results can be generalized 
(external validity); requires that researchers seek to understand and 
correct for implicit bias in the formulation of research questions and 
methods; and requires the use of measures that accurately capture the 
intended information (measurement reliability and validity). Implicit 
biases are discriminatory biases that reflect unidentified traces of 
past experience, including implicit attitudes that affect feelings, 
thoughts or actions, and implicit stereotypes that affect how others 
are characterized. Survey instruments are pre-tested with members of 
the population to be studied to increase measurement validity. When 
statistically appropriate, and particularly if the number of hypotheses 
being tested is large, HUD will require the use of commonly accepted 
adjustments to classical statistical testing to reduce the probability 
that random outliers are presented as meaningful.
    In assessing the effects of programs or services, HUD evaluations 
use methods that isolate to the greatest extent possible the impacts of 
the programs or services from other influences, such as trends over 
time, geographic variation, or pre-existing differences between 
participants and non-participants. Where feasible, research should 
employ a treatment group and a counterfactual. A treatment group is a 
population that has received an intervention. A counterfactual is a 
population that did not receive the intervention and can credibly 
represent what would have happened to the treatment group in the 
absence of the intervention according to the above standard for rigor. 
For such causal questions, experimental approaches are preferred. When 
experimental approaches are not feasible, HUD uses the most rigorous 
approach that is feasible.
    In both quantitative and qualitative research, rigor means having 
clear research questions and an explicit analytic framework; 
justification for case selection and sampling methods in relation to 
research goals; and transparent, verifiable methods of systematic data 
collection and analysis, auditable records, and attention to possible 
alternative interpretations during analysis and writing.
    HUD ensures that contractors and grantees conducting evaluations 
have appropriate expertise through emphasizing the requirement for 
rigor in requests for proposals and funding opportunity announcements, 
noting that applicants' capacity for rigor will be evaluated in the 
selection process. In addition, HUD will judge research teams with 
equal capacity for rigor to be more qualified if the team includes 
researchers demographically similar to or knowledgeable about the 
perspectives and lived experiences of the populations studied.
    HUD employs a strategic human capital development plan to hire, 
train, and retain a workforce that ensures staff have the tools and 
resources to accomplish the mission.

Relevance

    The HUD evaluation agenda reflects the legislative requirements and 
policy issues related to HUD's mission. HUD solicits input from 
stakeholders, both internal and external, including stakeholders with 
lived experience, such as program participants, and grantees, on the 
selection of programs to be evaluated, initiatives, demonstrations, and 
research questions. For new initiatives and demonstrations in 
particular, evaluations will be more feasible and useful when planned 
in advance, in concert with the development of the initiative or 
demonstration, rather than as an afterthought. HUD strives to 
understand the relevance of its completed research through concerted 
stakeholder engagement, including with people and grantees affected by 
HUD programs, to continuously improve its research agenda.
    Expert panels include research and other subject matter experts and 
are diverse in ways tailored to the study, including racial and ethnic 
diversity and representatives of the studied populations.
    HUD strives to design program evaluations and other analyses to 
better understand structural racism and to reveal unequal benefits and 
harms across social groups as relevant, with special attention to race, 
national origin, color, familial status, religion, disability, age, and 
sex (including gender identity and sexual orientation).
    To support this goal, insofar as feasible, HUD collects and reports 
data on race, ethnicity, gender, and income, and other characteristics 
of underrepresented and underserved communities relevant for research 
and analysis efforts.
    HUD also encourages research to engage studied populations. 
``Engagement'' means the deliberate and intentional inclusion of the 
thoughts and perspectives of studied groups, such as program 
participants, grantees, and underrepresented and underserved 
populations. This includes collecting people's thoughts and 
perspectives through standard (valid and rigorous) research methods 
such as surveys, focus groups, in-depth interviews, or ethnography to 
produce published research that conveys studied populations' thoughts 
and perspectives. In another type of engagement, people influence the 
research that is about them. This includes a wide range of activities 
that lie on a continuum from simple input (which should be documented 
and published) to full co-creation of any aspect of the research, from 
topic selection to research design, data collection, data analysis, 
interpretation, writing, or dissemination, or even being on the 
research team. An example of simple input is to include in the final 
report a summary of comments on the research by members of studied 
groups. Examples of collaboration include, but are not limited to, co-
creating the list of topics to be covered in a survey or having members 
of a studied group on the research team.
    To raise awareness of and spur creative approaches to engagement of 
studied populations in program evaluations and HUD-sponsored research, 
HUD may require contractors and grantees to explain how their research 
will and will not engage studied populations. HUD recognizes that 
engagement must be tailored to particular research efforts.
    HUD retains the right to determine research methods.
    HUD disseminates findings in ways that are accessible and useful to 
policymakers, practitioners, and members of communities affected by HUD 
programs and policies. Published findings will be accessible to 
individuals with disabilities pursuant to Section 508 of the 
Rehabilitation Act. PD&R partners with other HUD program offices to 
inform internal and external stakeholders through disseminating

[[Page 44740]]

evidence from HUD-sponsored evaluations.

Transparency

    HUD will release methodologically valid evaluations without regard 
to the findings. Evaluation reports must describe the methods used, 
including strengths and weaknesses, and discuss the generalizability of 
the findings. Evaluation reports must present comprehensive results, 
including favorable, unfavorable, and null findings. HUD will publish 
interim findings, as projected in the initial research design. If there 
are indications that the findings of the final report may differ, HUD 
will provide appropriate qualifications accompanying the publication to 
guard against misunderstanding or misuse of the interim findings. If 
there are interim findings, HUD will publish those findings even if 
there are indications that the findings of the final report may differ. 
When findings are highly relevant to current policy, HUD evaluations 
carry a foreword articulating the policy position of the Department 
with respect to those findings.
    If the findings of a HUD evaluation will have broad public interest 
and includes a counterfactual, PD&R will publish a synopsis of the 
research design, data collection and analysis plan soon after it is 
approved and will require interim and final reports that deviate from 
that document to explain how they deviate and why.
    HUD publishes a 5-year Learning Agenda \4\ that outlines the 
research and evaluation that it believes would be of greatest value to 
public policy. PD&R lists all ongoing evaluation projects at the 
HUDUSER.gov website \5\ and updates it quarterly. PD&R will release 
evaluation results timely, usually within 4 months of receiving the 
final report.
---------------------------------------------------------------------------

    \4\ https://www.huduser.gov/portal/about/pdr_learningagenda.html.
    \5\ https://www.huduser.gov/portal/about/PDR-Research.html.
---------------------------------------------------------------------------

    HUD will, where possible, archive administrative and evaluation 
data for secondary use by interested researchers. HUD typically builds 
requirements into contracts to prepare data sets for secondary use. 
Access for external researchers may be provided directly through data 
licenses or indirectly through inter-agency agreements. This policy may 
not apply for data that has obvious commercial value, such as mortgage 
performance data. HUD staff may publish the results of their 
scholarship and analysis in any forum, so long as they do not claim to 
speak for the Department.
    HUD evaluation contracts will generally permit contractors to 
publish their findings within 6 months of the termination of the 
contract if HUD has not already published them.

Independence

    Independence and objectivity are core principles of evaluation. 
Agency and program leadership, program staff, service providers, and 
others participate actively in setting evaluation priorities, 
identifying evaluation questions, and assessing the implications of 
findings. However, it is important to insulate evaluation functions 
from undue influence and from both the appearance and the reality of 
bias. To promote objectivity, HUD protects independence in the design, 
conduct, and analysis of evaluations. To this end:
     HUD conducts evaluations through the competitive award of 
grants and contracts to external experts who are free from conflicts of 
interest.
     HUD also conducts evaluations in-house and supports 
unsolicited external evaluation proposals with funding, data, or both.
     The Evaluation Officer will consult with the HUD office 
with lead responsibility on the design of evaluation projects and 
analysis plans and will advise that office on whether to publish 
evaluation reports.

Ethics

    HUD-sponsored evaluations must be conducted in an ethical manner 
and safeguard the dignity, rights, safety, and privacy of participants. 
HUD-sponsored evaluations must comply with both the spirit and the 
letter of relevant requirements such as regulations governing research 
involving human subjects. In particular, PD&R protects the privacy of 
HUD-assisted households and HUD-insured borrowers through its Rule of 
Eleven; that is, PD&R allows no disclosure of information about the 
characteristics of any group of individuals or households numbering 
fewer than eleven by PD&R staff, contractors, grantees, or licensees.
    HUD is a signatory to the Federal Policy for the Protection of 
Human Subjects, generally known as the ``Common Rule.'' 24 CFR part 60, 
which includes its own requirements for ensuring adequate provisions to 
protect the privacy of human subjects research.\6\
---------------------------------------------------------------------------

    \6\ https://www.ecfr.gov/cgi-bin/retrieveECFR?n=pt24.1.60.
---------------------------------------------------------------------------

    HUD does not tolerate plagiarism, or fabrication or deliberate 
mischaracterization of data by staff, contractors or grantees who are 
engaged in evaluation activity.

Technical Innovation

    PD&R supports and employs new methods of data collection and 
analysis that more reliably and efficiently answer research questions 
than old methods do.

Application of These Principles to Economic Analysis of Regulations

    Economic analysis of both existing and proposed regulations, 
properly conducted, is a critical tool in improving public policy. 
Economists at HUD rely on the insights, data, and empirical estimates 
from rigorous program evaluations when predicting the economic impact 
of an incremental change to the program. In any HUD Regulatory Impact 
Analysis:
     HUD analyzes whether the issues addressed by the 
regulation stem from a market failure, government failure, or other 
systemic problem, and whether the regulation addresses the root causes 
of those problems.
     HUD uses and as necessary produces the best objective 
estimates of the benefits, costs, and transfers resulting from the 
regulation, taking into account gaps and uncertainties in the available 
data and methodologies.
     HUD assesses the economic benefits, costs, and transfers 
of proposed regulatory actions as required by Executive Order 12866.\7\ 
HUD provides additional analysis of impacts across groups defined by 
race, ethnicity, and other characteristics that may define 
underrepresented and underserved groups when such analyses are relevant 
and feasible.
---------------------------------------------------------------------------

    \7\ See https://www.archives.gov/files/federal-register/executive-orders/pdf/12866.pdf for the original order, and see 
https://www.archives.gov/federal-register/executive-orders/1993-clinton.html for citations of, and links to, other executive orders 
that amended or supplemented this order.
---------------------------------------------------------------------------

     Where clear alternatives to the regulatory actions exist, 
HUD objectively estimates the benefits, costs, and transfers of those 
alternatives, and additional analysis of impacts of those alternatives 
across underrepresented or underserved groups as well.

Todd M. Richardson,
Evaluation Officer.
[FR Doc. 2021-17339 Filed 8-12-21; 8:45 am]
BILLING CODE 4210-67-P