Rental Housing: HUD Can Improve Its Process for Estimating Fair
Market Rents (31-MAR-05, GAO-05-342).
The Department of Housing and Urban Development (HUD) annually
estimates fair market rents (FMR) for standard quality rental
units throughout the United States. Among other uses, FMRs help
determine subsidies for almost 2 million low-income families in
the nation's largest rental assistance program. However, concerns
exist that FMRs can be inaccurate--often, too low, preventing
program participants from finding affordable housing. Also, HUD
will soon derive FMRs from a new source, the American Community
Survey (ACS), which processes data somewhat differently than
HUD's current data sources, including the decennial census. You
asked us to review (1) how HUD estimates FMRs, (2) how accurate
FMRs have been, (3) how ACS data may affect accuracy, and (4)
other changes HUD can make to improve the estimates.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-342
ACCNO: A20577
TITLE: Rental Housing: HUD Can Improve Its Process for
Estimating Fair Market Rents
DATE: 03/31/2005
SUBJECT: Census
Comparative analysis
Data collection
Data integrity
Fair market value
Housing programs
Low income housing
Population statistics
Rent subsidies
Rental housing
Strategic planning
Surveys
Urban development programs
1990 Decennial Census
2000 Decennial Census
Census Bureau American Community Survey
Census Bureau American Housing Survey
HUD Housing Choice Voucher Program
HUD Section 8 Tenant-Based Program
Consumer Price Index
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-342
* Report to the Ranking Minority Member, Subcommittee on Housing and
Transportation, Committee on Banking, Housing, and Urban Affairs, U.S.
Senate
* March 2005
* RENTAL HOUSING
* HUD Can Improve Its Process for Estimating Fair Market Rents
* Contents
* Results in Brief
* Background
* Fortieth Percentile of Rents
* Gross Rent
* Typical, Standard Rental Units
* Recent Movers
* HUD Estimates FMRs by Defining Housing Markets, Choosing Data
Sources, Updating Rent Data, and Evaluating Public Input
* HUD Establishes Areas, Uses Survey Data to Benchmark and
Adjust FMRs
* HUD Provides Opportunities for Public Input on Proposed FMRs
as Well as Those in Effect
* Most FMR Estimates Were Accurate within 10 Percent of the Census
or Other Rebenchmarking Surveys
* Over Two-thirds of All FMRs for 2000 Were Accurate within 10
Percent of Rents Derived from the 2000 Census
* Quality Survey Data Tended to Produce FMR Estimates That
Were Accurate within 10 Percent
* More Recent Data
* Data from Higher Quality Surveys
* More Relevant (Local) Data
* HUD Believes That Other Factors May Influence the Accuracy
of FMR Estimates
* General Survey Error
* Characteristics of Nonmetropolitan Areas
* Difficulty in Estimating Utility Costs
* Recent Mover Rent Changes in Metropolitan Areas
* ACS Could Improve the Accuracy of FMRs by Providing HUD with More
Recent, Better Data
* The ACS Is a Higher Quality Survey That Provides More Recent
and Local Data
* Higher Quality Survey
* More Recent Data
* More Local Data
* HUD Expects to First Use ACS Data to Update Fiscal Year 2006
FMRs
* ACS Data Pose Certain Challenges to HUD That May Affect FMR
Estimation and Accuracy
* Averaging
* Inflation-Adjusted Costs
* Techniques to Deal with Missing Responses
* Reporting Differences between the Decennial Census Long
Form and the ACS
* Despite Challenges, the ACS Remains Likely the Best Data
Source for FMRs
* HUD Did Not Follow One of Its Data Quality Guidelines and May
Lack Data Sources to Assess the Accuracy of Future FMRs
* HUD Has Not Followed Its Data Quality Guideline on
Objectivity
* HUD's Declining Use of RDD Surveys and AHS Data May Limit
Its Options for Assessing the Accuracy of Future FMRs
* Conclusions
* Recommendations for Executive Action
* Agency Comments and Our Evaluation
* Objectives, Scope, and Methodology
* Comments from the Department of Housing and Urban Development
* GAO Contacts and Staff Acknowledgments
* GAO Contacts
* Staff Acknowledgments
United States Government Accountability Office
Report to the Ranking Minority Member,
Subcommittee on Housing and Transportation, Committee on Banking, Housing, and
Urban Affairs, U.S. Senate
March 2005
RENTAL HOUSING
HUD Can Improve Its Process for Estimating Fair Market Rents
a
GAO-05-342
RENTAL HOUSING
HUD Can Improve Its Process for Estimating Fair Market Rents
What GAO Found
According to HUD, the typical process for estimating FMRs includes
benchmarking, or developing baseline rents for each FMR area (generally
county-based) using census data or other surveys for the years between
censuses; adjusting those rents to bring them up to date; and seeking
public comment before finalizing the numbers. HUD generally uses Consumer
Price Index and telephone survey data to adjust baseline rents-that is, to
account for rent changes since data used for baseline estimates were
collected and to project the estimates into the next fiscal year (when
they will be in use for subsidy purposes). HUD then lists the proposed
FMRs in the Federal Register for public comment. These comments can lead
to changes in FMRs, but only when they include new data or lead HUD to
conduct a new survey.
About 69 percent of all areas had FMR estimates in use in 2000 that were
within 10 percent of rents indicated by the 2000 decennial census-the most
accurate comparison data available for each FMR area. This represents an
improvement over HUD's 1990 estimates, as the table below shows.
Similarly, about 73 percent of 153 areas whose FMRs HUD rebenchmarked
after 2000 were within 10 percent of rents derived from recent surveys. In
general, GAO found that areas that are rebenchmarked with more recent data
tended to have FMRs in the most accurate range (within 10 percent).
Using ACS data could improve the accuracy of FMRs by allowing HUD to
benchmark more areas more frequently than is possible with current data
sources, using more recent data-a factor that GAO's analysis suggests is
related to accuracy. HUD's first use of ACS data will be to update
existing baseline estimates for the fiscal year 2006 FMRs; HUD expects to
use ACS data to set baseline rents for some fiscal year 2008 FMRs.
HUD could improve its FMR estimation process by consistently following its
guidelines relating to the transparency of FMRs and ensuring that it can
assess the accuracy of ACS-based FMRs. Transparency would be improved by
fully documenting the estimation process so that FMRs can be independently
reproduced. Even ACS-based FMRs may not always be accurate, and HUD's
policies require mechanisms to correct information it disseminates.
Accuracy of HUD's Fiscal Years 2000 and 1990 FMR Estimates
Compared with decennial census rents-percentage of FMRs that were:
Higher by Higher by Lower by Lower by20% or 10% to10% to 20% orFiscal year
more 19.9% Within 10% 19.9% more
1990 25 3039 4
Sources: GAO analysis of HUD data (2000 figures) and HUD (1990 figures).
United States Government Accountability Office
Contents
Letter 1
Results in Brief 3 Background 6 HUD Estimates FMRs by Defining Housing
Markets, Choosing Data
Sources, Updating Rent Data, and Evaluating Public Input 10 Most FMR
Estimates Were Accurate within 10 Percent of the Census
or Other Rebenchmarking Surveys 19 ACS Could Improve the Accuracy of FMRs
by Providing HUD with
More Recent, Better Data 27 HUD Did Not Follow One of Its Data Quality
Guidelines and May
Lack Data Sources to Assess the Accuracy of Future FMRs 34 Conclusions 37
Recommendations for Executive Action 38 Agency Comments and Our Evaluation
38
Appendixes
Appendix I: Objectives, Scope, and Methodology 42
Appendix II: Comments from the Department of Housing and Urban Development
45
Appendix III: GAO Contacts and Staff Acknowledgments 52
GAO Contacts 52
Staff Acknowledgments 52
Table 1:
Tables
Table 2: Table 3: Table 4:
Table 5: Accuracy of HUD's Fiscal Years 2000 and 1990 FMR Estimates
Compared with Rents from Census 21 Accuracy of HUD's FMR Estimates
Compared with Rents from RDD Surveys (by Reason for Survey, 2001-05) 22
Accuracy of FMR Estimates in 2000 Compared with Rents from Census (Based
on Age of Baseline FMR Data) 23 Accuracy of FMR Estimates in 2000 Compared
with Rents from Census (Based on Type of Rebenchmarking Survey) 24
Accuracy of FMR Estimates in 2000 Compared with Rents from Census (by Type
of Update Factor) 25
Figure 1: Example of 40th Percentile of Rent 9
Figures
Figure 2: HUD's Typical Process for Estimating FMRs 12 Figure 3: HUD
Regions 16
Page i GAO-05-342 Fair Market Rents
Contents
Figure 4: Accuracy of HUD's Fiscal Year 2000 FMR Estimates 20
Figure 5: Scope of ACS Rebenchmarking as Related to FMR Area
Size and Housing Choice Voucher Program Data 30
Abbreviations
ACS American Community Survey
AHS American Housing Survey
BAH basic allowance for housing
CPI Consumer Price Index
DOD Department of Defense
FMR fair market rent
HOPWA Housing Opportunities for Persons with AIDS
HUD Department of Housing and Urban Development
LIHTC Low Income Housing Tax Credit
NAS National Academy of Sciences
OMB Office of Management and Budget
PHA public housing agency
RDD random digit dialing
SRO Moderate Rehabilitation Single-Room Occupancy
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States Government Accountability Office Washington, D.C. 20548
March 31, 2005
The Honorable Jack Reed Ranking Minority Member Subcommittee on Housing
and Transportation Committee on Banking, Housing, and Urban Affairs United
States Senate
Dear Senator Reed:
The Department of Housing and Urban Development's (HUD) Housing Choice
Voucher Program, commonly known as "Section 8" tenant-based assistance, is
the largest ongoing rental assistance program in the United States,
serving almost 2 million families with a budget of about $16.9 billion for
fiscal year 2005. The Housing Choice Voucher Program provides subsidies to
help low-income families afford rental housing in the private market. To
determine the amounts of the subsidies it will provide to low-income
families under the Voucher Program, and for other purposes, HUD annually
estimates fair market rents (FMR)-that is, rent estimates that include
utilities. From time to time, public housing agencies and others have
expressed concern that FMR estimates can be inaccurate-often, too
low-thereby preventing voucher holders from being able to find affordable
housing in certain areas.
HUD estimates FMRs for all bedroom size units for each area in the entire
United States (typically, counties) in advance of the year during which
they will be effective. HUD currently uses rent data from a variety of
surveys- the Bureau of the Census' decennial census long form is the major
survey used-as a baseline (or benchmark) for estimating FMRs throughout
the country.1 Between censuses, HUD's practice has been to rebenchmark
census-based FMRs with data from the American Housing Survey (AHS), a
Census Bureau survey performed in certain metropolitan areas every few
years, and from Random Digit Dialing (RDD) surveys, telephone interviews
that gather rent and other data for estimating FMRs for a limited number
of metropolitan and nonmetropolitan areas annually, conducted by HUD
contractors. However, a new Census Bureau product, the American Community
Survey (ACS), is replacing the decennial census long form and
1As part of the decennial census since 1960, the Census Bureau has mailed
separate long-form questionnaires to a sample of households to collect
detailed information on demographic, housing, social, and economic
characteristics.
will become the major source of rent data for FMR estimates in every area.
With the ACS, the Census Bureau will publish results annually based on 1-,
3-, or 5-year averages, depending on the population size of the area
surveyed, rather than every 10 years. For example, HUD will receive 1-year
average data (the average of 12 months) annually for areas in which the
majority of voucher holders reside.
You asked us to review HUD's process for estimating FMRs and the impact
that the incorporation of the ACS could have on the accuracy of FMRs. Our
report discusses (1) how HUD estimates FMRs, (2) how accurate HUD's FMR
estimates have been, (3) how and when the use of ACS data to estimate FMRs
may affect their accuracy, and (4) the potential for other changes HUD
could make to improve the way it estimates FMRs and their accuracy.
To determine the general process for how HUD estimates FMRs, we analyzed
statutes, regulations, and agency documents and interviewed HUD officials.
To determine how accurate FMR estimates were, we compared all two-bedroom
FMRs that HUD put in effect for fiscal year 2000 with census data for the
same year because (1) the decennial census rent estimates are considered
to be the closest estimates of the true value of those rents and (2) HUD
estimates FMRs for other bedroom sizes as a multiple of the FMR it sets
for two-bedroom units. We also compared HUD's estimated FMRs in effect
during fiscal years 2001-05 for selected geographic areas with rents
estimated using data from surveys HUD and others conducted over this
period. After making these comparisons, we performed an associative
analysis-that is, we analyzed specific components of (or data inputs to)
the FMR estimation process to see how they might relate to the accuracy of
FMRs. To determine how and when HUD will use ACS data to estimate FMRs and
what their potential effects on the accuracy of FMRs would be, we compared
ACS with the other major surveys HUD uses to estimate FMRs, identified
salient characteristics of the ACS data, and reviewed HUD's plans for
using ACS data. To determine other changes HUD could make to improve its
estimation process and the accuracy of FMRs, we analyzed data quality
guidelines and then assessed HUD's estimation process against the
guidelines. We also interviewed officials from HUD headquarters and field
offices, as well as experts and researchers who routinely work with
housing data sources. Appendix I provides additional details on our
objectives, scope, and methodology.
Throughout this report, we refer to the "quality" of surveys or the
"quality" of data. We use quality as an overarching term for important
characteristics related to the accuracy, recency, and relevance of data
sources and surveys. Specifically, for purposes of this report, we
describe quality data obtained from surveys as
o "accurate" when all types of rental housing units have a chance of
being selected for the survey and the sample size is large enough to
provide a 90 or 95 percent likelihood that the survey's estimates will
be within 5 to 10 percent of what would be found if the entire
population (i.e., all rents) were known;
o "recent" to the extent that the time between when data are collected
and subsequently used is minimized; and
o "relevant" when surveys collect, at a minimum, data on rents for HUD's
program purposes and, among the survey data sources available, HUD
chooses the survey that most closely corresponds to the FMR area.
These characteristics generally match those in data quality guidelines
used by other federal agencies, and the characteristics of data or survey
quality required by HUD through statute, regulations, or guidance for data
submissions.
We conducted our work in Washington, D.C., between May 2004 and February
2005 in accordance with generally accepted government auditing standards.
According to HUD, the typical process to estimate FMRs includes
Results in Brief
developing baseline rents from what it judges to be the best rent data
available for each area, adjusting those rents to bring them up to date,
and seeking public comment on its estimates prior to publishing them for
public housing agencies and others to use. Once HUD determines the FMR
areas, it uses decennial census housing data when they are first released
to establish baseline rent estimates, or benchmarks, for each. For
subsequent years, HUD uses data from other surveys-either the AHS or RDD
surveys-to establish a new baseline, or to "rebenchmark" FMRs for certain
areas. To compensate for the time lag between when data are collected and
when HUD first uses them, HUD annually adjusts its baseline estimates in
two ways. First, HUD updates the estimates to December 31 of the current
fiscal year using annual percentage changes in rent and utility costs from
the local Consumer Price Index for major metropolitan areas, or similar
information from RDD surveys for other areas. Second, to make FMRs
relevant for the fiscal year in which they will be in effect, HUD trends,
or projects, the updated figure to the midpoint of the next fiscal year by
applying a national estimate of annual rent increases between the censuses
from the decennial census data. After making these adjustments, HUD
publishes the proposed FMRs in the Federal Register for public comment.
Although HUD considers all of the comments it receives, it typically
changes the proposed FMRs only if the comments are supported by data that
meet HUD's standards. After the period of 60 days to comment on the
Federal Register ends, HUD still considers other requests and submissions
throughout the year.
Over two-thirds of FMRs that HUD estimated for fiscal year 2000, as well
as those it estimated for areas rebenchmarked after 2000, were within 10
percent of the rents indicated by a subsequent quality survey, such the
AHS. For example, when we compared the fiscal year 2000 FMRs (which HUD
estimated in 1999) with rents from the 2000 census data that were
collected during the same period the FMRs were in effect, 69 percent of
all of HUD's FMR area estimates were within 10 percent of the census
figure-an improvement over HUD's 1990 estimates, when 39 percent of areas
were within 10 percent of the 1990 census. When we compared a limited
number of FMRs that HUD estimated after 2000 with rents indicated by data
from the AHS or RDD surveys that HUD or public housing agencies (PHA)
subsequently conducted, a similar proportion of FMRs (73 percent) fell in
the most accurate range. While our associative analysis did not
demonstrate what factors definitively cause accuracy or how much each
contributes, it did show that when HUD used more recent, relevant data
taken from a higher quality survey than some HUD used to rebenchmark in
the past, FMR estimates were more often within 10 percent of the rents
derived from a rebenchmarking survey. For example, FMR estimates from
areas based on more recent survey data-within 1 to 4 years-produced a
significantly higher proportion of FMRs that were within 10 percent of
rents derived from the census than FMR estimates from areas surveyed less
recently.
The ACS could improve the accuracy of FMR estimates because it is a higher
quality survey than some HUD has used in the past and provides more recent
and local data than are currently available-beginning in fiscal year 2006
when HUD first uses ACS data to update FMRs, and subsequently in fiscal
year 2008 when it will likely rebenchmark FMRs in certain areas. HUD will
be able to use ACS data to rebenchmark FMRs annually (or every 3 or 5
years for areas with smaller populations), doing so in generally the same
way it used the decennial census to estimate baseline rents. Certain
challenges related to the manner in which ACS data are processed and
reported may affect FMR accuracy. For example, ACS data are averages of
monthly survey data, which may "smooth" rental market shifts or trends.
According to HUD officials, they will begin to address these challenges
when the Census Bureau releases the fiscal year 2005 data (in Fall 2006),
the data collected during the first year of full implementation for the
ACS. Despite the challenges in using the data, neither we nor experts and
researchers who routinely work with housing data sources identified viable
alternatives to the ACS.
Potential exists for HUD to improve its estimation process for FMRs and
their accuracy because the agency (1) presently does not follow its
objectivity guideline for ensuring the transparency and reproducibility of
its FMR estimates and (2) may in the future lack a way to assess the
accuracy of ACS-based rent estimates. HUD, like other federal agencies,
has developed guidelines to ensure that it disseminates quality data.
HUD's guidelines include ensuring the utility (usefulness), integrity
(protection from unauthorized access), and objectivity (transparency and
reproducibility) of data. Of the three, HUD appears to be following the
utility and integrity guidelines as they relate to the FMR estimation
process. For example, HUD meets its utility guidelines by estimating FMRs
on an annual schedule and making the estimates public and easily
accessible. HUD does not follow one of these three-its objectivity
guideline- because it has made neither the data it uses nor its methods
for estimating FMRs sufficiently transparent for an independent party,
such as GAO, to be able to substantially reproduce FMRs using publicly
available information. Finally, as HUD transitions to ACS-based FMRs, it
will not only stop using the decennial census long form but it will rely
less on RDD surveys and the AHS because of cost and quality concerns about
these surveys. As a result, HUD may not have a means to assess the
accuracy of future FMR estimates once it relies almost exclusively on the
ACS.
This report contains recommendations designed to improve HUD's processes
for estimating FMRs and their accuracy. We provided HUD with a draft of
this report for its review and comment. HUD agreed that it can better
document its methods for estimating FMRs and described efforts it has
under way to improve the transparency and reproducibility of its methods.
HUD also requested that we clarify certain transparency and
reproducibility issues in our report and recognize its ongoing efforts.
HUD disagreed with our recommendation to use state-level ACS data in
fiscal year 2006 FMRs, stating that it has concerns about the adequacy of
ACS sample sizes for the fiscal year 2006 estimates. We have retained this
Background
recommendation because it contains a caution that HUD should do so as much
as possible, but only in instances where HUD determines that the ACS data
are sufficiently reliable for this purpose. HUD did not explicitly state
whether it agrees or disagrees with our recommendation that it develop a
mechanism to assess the accuracy of future FMRs, but it did indicate that
it recognizes there are areas, such as those with unusual rent increases
or decreases, that could experience FMR estimation errors when HUD uses
ACS data for its estimates. HUD also indicated that it anticipates
continuing to review AHS surveys and making limited use of RDD surveys
while it explores other long-term alternatives for assessing the accuracy
of FMRs. Because HUD recognized the challenge we pointed out relating to
the accuracy of FMRs and stated that it is currently exploring ways to
address this issue, we have retained our recommendation. HUD also
suggested a number of technical clarifications to our report, which we
have made, as appropriate.
HUD estimates FMRs in order to set upper and lower bounds on the cost and
quality of typical, standard quality units voucher holders rent and, in
doing so, ensure that the units rented are modest (not luxurious), meet
the housing quality standards HUD sets for them, and are available in
sufficient numbers to those seeking to use the vouchers. Local PHAs use
FMRs to set payment standards, which are the basis for determining the
subsidies HUD provides to help low-income families afford housing in the
private rental market under the Housing Choice Voucher Program.
Specifically, PHAs may set payment standards at 90 to 110 percent of the
FMR for their area and, with HUD approval, above 110 percent of the FMR.
Because HUD generally requires voucher holders to contribute 30 percent of
their income as rent, the amount of HUD's subsidy (the rental assistance)
then becomes the difference between the PHA's payment standard and 30
percent of the family's monthly income.2
While FMRs are primarily used in the Housing Choice Voucher Program, other
programs both inside and outside of HUD also use FMRs. For example, HUD
uses FMRs to
2When households rent units for less than the payment standard, the HUD
subsidy is the difference between their gross rent and their income
contribution.
Page 6 GAO-05-342 Fair Market Rents
o determine initial rents for housing assistance payments in the
Moderate Rehabilitation Single-Room Occupancy program;3
o determine initial renewal rents for units in some expiring
project-based "Section 8" contracts under the Mark-to-Market Program;4
o set maximum rents under the HOME Program;5
o set standard rent ceilings in the Housing Opportunities for Persons
with AIDS (HOPWA) Program;6
o make calculations for the "difficult development" areas under the Low
Income Housing Tax Credit (LIHTC) Program;7 and
o review the feasibility of proposed LIHTC projects.
The Department of Defense (DOD) compares its basic allowance for housing
(BAH) amounts, which is housing assistance it provides military personnel,
to HUD's FMRs. More specifically, when DOD determines that it is not
cost-effective to collect proprietary survey data on housing costs, it
uses FMRs as a basis for calculating comparable figures.
3The Moderate Rehabilitation Single-Room Occupancy (SRO) program provides
rental assistance for homeless persons in connection with the
rehabilitation of SRO dwellings.
4HUD's Mark-to-Market Program reduces rents to market levels for expiring
housing subsidy contracts and restructures existing debt to levels
supportable by these rents on thousands of privately owned multifamily
properties with federally insured mortgages.
5HUD's HOME Program helps to expand the supply of decent, affordable
housing for low- and very-low-income families by providing grants to
states and local governments to fund housing programs that meet local
needs and priorities.
6HOPWA addresses the specific needs of persons living with HIV/AIDS and
their families by making grants to local communities, states, and
nonprofit organizations for purposes such as facility operations or rental
assistance.
7The LIHTC Program is an indirect federal subsidy used to increase the
supply of affordable housing in communities by financing the development
of affordable rental housing for low-income households. Difficult
development areas are designated by the Secretary of HUD as areas that
have high construction, land, and utility costs relative to the area
median gross income.
Whatever its programmatic use, an FMR must fall within certain statutory
and regulatory parameters. The U.S. Housing Act of 1937, as amended,
requires HUD to base FMRs on the most recent available data to estimate
rents of various sizes and types within a market.8 HUD regulations and
guidance on FMRs further emphasize that rent survey data must be the most
accurate and current available.9 HUD specifically requires that the survey
methodology provide statistically reliable, unbiased estimates of gross
rents by, among other things, having a large enough sample so that there
is a 95 percent likelihood that the survey's estimates will be within 5 to
10 percent of what would be found if the entire population (i.e., all
rents) were collected. HUD also requires that survey samples be random and
reflect rent levels that exist for housing units of different ages, types,
and geographic locations within the entire FMR area. Using these
considerations, HUD's three primary data sources for FMRs are the
decennial census (long form), the AHS, and RDD surveys. A RDD survey is a
computer-aided telephone survey of randomly selected households that may
be conducted by HUD, individual PHAs, or others.
Finally, FMRs are specifically defined as annual estimates of the 40th
percentile of gross rents for typical, nonsubstandard market-rate rental
units occupied by recent movers.10
Fortieth Percentile of Rents The 40th percentile is the point in a
distribution of numbers at which 40 percent of the numbers are at or below
that point; for FMR purposes, this is the dollar amount below which 40
percent of the standard quality rental units in an area have rented. For
example, in the distribution in figure 1, $670 is the 40th percentile
because 4 of the 10 rents are at or below that point:
842 U.S.C. S: 1437f(c)(1).
9See 24 C.F.R. S: 888.113 for regulations governing the FMR methodology.
10Beginning in 2001, HUD set FMRs for 39 metropolitan areas at the 50th
percentile, because it determined that an FMR increase was needed to
promote residential choice, help families move closer to areas of job
growth, and alleviate concentrations of poverty.
Page 8 GAO-05-342 Fair Market Rents
Figure 1: Example of 40th Percentile of Rent
Source: GAO.
Gross Rent A gross rent is the rent a tenant pays to the owner-sometimes
called "shelter" costs-plus the cost of utilities (usually, electricity,
gas, water and sewer, and trash removal charges, but not telephone
service). If utilities are included in the rent, then the gross rent is
simply the amount paid to the owner.
Typical, Standard Rental Units
By statute, FMRs are estimates of market rents for typical, standard
quality housing. HUD has determined that certain rental units should be
excluded from its data sources in order to meet this definition.
Specifically, these include rents for units built within the last 2 years
(which tend to be higher priced); units receiving some form of subsidy
(such as public housing) where the rent does not reflect a "market" price;
and substandard units- for example, units without adequate heating or
plumbing-that likely would not meet the housing quality standards
applicable to the voucher
11
program.
Recent Movers HUD has found that rents for units occupied by recent movers
(i.e., tenants who moved within the past 15 to 24 months) are typically
higher than what other renters pay. By linking FMR estimates to the rents
that recent movers have paid, HUD tries to ensure that they more closely
reflect the rents that low-income households new to the voucher program
may face when they look for rental housing.
11Rents for units on 10 or more acres and seasonal units, such as summer
rentals, are ineligible for the FMR estimation process.
Page 9 GAO-05-342 Fair Market Rents
HUD Estimates FMRs by Defining Housing Markets, Choosing Data Sources,
Updating Rent Data, and Evaluating Public Input
The Census Bureau is discontinuing the long form and has begun replacing
it with the ACS.12 Overall, the ACS will provide the same type of data as
the decennial census long form at the same level of geographic area
detail, but in a more timely way because it will be an ongoing survey (as
opposed to one conducted every 10 years). Specifically, the ACS will
collect data monthly and each year publish either 1-, 3-, or 5-year
averages (depending on the population in each area).13
According to HUD, the typical process it uses to estimate FMRs (rent
estimates that include utilities) includes choosing what it judges to be
the best rent data available for each area, adjusting those data so that
they are up to date, and seeking public comment on the estimates prior to
finalizing them for public housing agencies and others to use (see fig.
2). Once HUD determines each FMR area and receives decennial census data
or AHS or RDD data, it analyzes the rent data to establish a "benchmark"
FMR for each area by determining the 40th percentile of the rent
distribution. Then, HUD annually adjusts the estimates to reflect changes
in rent and utility costs to compensate for the lag between data
collection and the period in which the FMR will be in effect. After
adjusting the FMR for each area, HUD publishes the proposed FMRs for
public comment. Although HUD considers all of the comments it receives, it
typically changes FMRs only if the comments are supported with data that
meet HUD's standards. The public can also affect FMRs by (1) requesting
that HUD conduct an RDD for the area or (2) submitting comments with
supporting rent data or information that causes HUD to conduct additional
research.
12The ACS is subject to annual appropriations. Funding for the ACS to
cover all persons except those living in group quarters (e.g., college
dormitories and prisons) was approved beginning with fiscal year 2005.
Funding to cover all persons has been requested beginning with 2006.
13The first annual ACS data for geographic areas with populations larger
than 65,000 will be published beginning in 2006; publication of 3-year
averages for areas with populations of 20,000 to 65,000 will begin in
2008; and publication of 5-year averages for areas with less than 20,000
will begin in 2010.
[This page intentionally left blank]
Figure 2: HUD's Typical Process for Estimating FMRs
Source: GAO analysis of HUD documents.
HUD Establishes Areas, Uses Survey Data to Benchmark and Adjust FMRs
To ensure that the FMR estimates are useful, HUD's first step is to
determine FMR areas that they believe correlate with distinct housing
markets, typically the size of a county (see fig. 2). To determine FMR
areas, HUD generally uses the boundaries of Office of Management and
Budget (OMB)-defined metropolitan and nonmetropolitan areas.14 According
to HUD, it may also create new areas that do not correspond to OMB
boundaries, particularly within sprawling metropolitan areas that may have
separate housing markets. For instance, HUD created a separate FMR area
for West Virginia counties that had been included in OMB's Washington,
D.C., metropolitan area, because HUD did not consider these counties to be
part of the Washington housing market. Although HUD may revise FMR area
definitions at any time, it typically does so infrequently (not every year
when it develops FMRs).15 HUD publishes FMR estimates annually for 356
metropolitan FMR areas and 2,303 nonmetropolitan FMR areas in the United
States, Puerto Rico, the Virgin Islands, and Guam.
HUD's second step is to benchmark-that is, estimate baseline rents-for
two-bedroom units by identifying the 40th percentile of the estimated rent
distribution for each area with the most recent available data (for FMR
areas for which no new, recent rent data are available, HUD skips this
step and updates the existing FMR). HUD chooses from a variety of data for
benchmarking, including the decennial census, the AHS, RDD surveys, and
traditional surveys from the public. According to HUD officials:
o The decennial census provides the highest quality data to estimate
FMRs because it provides (1) rent estimates within 1 percent of the
true value of the 40th percentile of rents in metropolitan areas and
(2) the most consistent data for all areas to establish a baseline for
FMRs once every 10 years.
o Data from RDD surveys have sufficient quality to meet HUD's
requirements and provide estimates within 3.5 to 5 percent of the true
value of rents for a limited number of areas, usually metropolitan
areas.
14According to OMB, a metropolitan area generally consists of a core area
containing a substantial population nucleus, and adjacent communities
exhibiting a high degree of economic and social integration with the core.
15In 1994, we reported on a proposal to establish smaller FMR areas. See
GAO, Rental Housing: Use of Smaller Market Areas to Set Rent Subsidy
Levels Has Drawbacks, RCED-94-112 (Washington, D.C.: June 24, 1994).
Page 14 GAO-05-342 Fair Market Rents
* The AHS offers sufficient quality data with estimates within 7
percent of the true value of rents the survey is measuring and are
available for a limited number of metropolitan areas every few
years.
* According to HUD officials, to be consistent with the definition of
FMRs, HUD only uses survey data for rental housing units that are
o nonsubsidized and of "standard" quality;16
o more than 2 years old;
o nonseasonal (i.e., occupied year round);
o located on properties of less than 10 acres; and
o leased by recent movers (those who have moved within the last 15 to 24
months).
HUD adds estimated utility costs to the base rent estimates it derives
from RDDs because these surveys do not include that information. To do so,
HUD officials estimate the cost of utilities with PHA utility schedules,
which include a list of average monthly costs for each utility. The
decennial census and AHS data include utilities in their base year rent
estimates.
The third and fourth steps in the process involve adjusting FMRs. To
mitigate the time lag between data collection and FMR use, HUD first
updates FMRs to December 31 of the current fiscal year with information
about changes in the rent and utility index from the Consumer Price Index
(CPI) program for specific metropolitan areas or, for other metropolitan
and all nonmetropolitan areas, with the gross rent "change factors"
established by regional RDD surveys. To estimate the gross rent change
16Prior to 2005, HUD used information on unit quality and assistance from
the AHS to generate a proxy for subsidized (public and assisted) and
substandard housing. This adjustment was constant over the nation and did
not vary by bedroom size. To estimate fiscal year 2005 FMRs, HUD used ACS
and HUD administrative data to calculate a substandard housing adjustment
that is tailored to region and bedroom sizes. Specifically, HUD began to
use the 75th percentile of public housing rents from its administrative
data for each of its regions as a proxy to indicate which units are
subsidized and substandard. According to HUD, this new proxy allows for
larger adjustments in areas with more public and assisted housing units
and higher housing quality issues. HUD continues to use information from
RDD surveys and the AHS to eliminate subsidized and substandard units from
survey data.
factor, or the measure of rent change, HUD conducts regionwide RDD surveys
in each of its 10 multistate regions (see fig. 3).
Figure 3: HUD Regions
Source: HUD.
Once FMRs are updated, HUD attempts to make them useful for the fiscal
year in which they will be in effect by trending, or projecting, them to
the midpoint of that fiscal year. To do this, HUD uses a national measure
of annual rent increases (i.e., average rent increases during the 10 years
between the censuses, typically 3 percent, on the basis of decennial
census rent data).
HUD Provides Opportunities for Public Input on Proposed FMRs as Well as
Those in Effect
In the fifth step, HUD also estimates FMRs for other bedroom sizes (in
practice, one-, three-, and four-bedroom). Because HUD usually lacks
sufficient survey data to directly estimate FMRs for all unit sizes, it
typically benchmarks FMRs for two-bedroom units only and estimates rent
ratios for other sizes.17 According to HUD officials, these ratios are
based on local rent relationships derived from decennial census rent data.
Once HUD calculates these ratios, it ensures that they are "sequential,"
which means that FMRs increase as unit size increases (e.g., in 1994,
three-bedroom FMRs had to be at least 125 percent of two-bedroom FMRs, and
four-bedroom FMRs had to be at least 140 percent of two-bedroom FMRs).
After HUD estimates FMRs for each bedroom size unit, it applies a "bonus"
to increase FMRs for larger units (three-bedrooms or larger) to help
ensure that the units can be rented by voucher holders.
To provide for public input on proposed FMRs:
o HUD publishes the proposed FMRs in the Federal Register to solicit
public comments, usually in April or May of each year (sixth process
step).
o The public submits comments during the (approximate) 60-day public
comment period.
o After the comment period, HUD reviews the responses received and may
act on some of them prior to finalizing FMRs and publishing them again
in final form in the Federal Register in September (seventh process
step). FMRs are in effect for the next fiscal year, which starts
October 1.
After the period of 60 days to comment on the Federal Register ends, to
address situations in which existing FMRs are perceived to be inaccurate,
members of the public-often, PHAs-also can submit information on the
existing FMR for HUD to consider. For example, PHAs can at any time
conduct and submit to HUD the results of their own RDD surveys; HUD
applies the same criteria to these surveys as it does to those that PHAs
17The 2000 decennial census produced data sufficient to allow HUD to
directly estimate FMRs for all bedroom sizes for fiscal year 2005 FMRs and
update the bedroom ratios. According to HUD officials, they will use these
new ratios to estimate future non-two-bedroom FMRs.
Page 17 GAO-05-342 Fair Market Rents
submit in response to the proposed FMRs in the Federal Register.
Specifically, HUD requires that any PHA-submitted data it uses to change
FMRs must be statistically reliable; unbiased estimates of gross rents;
and, among other things, have a large enough sample that there is a 95
percent likelihood that the survey's estimates will be within 10 percent
of what would be found if the entire population (i.e., all rents) were
collected.18 Also, PHAs may at any time outside of the formal comment
process request that HUD conduct an RDD survey or submit information about
the existing FMR that may cause HUD to conduct additional research.
While the Quality Housing and Work Responsibility Act of 1998 gave PHAs
the flexibility to set payment standards at 90 to 110 percent of their
FMRs, they may also request an exception to further adjust either the
payment standard or the FMR for their area. Specifically, when PHAs
believe that payment standards at 110 percent of the FMR are insufficient
to allow voucher holders to successfully lease units, they may request
from HUD one of two possible exceptions: (1) increase the payment standard
to exceed the FMR by more than 10 percent or (2) benchmark the FMR
estimate at the 50th percentile of rent for the area, rather than the 40th
percentile of rent.19
18In very limited instances, HUD officials will accept data from PHAs in
areas with small populations that have not followed the requirements.
According to HUD officials, some areas with small populations will not be
able to comply due to limited budgets or small sample sizes within the FMR
area. HUD officials then evaluate the data on the basis of their
professional judgment.
19In order to obtain an exception to increase the payment standard by more
than 10 percent, the public must submit documentation that demonstrates
approval of the special exception is necessary to prevent financial
hardship for families in the exception area. This documentation can
include census rent data, locally funded quality surveys, lease rates, and
success rates. The request must be needed (1) to enable families to find
housing outside areas of high poverty and (2) because voucher holders have
trouble finding housing for lease.
Most FMR Estimates Were Accurate within 10 Percent of the Census or Other
Rebenchmarking Surveys
According to our analysis, more than two-thirds of (1) all FMRs that HUD
estimated for fiscal year 2000 and (2) a limited number of FMRs that HUD
rebenchmarked after 2000 were within 10 percent of the rents derived from
subsequent surveys such as the census, the AHS, or an RDD survey.
Specifically, 69 percent of all of HUD's FMR estimates for fiscal year
2000- published in 1999-were within 10 percent (the most accurate range)
of rent estimates derived from the 2000 census. Moreover, when we
considered FMRs by type of area, FMR estimates for 86 percent of
metropolitan areas and 66 percent of nonmetropolitan areas fell in the
most accurate range in 2000. Similarly, when we compared rents derived
from rebenchmarking surveys done for 153 FMR areas since 2000 with the FMR
estimates in place at the time of the rebenchmarking survey, 73 percent of
the estimates were within 10 percent of the rents derived from the
surveys. FMR estimates were more often associated with accuracy when HUD
based them on data that were more recent, taken from a higher quality
survey than some HUD has used in the past, or more relevant because the
survey covered an area closely matching the boundaries of the FMR area.20
Other factors not related to the specific survey HUD used to estimate
FMRs, such as difficulty in estimating utility costs, may also affect the
accuracy of FMR estimates.
Over Two-thirds of All FMRs According to our analysis, for fiscal year
2000, 69 percent of FMRs that HUD estimated for fiscal year 2000 were
within 10 percent of the 2000
for 2000 Were Accurate
within 10 Percent of Rents census rent estimates, the most accurate
comparison data available for Derived from the 2000 each FMR area (see
fig. 4).
Census
20We use "associated with accuracy" because our analysis does not enable
us to make causal links between survey or FMR area characteristics and the
accuracy of estimates.
Page 19 GAO-05-342 Fair Market Rents
$550 (new survey estimate) - $500 (existing FMR) = $50 difference
$50 difference/$550 (new survey estimate) = 9 percent
In this example, the original FMR was lower than the rent indicated by the
recent survey, but was within 10 percent.
The results for 2000 are a significant improvement over results from 1990
when HUD reported that 39 percent of FMRs were in the most accurate range
(see table 1). Furthermore, arraying the data by population to account for
areas where estimates affected more potential voucher holders shows that a
greater share of FMR estimates were within the most accurate range in 2000
than what HUD reported for 1990. Considering FMR estimates by type of
area, we also found that more metropolitan and nonmetropolitan areas were
within 10 percent accuracy in 2000 than HUD reported in 1990.21
Table 1: Accuracy of HUD's Fiscal Years 2000 and 1990 FMR Estimates Compared
with Rents from Census
Compared with decennial census rents-percentage of FMRs that were:
Higher by Higher by Lower by Lower by
Fiscal year 20% or more 10% to Within 10% to 20% or
19.9% 10% 19.9% more
2000 2% 8% 69% 19% 2%
1990 25 30 39 4 2
Weighted by population
2000 2 4 88 6 1
1990 5 10 73 12 1
Metropolitan areas
2000 3 6 86 4 1
1990 8 14 71 8 0
Nonmetropolitan areas
2000 2 8 66 21 3
1990 29 31 34 4 2
Sources: GAO analysis of HUD data (2000 figures) and HUD (1990 figures).
21Most households receiving tenant-based vouchers-85 percent-live in
metropolitan areas.
Page 21 GAO-05-342 Fair Market Rents
As table 1 shows, our analysis indicates that in 2000, where FMR estimates
were higher or lower than the census by 10 percent or more, most often the
FMR was too low, a different result from 1990 when HUD reported that most
FMR estimates outside of the most accurate range were too high.
Since the 2000 Census, HUD and others surveyed a limited number of FMR
areas (153, as of September 2004). When we compared the rents derived from
these surveys with FMR estimates in effect for these years, the outcome
was similar to the results we found in our comparison with the 2000
census-almost three-fourths (73 percent) of FMR estimates were within 10
percent of the survey rents. When analyzing the 153 areas, we also found a
difference in the results shown by rebenchmarking surveys undertaken for
different reasons. HUD and PHAs conducted rebenchmarking surveys for two
basic reasons: (1) HUD was adhering to a schedule in which it surveyed
selected large metropolitan areas on a rotational basis or (2) HUD, PHAs,
or others received information suggesting FMRs were inaccurate (usually a
complaint that an FMR was too low) in a specific area. As shown in table
2, complaint-driven surveys (RDD surveys that were conducted by HUD
following a PHA request or by a PHA itself) more often found inaccurate
FMRs (i.e., FMR estimates that were 10 percent or more different from the
rents derived from the survey).
Table 2: Accuracy of HUD's FMR Estimates Compared with Rents from RDD Surveys
(by Reason for Survey, 2001-05)
Compared with RDD survey rents-percentage of FMRs that were:
Higher by Higher by Lower by Lower by
Reason for survey 20% or 10% to 19.9% Within 10% 10% to 20% or more
more 19.9%
HUD schedule 0% 3% 87% 9% 1%
By request (HUD 2 2 66 26 5
surveyed)
PHA surveyed 0 0 53 42 5
Source: GAO analysis of HUD data.
Note: HUD estimated FMRs we used in this comparison prior to the public
comment step that takes place after its estimation process. When HUD
received the results of RDD surveys prior to the public comment step, it
used (and published) those rent estimates rather than the initial FMR
estimate it had developed. As a result, some of the estimates we use in
this comparison were never published by HUD as proposed FMRs.
According to HUD, those areas it surveyed because they were on its
schedule were, like the complaint-driven RDD surveys, not random
selections. Most often, HUD selected areas from its schedule because it
had not surveyed them recently, which means that HUD tended to choose
areas for which the length of time since the last rebenchmarking survey
was longer. According to HUD officials, choosing areas for RDD surveys for
this reason increases the likelihood that these surveys would find
inaccurate FMRs. Further, to the extent that complaints are more likely to
arise when FMRs are believed to be too low, rather than too high, it is
not surprising that complaint-driven surveys were much more likely to show
rents higher than FMRs, rather than lower.
Quality Survey Data Tended Survey data that had one or all of the
characteristics we summarize as "quality"-recent, accurate, or
relevant-tended to more often produce
to Produce FMR Estimates
That Were Accurate within FMRs within 10 percent of another rebenchmarking
survey. Specifically, our analysis showed that FMR estimates more often
fell in the most
10 Percent accurate range when HUD based FMRs on survey data that were (1)
more recent, (2) taken from a higher quality survey than some surveys HUD
has used in the past, or (3) more relevant because their source closely
matches the boundaries of the FMR area.
More Recent Data FMR estimates that HUD rebenchmarked with newer survey
data (1 to 4 years old) were associated with greater accuracy in 2000 (see
table 3). For example, our analysis found that 88 percent of all FMR
estimates based on newer data (i.e., 1 to 4 years old) were within 10
percent of the census estimates in 2000.
Table 3: Accuracy of FMR Estimates in 2000 Compared with Rents from Census
(Based on Age of Baseline FMR Data)
Compared with decennial census rents-percentage of FMRs that were:
Higher by Higher by Lower by Lower by Age of baseline FMR data 20% or more
10% to 19.9% Within 10% 10% to 19.9% 20% or more
1 to 4 years 1% 6% 88% 5% 0%
5 to 7 years 3 9 67 20
No survey from 1990 to 2000 2 7 67 20
Source: GAO analysis of HUD data.
Note: Areas based on 2000 decennial census data or 8-, 9-, or 10-year-old
non-Census data comprised too few areas from which to calculate separate
statistics.
In considering the association we found between recent data and accuracy,
HUD officials stated that the length of time since the last rebenchmarking
survey likely affected the accuracy of FMR estimates. As our analysis
showed, areas for which the baseline data were older (including those for
Data from Higher Quality Surveys
which there was no rebenchmarking survey between the 1990 and 2000
censuses) more often had FMR estimates that were 10 percent or more higher
or lower than the estimate from a recent survey.
When HUD used data from higher quality surveys than some surveys it had
used in the past, its FMR estimates were accurate more often than when it
relied on lesser-quality means, such as the traditional surveys some PHAs
conducted before HUD adopted the RDD survey methodology. Currently, HUD
uses the AHS or RDD surveys to rebenchmark FMRs between the decennial
censuses. However, until the mid-1990s, HUD also, on occasion, accepted
from PHAs and used for rebenchmarking FMRs survey data that PHAs collected
via less rigorous traditional or telephone surveys.22 The AHS and RDD
surveys can be considered higher quality than the less rigorous ones HUD
once accepted because they have (1) survey characteristics required by
HUD's regulations and guidelines and (2) data from a survey closely
corresponding to the boundaries of the FMR areas. As shown in table 4, the
higher quality survey sources-AHS and RDD surveys-more often led to FMRs
within 10 percent accuracy than the estimates based on less rigorous
methods.
Table 4: Accuracy of FMR Estimates in 2000 Compared with Rents from Census
(Based on Type of Rebenchmarking Survey)
Compared with decennial census rents-percentage of FMRs that were:
Type of last FMR Higher by Higher by Lower by Lower by
rebenchmarking
survey 20% or more 10% to Within 10% 10% to 20% or
19.9% 19.9% more
AHS 5% 0% 95% 0% 0%
RDD-HUD 2 7 86 5 0
RDD-PHA 3 22 72 4 0
Traditional 2 6 65 26 1
Telephone 11 28 61 0 0
No Survey from 1990 2 7 67 20 3
to 2000
Source: GAO analysis of HUD data.
22Traditional surveys are surveys of rent data in metropolitan areas with
relatively low populations in which a PHA or other entities have access to
all or almost all of the rents in the area-for example, in cities or towns
that require owners to register rents annually and maintain a database of
rents. Telephone surveys are generally derived from randomly selected
lists of residential telephone numbers but are not assisted by the use of
a computer to track telephone calls and the outcomes.
Page 24 GAO-05-342 Fair Market Rents
More Relevant (Local) Data
When HUD used more relevant (local) surveys to update FMRs-that is, to
adjust for inflation rather than to rebenchmark or revise the baseline-the
results were similar: FMR estimates were associated with greater accuracy.
As shown in table 5, when HUD updated FMR estimates with the more local
metro-specific CPI-a survey that generally matches the boundaries of
metropolitan FMR areas-91 percent of estimates were within 10 percent
accuracy. When HUD used regional RDD surveys-which cover much broader
areas than the FMR area boundaries-to update FMR estimates, many fewer
were within 10 percent accuracy.
Table 5: Accuracy of FMR Estimates in 2000 Compared with Rents from Census (by
Type of Update Factor)
Compared with decennial census rents-percentage of FMRs that were:
Higher by Higher by Lower by Lower by Type of update factor 20% or more
10% to 19.9% Within 10% 10% to 19.9% 20% or more
Metro-specific CPI 5% 1% 91% 3% 0%
RDD regional gross rent change factor 2 8 68 19
HUD Believes That Other Factors May Influence the Accuracy of FMR Estimates
Source: GAO analysis of HUD data.
According to HUD officials, the use of broad factors-that is, factors from
surveys covering a larger geographic area than the FMR area-for updating
and trending in the FMR estimation process contributes to inaccuracy in
the estimates. For instance, the update factors derived from regional RDD
surveys may not capture changes in the local economy within a specific FMR
area, such as a large employer leaving town or a sizable increase in the
housing supply that may affect rents. Furthermore, HUD officials stated
that the use of a nationwide factor for trending FMR estimates-the process
of projecting FMR estimates into the future year for which they will be
effective-may not capture local trends. (As previously noted, HUD
currently applies to all FMR areas a standard trending factor derived from
the change in the national average rents between the 1990 and 2000
censuses.)
In addition to the factors we identified as being associated with the
accuracy of FMR estimates, HUD officials indicated several more factors
that might also affect accuracy. Specifically, these officials cited (1)
general survey error common to all such estimates, (2) the characteristics
of nonmetropolitan areas, (3) difficulty in estimating utility costs, and
(4) recent mover rent changes differing from rent changes captured by the
CPI.
Page 25 GAO-05-342 Fair Market Rents
General Survey Error The data from the survey sources HUD uses
are estimates which, by
definition, can introduce error into FMR
estimates. All surveys are subject
to various types of error, which means
that survey data may not precisely
match the true value the survey is trying
to measure. For example,
sampling error occurs because a sample
rather than an entire population
was surveyed, and, according to HUD
officials, census data for FMR
estimates are generally subject to a 1
percent sampling error (in
metropolitan areas). While HUD considers
census data to be the best
source for rent estimates (primarily
because these data have a far larger
sample size than any other source used),
even the census includes some
areas with low sample sizes or low
response rates.
Our analysis showed that FMR estimates
Characteristics of for nonmetropolitan areas were
Nonmetropolitan Areas less likely to be based on quality data
(more recent, taken from a higher
quality survey and more relevant) and
were also less likely to be more
accurate. HUD officials told us that
nonmetropolitan areas are a lower
priority for rebenchmarking surveys
between the censuses because they
believe it is better to focus their
limited resources (for their own
rebenchmarking RDD surveys) on the areas
where more potential voucher
holders live (i.e., the metropolitan
areas). Nonmetropolitan areas were less
likely to have a recent rebenchmarking
survey (sponsored by HUD)-
between 1990 and 2000, HUD rebenchmarked
73 percent of all
metropolitan areas and 31 percent of all
nonmetropolitan areas. Also, HUD
updates almost all nonmetropolitan areas
using the broad update factors it
derives from its regional RDD surveys,
meaning that these areas' FMR
estimates are updated with data that are
less "local" than what HUD applies
to the larger metropolitan areas with
local CPI rent change estimates.
Additionally, surveys of nonmetropolitan
areas (even the census) often
have relatively lower sample sizes than
metropolitan areas, affecting the
quality of the data for rebenchmarking
FMR estimates there and, as a
result, the accuracy of these
estimates.23
According to HUD officials, utility cost
Difficulty in Estimating Utility data are a source of error in all
Costs three survey data sources HUD uses to
estimate FMRs. For example,
renters have been documented as
unreliable sources of the utility costs
23According to HUD, some nonmetropolitan areas have unusually low census
data sample sizes and unusually high levels of substandard and assisted
housing that may distort the accuracy of FMR estimates. For the fiscal
years 1996 through 2004 FMRs, HUD corrected for low FMR estimates that
were at or below the cost of operating housing by implementing a minimum
FMR level for each state.
Page 26 GAO-05-342 Fair Market Rents
Recent Mover Rent Changes in Metropolitan Areas
they pay, yet the census relies on them to report utility cost estimates.
Utility costs for RDD surveys come from a utility cost schedule supplied
by the local PHA; however, according to HUD officials, although PHAs
certify that the data are correct, utility schedules can be unreliable and
introduce bias into FMR estimates.24 The AHS uses a utility estimation
model (consisting of certain survey variables) that HUD officials believe
corrects to some extent for the error introduced by relying on tenant
reporting. Nonetheless, they noted that the AHS model is based on survey
estimates and thus remains subject to error in ways common to all surveys.
HUD officials told us that the local survey HUD uses for updating FMRs in
some metropolitan areas-the metro-specific CPI-may not capture sudden
changes in rents for recent movers. According to HUD, CPIs measure overall
rent changes for all renters in a fixed group of units. However, rent
changes for recent movers can be significantly different from changes for
all renters. For example, HUD officials stated that San Francisco and
Boston are among the more volatile housing markets in the country and, as
a result, among the most difficult for which to estimate FMRs.
Specifically, in 2000 and 2001, San Francisco's recent mover rents
increased sharply, then decreased suddenly in 2002. However, the CPI for
San Francisco, which covers all renters, showed above-average but not
exceptional rent increases in 2000 and 2001 and no change for 2002.
ACS Could Improve the Accuracy of FMRs by Providing HUD with More Recent,
Better Data
The ACS, which is replacing the decennial census long form, could improve
the accuracy of FMRs because it is a higher quality survey (compared with
others HUD has available between the decennial censuses) and it provides
more recent data that closely matches the boundaries of HUD's FMR areas.
HUD plans to begin to use ACS data for fiscal year 2006 FMRs. However,
certain challenges that we and others, including the National Academy of
Sciences (NAS), have identified may affect the extent to which HUD can use
ACS data to improve its estimates. County-level ACS data, which will be
available each year, could increase the accuracy of FMRs because HUD plans
to use them to rebenchmark all areas more frequently. Because the ACS data
are more recent than the decennial census data and generally of similar
quality and content, HUD plans to use ACS data to rebenchmark FMRs in
generally the same way that it used the decennial census data in
24According to HUD, a PHA utility schedule is a list of the average
monthly costs of various types of utilities, such as heating oil,
electricity, or water and sewer charges, subdivided by the number of
bedrooms in the unit.
Page 27 GAO-05-342 Fair Market Rents
the past, but it will be able do so more frequently. Certain challenges
for HUD regarding the ways ACS data are processed and reported may affect
its plans for using them. For example, the Census Bureau averages ACS data
over 1-, 3-, and 5-year time periods, and averaging could mask sharp
trends in rents because it can smooth changes that occur within the time
period. HUD plans to address these challenges after it receives fiscal
year 2005 ACS data-the data collected during the first year of full ACS
implementation-in Fall 2006. Despite the challenges in using these data,
neither we nor experts and researchers who routinely work with housing
data sources identified viable alternatives to the ACS.
The ACS Is a Higher Quality The ACS could improve the accuracy of FMRs
because it is a higher quality survey than HUD currently has available
between the decennial censuses
Survey That Provides More
Recent and Local Data and it provides more recent data closely matching
the boundaries of HUD's FMR areas.
Higher Quality Survey The ACS is of higher quality than data sources (RDD
surveys and the AHS) currently available to estimate FMRs between the
decennial censuses. According to the Census Bureau, like its long-form
predecessor, the ACS is the highest quality household survey currently
conducted by the Census Bureau, and it will provide data more
frequently.25 The ACS derives similar information as the decennial census
long form, and its results undergo stringent processing by the Census
Bureau. Moreover, according to HUD officials, the ACS is an impressive
improvement over data from any other source. For instance, although the
AHS is also a Census Bureau product, it is similar to RDD surveys because
it provides data for only a comparatively small number of areas and does
so less frequently. More specifically, the AHS covers a limited number of
the largest metropolitan areas every few years.
More Recent Data Using ACS data to estimate FMRs could improve their
accuracy because it provides HUD with more recent county-level data. More
specifically, the ACS will provide data each year that is based on 1-, 3-,
or 5-year rolling averages (i.e., the Census Bureau will collect data
monthly, average them over 12 months, and publish new 1-, 3-, and 5-year
averages each year).
25The Census Bureau reports that the 5-year averages will be about as
accurate as the long-form data; the annual and 3-year averages will be
significantly less reliable than the long-form data but more reliable than
existing household surveys the Census Bureau conducts.
Page 28 GAO-05-342 Fair Market Rents
More Local Data
Because our analysis indicated that FMRs estimated with recent data (i.e.,
data that are 4 years old or less) more often tended to be within 10
percent of the results of a rebenchmarking survey, FMRs estimated with
annual and 3-year average data could be more accurate. Even though FMRs
estimated with 5-year average data would be based on some data older than
4 years, they could also be more accurate than is now the case because HUD
could rebenchmark them every 5 years (as opposed to the 10 years between
censuses).
The ACS also will provide more local data-more specifically, state-level
data-that HUD could use to update FMRs and therefore lead to more accurate
FMRs. Currently, HUD updates FMRs for the majority of areas with regional
RDD surveys, each of which provides HUD with aggregate gross rent change
estimates based on data from up to eight states. As previously noted, our
analysis suggested that when HUD estimated FMRs with more local data
(i.e., data from a survey that closely corresponds to the boundaries of
the FMR areas) more FMRs fell within the most accurate range. As a result,
annual state-level ACS data could enable HUD to more accurately update
FMRs. Although the state-level data do not closely correspond to the
boundaries of FMR areas, they cover areas much smaller than the currently
used RDD surveys.
HUD Expects to First Use ACS Data to Update Fiscal Year 2006 FMRs
HUD expects to first use ACS data to update its estimated baseline rents
when preparing fiscal year 2006 FMRs. To do so, HUD plans to use
regional-level ACS data, rather than the more local state-level ACS data
that will be available to it. The state-level ACS data would provide
reliable data for geographic levels smaller than the areas covered by the
regional ACS (or regional RDD surveys). However, according to HUD
officials, they believe they need to obtain and work with the ACS data,
assuring themselves of its reliability and usefulness before they will
consider updating FMRs with the state-level ACS data.
The effect of ACS data on FMR accuracy could be most notable once HUD
begins to rebenchmark-not just update-FMRs with these data, which will
likely begin with the fiscal year 2008 FMRs. HUD will use the first data
available under ACS full implementation in Fall 2006 to rebenchmark fiscal
year 2008 FMRs and plans to use them in ways similar to how it had used
decennial census data because their content and quality are similar to
that of the decennial census data. Figure 5 describes how often HUD could
rebenchmark different-sized areas with ACS data, showing that, for
example, HUD will likely rebenchmark FMRs for large metropolitan areas-where the
most potential voucher holders live-every year.
Figure 5: Scope of ACS Rebenchmarking as Related to FMR Area Size and Housing
Choice Voucher Program Data
Source: GAO analysis of HUD data.
Note: The most recent available data for population and number of housing
choice vouchers per FMR area are from fiscal years 2000 and 2003,
respectively. We estimated the "voucher dollar" to approximate the
relative dollar amounts of housing choice vouchers in each area. To do so,
we multiplied the FMR (FY 2004) and the number of vouchers for each FMR
area over 12 months.
Because data developed from a single year of ACS data will be based on
samples that are approximately one-sixth as large as decennial census
long-form samples, HUD may need more data points than what the ACS will
provide for communities with smaller populations in order to estimate
FMRs. More specifically, according to HUD officials, to obtain a
sufficient sample of rent data for HUD's program purposes, the agency
needs data from areas with larger populations-that is, areas that can
provide more data points-than the ACS will publicly report. For instance,
in an annual ACS sample from a metropolitan area with a population of
100,000, HUD could expect to find in ACS data only 48 recent movers in
two-bedroom rental units, but it needs 200 recent movers for its purposes.
In order for HUD to obtain its needed minimum sample of 200 units, it will
likely need to use 1-year average data for counties with populations of
more than 400,000; 3-year average county-level data for areas with
populations of 133,000 to 400,000; and 5-year average county-level data
for areas with populations of less than 133,000.
In addition, although the Census Bureau will publish 3- and 5-year rolling
average ACS data every year beginning in 2008 and 2011, respectively, HUD
may not use these data every year because of concerns about their
reliability for HUD's FMR estimation purposes. According to the Census
Bureau, reliable measures of changes in multiyear averages-such as what
HUD needs in order to estimate FMRs-should only be calculated using
averages with no overlapping years. The 3- and 5-year rolling average ACS
data that the Census Bureau publishes every year will have overlapping
years. For example, in 2008, the Census Bureau will publish 3-year average
ACS data covering 2005, 2006, and 2007; in 2009, it will publish 3-year
average ACS data for 2006, 2007, and 2008, overlapping the previous year's
estimate by including 2006 and 2007 data. For HUD's purposes, a reliable
time series of 3-year averages would consist of the ACS data that the
Census Bureau will publish in 2008 (2005-07 averages), 2011 (2008-10
averages), 2014 (2011-13 averages), and so on because these would not have
overlapping years.
ACS Data Pose Certain HUD's consultant ORC Macro, NAS, the Census Bureau,
and we have identified certain challenges associated with using ACS data
that may
Challenges to HUD That
May Affect FMR Estimation affect how and when HUD could use the data and
improve the accuracy of FMRs. The challenges include issues related to the
averaging of the ACS
and Accuracy data, presentation of inflation-adjusted costs (such as
rents), techniques to deal with missing responses, and reporting
differences between the decennial census and the ACS.
Averaging The Census Bureau collects data for the ACS monthly and
continuously averages them over 1-, 3-, and 5-year time periods. However,
this averaging could hide rental market shifts because moving averages
tend to "smooth" changes in data over time.26 For example, if from January
through September of a given year the rent for an area is $800, and from
October through December of the same year the rent is $1,200, the average
annual rent reported by the ACS would be $900, which is far less than the
current monthly rent of $1,200. As a result, the moving averages'
"smoothing" effect may hide a turning point, or, current prices in the
rental housing market.
26See ORC Macro, The American Community Survey: Challenges and
Opportunities for HUD (Calverton, MD: Sept. 27, 2002). ORC Macro is the
consultant HUD hired.
Page 31 GAO-05-342 Fair Market Rents
Inflation-Adjusted Costs To adjust for general inflation, the
Census Bureau will use a general
adjustment factor rather than an index
that is specifically related to data
items, such as rents or utilities, to
present dollar-denominated data from
the ACS. This could limit the usefulness
of the data for HUD's program
purposes because using a general
adjustment factor (i.e., national CPI)
rather than using an index that is
specifically related to the dollar-
denominated item (i.e., a rent index)
could result in a less-precise
estimate.27 The treatment of
dollar-denominated data is critical to all
users
of these data, and particularly to HUD,
which will be using the ACS to
determine FMRs based on rent data. If HUD
had access to the Census
Bureau's unadjusted annual data, it could
then adjust the data pertinent to
its FMR estimation using rent or utility
indexes. We previously raised
concerns about the Census Bureau inflation
adjustment. 28 In response, the
Census Bureau did not provide a rationale
for using the general adjustment
factor, rather than a more specific index,
but did indicate that the bureau
would reconsider its present policy of
showing only the inflation-adjusted
annual estimates.
A NAS panel and we have raised concerns
Techniques to Deal with Missing about how imputation-a
Responses technique used to deal with surveys with
missing responses-could affect
the accuracy of ACS data, especially in
smaller areas. The NAS panel that
reviewed the 2000 Census raised issues
about the potential effects of
imputation on ACS results. Unlike the
process used for the decennial
census-100 percent follow-up for all
nonrespondents-the Census Bureau
conducts follow-up on only 33 percent of
nonrespondents to the ACS. The
Census Bureau uses the responses from the
follow-up surveys to attribute a
similar pattern of responses to the
remaining 66 percent of
nonrespondents. The NAS panel called on
the Census Bureau to analyze
the associated trade-offs in costs and
accuracy between imputation and
additional fieldwork to gather more data.
29
In a 2004 study, the Census Bureau found
Reporting Differences between that when the decennial census
the Decennial Census Long Form long form and the ACS were used to survey
the same area, they reported a
and the ACS number of variables differently, including
those HUD uses to estimate
27See GAO, American Community Survey: Key Unresolved Issues, GAO-05-82
(Washington, D.C.: Oct. 8, 2004). 28See GAO-05-82. 29See GAO-05-82.
Page 32 GAO-05-342 Fair Market Rents
Despite Challenges, the ACS Remains Likely the Best Data Source for FMRs
FMRs.30 The variables they reported differently include housing occupancy,
the year the structure was built, the number of rooms, and gross rent. For
instance, the study found that for certain areas, the ACS reported
moderately lower gross rents than did the decennial census. According to
the Census Bureau, the differences may result partly from different survey
processing techniques or from the multiyear aspect of ACS data. Regardless
of the cause, FMRs for fiscal year 2008 (the first year of rebenchmarking
with ACS data) could show bigger changes than would be the case using
decennial census data. According to HUD, consistent FMRs-that is,
estimates that change gradually from year to year-are important because
wide year-to-year fluctuations, especially those changes that lower the
FMR, can be disruptive to PHAs, which must annually reconsider their
payment standards any time HUD changes the FMR.
HUD will address the ACS challenges when it receives and begins to analyze
2005 ACS data-that is, the data collected during the first year when the
ACS is fully implemented-in Fall 2006. HUD may choose to participate in an
ACS Technical Workshop led by the Census Bureau, which may help the agency
address these challenges.
Despite the challenges the ACS poses for HUD, neither we nor various
researchers and industry experts found reason to suggest (1) that HUD
should not go forward with its plans to use the ACS or (2) that there are
viable alternatives to the ACS. Other sources of information, such as
private-market rent data and tax assessment data, typically do not contain
the information that HUD needs to estimate FMRs. For example:
o Private-market rent data typically include more expensive properties
(i.e., luxury units, usually large apartment complexes, in
metropolitan areas). Most voucher holders do not rent such properties
because they cannot afford them. Additionally, these data do not
include single-family homes-properties that voucher holders may also
lease.
o Private-market and tax assessment data are typically of lesser quality
compared with the data sources that HUD generally uses to estimate
30U.S. Census Bureau, Meeting 21st Century Demographic Data Needs -
Implementing the American Community Survey Report 8: Comparison of the
American Community Survey Three-Year Averages and the Census Sample for a
Sample of Counties and Tracts
(Washington, D.C.: 2004).
Page 33 GAO-05-342 Fair Market Rents
HUD Did Not Follow One of Its Data Quality Guidelines and May Lack Data
Sources to Assess the Accuracy of Future FMRs
FMRs. Private-market rent data often do not contain a representative
sample of the full rent distribution in an FMR area.
o Private-market or tax assessment surveys that include rent data may not
consistently include questions that ensure the units included adhere to
HUD's criteria (e.g., rents only from recent movers).
The potential exists for HUD to improve how it estimates FMRs and their
accuracy because (1) the agency presently does not follow its objectivity
guideline for ensuring the transparency and reproducibility of its data
and methods for estimating its FMRs and (2) it may in the future lack a
way to assess the accuracy of ACS-based rent estimates when other
information, such as comments from public housing agencies, suggests it
may need to do so. Various federal agencies, including HUD, have developed
guidelines to ensure they disseminate quality data. Three of HUD's
standards-utility, integrity, and objectivity-apply to FMR estimation.
Although HUD appears to be following the utility and integrity guidelines,
it did not follow its objectivity guideline-which calls for the agency to
make its data sources and methods transparent so the results can be
independently reproduced. Additionally, as HUD comes to depend less on RDD
survey and AHS data, it may not have a means to assess the accuracy of
future FMR estimates.
HUD Has Not Followed Its Data Quality Guideline on Objectivity
Section 515 of the Treasury and General Government Appropriations Act for
Fiscal Year 2001 (Pub. L. No. 106-554) directs OMB to issue governmentwide
guidelines that provide policy and procedural guidance to federal agencies
for ensuring and maximizing the quality, objectivity, utility, and
integrity of information disseminated by the agencies. According to OMB,
information that has been subject to independent reanalysis is generally
presumed to be of acceptable objectivity and therefore reliable to the
user. In addition, OMB states that an important benefit of transparency
and reproducibility (objectivity) is that the public can assess how much
an agency's information hinges on the specific analytical choices of the
agency. In response to OMB's guidelines, various federal agencies,
including HUD, have developed similar guidelines for ensuring that they
disseminate quality information. HUD's guidelines include ensuring the
utility (usefulness), integrity (protection from unauthorized access), and
objectivity (transparency and reproducibility) of the data it
disseminates.
Based on our review of available information, HUD appears to be following
the utility and integrity components of its guidelines for FMRs. HUD's
utility guideline states that the information disseminated should be
useful-a standard that encompasses accessibility and timeliness. HUD
follows this guideline by estimating FMRs on an annual schedule and making
FMRs public and easily accessible by publishing them on its Web site and
in the Federal Register. HUD's integrity guideline states that the
information disseminated should be protected from corruption or
falsification by unauthorized access or revision. According to HUD
officials, FMR data are kept on an internal server with highly restricted
access. Furthermore, to ensure the security of the system, the officials
said they maintain full electronic backups of all systems.
However, we found that HUD does not follow its guideline pertaining to
objectivity. HUD's guidelines state that it will make publicly available
the sources, data, and methods used to develop the information it
disseminates, and that results must be capable of being "substantially
reproduced." This means that independent reanalysis of original or
supporting data using the same methods should generate similar analytical
results. Although HUD generally describes its overall methodology for
estimating FMRs in publicly available documents, the agency has not
documented its methodology in sufficient detail to permit the results to
be independently reproduced. For example, although we obtained information
on the data and methods HUD used to estimate FMRs for fiscal years 200005,
HUD's process was not sufficiently documented to allow us to reproduce
FMRs without contacting HUD staff to assist us in doing so. In part, this
was because some of the data HUD used to estimate FMRs, such as utility
cost data, no longer exist after the agency upgraded the software it uses
to develop FMRs. Also, HUD did not document some of the key procedures,
variables, and data it used in estimating FMRs, such as the source of
benchmarking data (and its rationale for choosing each source in any given
year).31 Sufficient documentation would have allowed outside parties to
understand and assess how HUD developed any given FMR. For example,
sufficient documentation would allow an outside party to determine (1)
every decision HUD made (such as the FMR area definition or survey
source), (2) the decision rules it applied in making that decision, and
(3) the extent to which HUD consistently applied these rules.
31For example, if the survey source for an FMR estimate was the AHS, HUD's
documentation did not indicate what other sources, if any, it considered
that year and why it chose the AHS over any other available sources of
data for that year.
Page 35 GAO-05-342 Fair Market Rents
HUD's Declining Use of RDD Surveys and AHS Data May Limit Its Options for
Assessing the Accuracy of Future FMRs
HUD officials state that they do not have a plan to assess the accuracy of
FMRs after they start using ACS data to estimate them, in part because
they believe they will no longer have a quality comparison point or data
with which to do so. In the past, HUD assessed accuracy by comparing FMR
estimates with the rents derived from a subsequent RDD survey, the AHS, or
a decennial census. However, HUD plans to limit its future use of RDD
surveys and the AHS because of their concerns about cost and quality.
According to HUD officials, RDD surveys are very expensive (costing
upwards of $20,000) and their reliability is decreasing. Currently,
according to HUD officials, the agency has to start with a sample of
97,000 units to obtain a usable sample of 200 with which to estimate FMRs.
Moreover, the response rate for RDD surveys is about 40 percent, compared
with 90 percent for the ACS, and RDD surveys may have nonresponse bias
(i.e., people who respond to surveys may answer questions differently than
those who do not). Similarly, the AHS is becoming less useful for HUD's
purposes than when that survey first began. According to HUD officials,
the number and sample sizes of AHS metropolitan area surveys has been
decreasing over the past two decades, and they are not timely for HUD's
program purposes, thereby making them less useful for estimating FMRs than
has been the case in the past. Rent data from other sources, such as
private-market rent surveys and tax assessment records, also would not
provide HUD with a usable comparison point with which to assess FMR
32
accuracy.
Nonetheless, HUD's regulations require that the agency allow the public to
provide comments on proposed FMRs, and its information quality guidelines
permit affected parties to seek and obtain correction of information
disseminated by the agency. This extends to the accuracy of FMRs. In
addition to what its policies may require, even though FMRs based on ACS
data will most likely be more accurate than previous FMRs, HUD officials
acknowledge that ACS-based FMR estimates may be inaccurate from time to
time. For example, FMRs for the smaller areas (rebenchmarked every 3 or 5
years with ACS data) may need to be assessed within the interval to ensure
that they remain accurate between rebenchmarkings. Moreover, FMRs for any
areas with volatile rental
32According to ORC Macro, HUD may be able to use special ACS tabulations
from the Census Bureau to detect shifts in rent trends for areas where HUD
will use multiyear average data to estimate FMRs. These data for each FMR
area may not contain enough samples to estimate FMRs, but would give HUD
an indication that an existing FMR may be inaccurate.
Page 36 GAO-05-342 Fair Market Rents
Conclusions
markets may need to be assessed with some frequency to ensure that they
are accurate. However, as previously noted, HUD may lack sources of
comparable data in the future and may be unable to perform these
assessments.
HUD's task in accurately estimating FMRs is formidable. It must produce
estimates for hundreds of areas throughout the country despite having few
comprehensive, reliable data sources with which to do so. Additionally,
HUD faces the normal difficulties associated with predicting how rents and
housing markets will change months (or years) into the future.
Nonetheless, for those affected by FMRs, such as voucher holders, HUD's
ability to produce accurate estimates each and every year is vital-
estimates that are too low make it more difficult for low-income
households to find housing they can rent with a voucher, while estimates
that are too high may needlessly waste resources or prevent housing
agencies from serving more households.
At the time of our review, HUD could not dispel concerns about its process
for estimating FMRs because its methodology is not transparent enough to
allow others-including GAO-to independently analyze its rent data and
produce similar results. HUD and those who use FMRs would benefit from a
more transparent methodology because this could enhance the credibility of
the estimates by clearly delineating the choices HUD makes, what
alternatives it may have had in making those choices, and the decision
rules it applied; for example, whether to use OMB's area definitions or
how much to modify them (and the basis for doing so). Making the
methodology transparent would also give users more and better information
with which to consider whether FMRs reliably reflect an accurate estimate
of the rents voucher holders and others will encounter.
The advent of a new data source holds promise for HUD because a system of
FMRs that are largely based on the ACS will likely improve the quality and
accuracy of these estimates. However, the level of improvement in the
quality and accuracy of FMR estimates depends on how HUD uses the ACS
data. By choosing to use regional-level data to update fiscal year 2006
FMRs rather than the more local state-level data, HUD may not be taking
full advantage of the new data source as soon as it can.
In addition, as it transitions to the ACS, HUD expects to discontinue its
use of other surveys like the RDD surveys and the AHS to assess the
accuracy of its FMRs and, therefore, will not have a means to assure
itself and others that any given FMR estimate is accurate, particularly
when it receives public comments or other information suggesting it needs
to do so. While we agree that HUD is right to be concerned about the
escalating costs and declining quality of surveys such as the RDD surveys,
having no reasonable alternative to assess the accuracy of an FMR will not
likely address the concerns of PHAs with reason to question FMR accuracy
and may also contradict HUD's own data quality guidelines.
To improve the usefulness of its FMR estimates, we recommend that the
Recommendations for
Secretary of HUD take the following three steps:
o ensure that HUD fully documents its method for estimating FMRs by
following all of its data dissemination quality guidelines,
particularly those pertaining to the transparency and reproducibility
of its methodology;
o use, as much as possible, the ACS data that corresponds more closely
to FMR areas to update the fiscal year 2006 FMRs; and
o develop a mechanism to assess the accuracy of future FMRs, including
those that are based on the ACS, in instances where HUD learns of
information suggesting it needs to do so.
We provided a draft of this report to HUD for its review and comment. In a
Agency Comments and
letter from the Assistant Secretary for Policy Development and Research
(see app. II), HUD described our report as a good summary of the intent of
FMR estimates and the implementation of its methods. HUD also suggested
certain changes and clarifications to our report. For example:
o HUD suggested that we present population-weighted accuracy estimates
in the "Highlights" page of our report. We agree that
population-weighted estimates are important and note that we present
the information in the body of the report rather than the "Highlights"
page.
* HUD provided a revised statement describing the process they use
to eliminate subsidized and nonstandard housing units from the
rent distribution. As HUD requested, we incorporated the new
language in footnote 16.
* HUD agreed with our recommendation that it can better document
its methods for estimating FMRs, but also requested that we
clarify certain transparency and reproducibility issues in our
report and recognize its ongoing efforts in this regard. Among
other things:
o HUD noted a distinction between process transparency and
reproducibility of results, stating that the public's needs are better
met by providing an overview of how FMRs are calculated and then
showing the individual calculations for each FMR estimate, rather than
providing system technical documentation, such as computer programs
and input data.
o HUD has sought to make the data and calculation process publicly
available and transparent. For example, HUD noted that it currently
posts on its Web site publicly releasable versions of 2000 decennial
census detailed rent distribution files; FMR history files from the
Federal Register, including Annual Adjustment Factors; and a summary
of the general methodology and major data sources it uses to estimate
FMRs.
o HUD stated that it provided us with additional information, such as
computer programs and input data, it used to estimate FMRs, and met
with us as needed to explain the FMR methodology, including the large
number of different data sources, decision rules, complex decision
trees, and complex series of computer programs it uses to estimate
FMRs.
We agree that providing step-by-step calculation details for each FMR
estimate would contribute to process transparency. Moreover, we agree that
HUD currently makes the major data sources and general methodology it uses
to estimate FMRs publicly available on its Web site. However, as our draft
report noted, the current information that HUD makes publicly available
does not show the individual calculations for each FMR estimate and
therefore is not sufficient to substantially reproduce FMRs, a standard
set out in HUD's data quality guidelines.
With respect to reproducibility, in reviewing HUD's process for estimating
FMRs, we asked for and HUD provided additional information, such as
computer programs, input data, and associated documentation. Because HUD
did not have and could not provide us with critical documents, such as a
clear step-by-step guide or data dictionary, HUD officials met with us to
explain the various computer programs and variables they used-a step that
should not be necessary if the objective is for us to be able to
independently substantially reproduce FMR estimates. Nonetheless, the
information and explanations HUD provided were not sufficient to allow us
to independently reproduce FMR estimates. As HUD noted in its comments,
documentation of computer programs and input data, such as it provided us,
is not as useful as step-by-step guidelines that clearly detail how it
produces each FMR. As a result, HUD indicated it plans to consolidate in
one place all of the information it uses to estimate FMRs and create a new
tool, for release in April 2005, detailing how it develops each FMR. By
making this information publicly available on its Web site, HUD expects to
improve the transparency and reproducibility of its FMR estimates,
particularly for the users of these estimates.
HUD disagreed with our recommendation that it use, as much as possible,
the ACS data that corresponds more close to FMR areas to update its fiscal
year 2006 FMRs. HUD indicated that many of the annual state-level rent
numbers have a pattern of erratic changes. However, according to the
Census Bureau, for states with populations of 1 million or more, annual
ACS changes for 2001 to 2004 are generally reliable. More importantly, as
HUD officials indicated to us during our review, a necessary first step in
using these data to update fiscal year 2006 FMRs would be to assess for
each state whether anomalies or other concerns might indicate a need to
defer in certain instances using the state-level ACS data. Accordingly,
our recommendation was for HUD to use the state-level data as much as
possible, recognizing that the agency could do so only in instances where
the ACS data are sufficiently reliable for this purpose, and we have
retained the recommendation.
HUD did not explicitly agree or disagree with our recommendation that it
develop a mechanism to assess the accuracy of future FMR estimates.
However, HUD disagreed with our draft report's statement that declining
use of RDD surveys and the AHS may limit its options for assessing FMR
accuracy. Specifically, HUD stated that even though the ACS will be much
more accurate than any other survey unless the other survey offers more
current estimates, ACS rent estimates will always lag by at least a year
(from the midpoint of the survey estimate); thus, use of national rent
data to trend the FMR estimate could lead to estimation errors in housing
markets with unusual rent increases or decreases. Accordingly, HUD noted
that one of the major challenges posed by the ACS is how to identify those
areas where the use of regional or national trending factors results in
estimation inaccuracy, and stated that it is currently exploring two
alternatives to deal with the issues. Thus, although HUD stated that it
disagreed with our statement, the actions that it intends to take are
consistent with our recommendation.
HUD also suggested technical clarifications to our report, which we have
incorporated as appropriate.
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
after the date of this letter. At that time, we will send copies to the
appropriate congressional committees and to the Secretary of Housing and
Urban Development. We will also make copies available to others upon
request. In addition, this report will be available at no charge on the
GAO Web site at
http://www.gao.gov.
If you have any questions about this report, please contact me at (202)
512-6878 or [email protected] or Bill MacBlane, Assistant Director, at (202)
512-6764 or [email protected]. Key contributors to this report are listed
in appendix III.
Sincerely yours,
David G. Wood
Director, Financial Markets and Community Investment
Appendix I
Objectives, Scope, and Methodology
To describe how the Department of Housing and Urban Development (HUD)
estimates fair market rents (FMR), we first analyzed statutes and HUD
regulations, reviewed HUD documents, and interviewed HUD officials to
identify each step that HUD takes to estimate FMRs, including the role
that the public has in the process. Further, we also spoke with nine HUD
field economists for each HUD region-typically, the first point of contact
for the public-to further understand the role that the public can play in
adjusting the FMR estimate.1 To identify and describe the relevant
characteristics of the major data sources HUD uses to estimate FMRs, we
reviewed agency documents.
To determine how accurate FMRs were, we compared two-bedroom FMRs that HUD
had in place for fiscal year 2000-that is, estimates derived from HUD's
revisions to its baselines and from its update processes-with the results
of the 2000 census.2 In addition, we compared two-bedroom FMRs that HUD
estimated for fiscal years 2001-05 with data from surveys HUD and others
conducted for 153 FMR areas over this period. We assessed accuracy by way
of a comparison to the decennial census or other surveys because our own
methodological experts as well as others conducting similar research on
these issues determined that such a comparison is the best way to do so
when the true values-that is, the distribution of all rents-cannot be
known. In conducting both of these comparisons, we focused on two-bedroom
units because HUD directly estimates FMRs for these units from the
decennial census and its other rebenchmarking surveys. HUD does not
directly estimate FMRs for other bedroom sizes, making it not possible to
do a comparison of those FMRs to the results of a survey such as the
American Housing Survey (AHS) or a Random Digit Dialing (RDD) survey.3
1As of December 2004, nine HUD regional field economists managed the
agency's economic work in the 10 HUD regions because there was a vacancy
in Region 2 (New York/New Jersey).
2In order to use the 2000 decennial census data we obtained from HUD to
assess the accuracy of FMRs, we verified the reliability of the census
data by asking HUD officials a series of data reliability questions.
3For non-two-bedroom units in the 2000 decennial census survey and 153
subsequent rebenchmarking surveys, HUD took the survey results for
two-bedroom rents and applied a rent ratio that, in HUD's view, captured
the approximate relationship between rents for two-bedroom units and other
sizes. For example, through fiscal year 2004, for three-bedroom units, HUD
determined that the relationship between these and two-bedroom rents was
1.25, so the three-bedroom FMR would be 125 percent of what HUD estimated
for two-bedroom units.
Appendix I Objectives, Scope, and Methodology
We performed an associative analysis to determine what components of HUD's
FMR estimation process may have explained the results we found when we
assessed accuracy (e.g., whether the estimate was for a metropolitan or
nonmetropolitan area).4 Our analysis was limited to making associations
between the components of HUD's methodology and the accuracy of its FMR
estimates; it did not allow us to make a direct causal link between the
two because all of the information we needed was either no longer
available or may not be able to be captured by HUD's method for making
these estimates. Specifically, (1) HUD could not provide all of the data
used to estimate FMRs from 1990 to 2005, such as utility cost data,
because these were kept on individual staff's computers and in many cases
were not transferred when HUD moved its FMR data systems to a more
advanced server; (2) the lack of transparency we found relative to HUD's
objectivity guideline for data quality meant that we could not identify
and isolate specific components of its methodology to attempt a causal
(rather than associative) analysis; and (3) neither we nor HUD can control
for factors outside of HUD's estimation process that may affect accuracy,
such as sudden employment changes that cause an area's rents to increase
rapidly.
We present our analysis of the accuracy of FMR estimates in terms of the
degree (percentage) to which the FMR matched or was close to the
corresponding survey. For example, for the corresponding fiscal year 2000
FMRs and census data, we calculated the following for each FMR:
Survey (census) - Fair Market Rent Estimate = x percent Survey (census)
This calculation produced a percentage that, in this example, we
characterize as the estimate being within x percent of the census. For
descriptive purposes, we arrayed these comparisons in increments of 10
percent because, in terms of the initial FMR, this is the range (90 to 110
percent of the FMR) in which the public housing agencies may set their
payment standards without prior approval from HUD.
4When we compared the accuracy of FMR estimates with the two types of
update factors HUD uses (metro-specific Consumer Price Index or RDD
regional gross rent change factor), we excluded a limited number of FMR
areas because HUD applies special rules for updating this group, making
the update calculations too dissimilar for our purposes.
Page 43 GAO-05-342 Fair Market Rents Appendix I Objectives, Scope, and
Methodology
To determine how and when the incorporation of the American Community
Survey (ACS) data might affect the accuracy of the FMR estimates, we
reviewed agency documents and interviewed HUD officials to determine how
the agency plans to use ACS data to estimate FMRs. We also analyzed Bureau
of the Census documents to compare characteristics of the ACS data with
those of the data sources HUD currently uses (the decennial census long
form, the AHS, and RDD surveys) to estimate FMRs. Additionally, we
reviewed research by the National Academy of Sciences and ORC Macro, in
addition to our own, on the use of ACS data.
To identify changes HUD could make to improve the way it estimates FMRs
and their accuracy, we first assessed HUD's process for estimating FMRs
against its data quality guidelines. More specifically, we analyzed each
HUD guideline-utility, integrity, and objectivity-and compared them with
HUD's method for estimating FMRs. We also interviewed HUD officials to
determine how the guidelines related to FMRs. Additionally, on the basis
of our analysis of the data characteristics we found to be associated with
greater accuracy in FMRs (recent, higher quality, and more local), we
interviewed housing industry experts that either routinely work with
housing data or are familiar with HUD's data needs to identify potential
alternative data sources that HUD could use to estimate FMRs. We also
interviewed HUD officials to determine the availability and merits of
alternative data sources.
We conducted our work in Washington, D.C., between May 2004 and February
2005 in accordance with generally accepted government auditing standards.
Appendix II
Comments from the Department of Housing and Urban Development
Appendix II Comments from the Department of Housing and Urban Development
Appendix II Comments from the Department of Housing and Urban Development
Appendix II Comments from the Department of Housing and Urban Development
Appendix II Comments from the Department of Housing and Urban Development
Appendix II Comments from the Department of Housing and Urban Development
Appendix II Comments from the Department of Housing and Urban Development
Appendix III
GAO Contacts and Staff Acknowledgments
David G. Wood, (202) 512-6878
GAO Contacts
Bill MacBlane, (202) 512-6764
In addition to the individuals named above, Triana Bash, Tania Calhoun,
Staff
Steve Brown, John Larsen, Marc Molino, Chris Moriarity, Robert Parker,
MacDonald Phillips, Carl Ramirez, Barbara Roesmann, and Anita Visser made
key contributions to this report.
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts GAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
Contact:
To Report Fraud, Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Relations
Washington, D.C. 20548
Paul Anderson, Managing Director, [email protected] (202) 512-4800
Public Affairs
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***