Public Housing: HUD Should Improve the Usefulness and Accuracy of Its
Management Assessment Program (Chapter Report, 01/29/97, GAO/RCED-97-27).
The Department of Housing and Urban Development (HUD) is responsible for
ensuring that the 3,000 independent, state-chartered public housing
authorities provide safe and decent housing and protect the federal
investment in their properties. However, the public housing management
assessment program--HUD's primary tool for measuring the performance of
housing authorities--has been criticized as unreliable, inaccurate, and
at times conflicting with good property management practices.
Nevertheless, because no other measurement tool exists, Congress and HUD
have proposed using the program as a basis for deregulating or rewarding
high-scoring housing authorities. This report reviews HUD's
implementation of the program. GAO discusses whether (1) HUD's field
offices are using the program and complying with the program's statutory
and regulatory requirements to monitor and provide technical assistance
to housing authorities; (2) program scores have increased and how HUD
uses the program to inform HUD's Secretary and Congress about the
performance of housing authorities; and (3) program scores are
consistently accurate and can be considered a generally accepted measure
of good property management.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-97-27
TITLE: Public Housing: HUD Should Improve the Usefulness and
Accuracy of Its Management Assessment Program
DATE: 01/29/97
SUBJECT: Public housing
Technical assistance
Local governments
Federal agency reorganization
Federal property management
Housing programs
Audits
IDENTIFIER: HUD Public Housing Management Assessment Program
HUD System for Management Information Retrieval-Public
Housing Data Base
Birmingham (AL)
Kansas City (MO)
Minneapolis (MN)
San Antonio (TX)
San Francisco (CA)
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Housing and Community
Opportunity, Committee on Banking and Financial Services, House of
Representatives
January 1997
PUBLIC HOUSING - HUD SHOULD
IMPROVE THE USEFULNESS AND
ACCURACY OF ITS MANAGEMENT
ASSESSMENT PROGRAM
GAO/RCED-97-27
Public Housing
(385602)
Abbreviations
=============================================================== ABBREV
GAO - General Accounting Office
HUD - Department of Housing and Urban Development
IG - Office of Inspector General
MOA - memorandum of agreement
PHA - public housing authority
PHMAP - Public Housing Management Assessment Program
SMIRPH - System for Management Information Retrieval-Public Housing
Letter
=============================================================== LETTER
B-275648
January 29, 1997
The Honorable Rick Lazio
Chairman, Subcommittee on Housing
and Community Opportunity
Committee on Banking and
Financial Services
House of Representatives
Dear Mr. Chairman:
This report responds to your request that we review the Department of
Housing and Urban Development's (HUD) use of its Public Housing
Management Assessment Program (PHMAP). Specifically, the report
discusses (1) HUD's use and implementation of the program at its
field offices, (2) public housing authorities' (PHA) PHMAP scores
over the first 4 years of the program, and (3) limits on any
additional uses for the program.
We are sending copies of this report to the Secretary of Housing and
Urban Development and will make copies available to others upon
request. Major contributors to this report are listed in appendix
VI.
Please call me at (202) 512-7631 if you or your staff have any
questions.
Sincerely yours,
Judy A. England-Joseph
Director, Housing and Community
Development Issues
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
At a current annual cost of $5.4 billion, the Department of Housing
and Urban Development (HUD) subsidizes the operation, maintenance,
and modernization of the nation's public housing, a $90 billion
investment that provides homes to 3 million people. Because HUD
provides this subsidy to more than 3,000 independent, state-chartered
public housing authorities, the Congress holds HUD responsible for
ensuring that these authorities efficiently provide safe and decent
housing and protect the federal investment in their properties.
However, the Public Housing Management Assessment Program
(PHMAP)--HUD's primary tool for measuring housing authorities'
performance--has been criticized as unreliable, inaccurate, and at
times conflicting with good property management practices.
Nevertheless, because no other measurement tool exists, the Congress
and HUD have proposed at different times using the program as a basis
for deregulating or rewarding high-scoring housing authorities.
Stressing the need for HUD to hold housing authorities accountable
while making better use of the data PHMAP produces, the Chairman of
the Subcommittee on Housing and Community Opportunity, House
Committee on Banking and Financial Services, asked GAO to review
HUD's implementation and use of PHMAP. Specifically, the Chairman
asked GAO to determine
-- whether HUD's field offices are using PHMAP and complying with
the program's statutory and regulatory requirements to monitor
and provide technical assistance to housing authorities;
-- whether PHMAP scores have increased and how HUD uses the program
to inform HUD's Secretary and the Congress about the performance
of housing authorities; and
-- whether PHMAP scores are consistently accurate and can be
considered a generally accepted measure of good property
management.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
The National Affordable Housing Act of 1990 directed HUD to use
certain indicators, including vacancy rates and the percentage of
rents uncollected, to assess the management performance of local
housing authorities. The act also directed HUD to determine the
cause of an authority's management problems, commit both HUD and the
housing authority to a specific course of corrective action, and
document agreed upon corrective actions in memorandums of agreement.
To meet the act's requirements, HUD developed PHMAP to annually
obtain data from each housing authority on 12 basic indicators of
management performance, such as vacancy rates and operating expenses.
On the basis of aggregate performance against these indicators, HUD
calculates a score from 0 to 100 for each authority and assigns one
of the following three designations: "troubled performer" for a
score less than 60, "standard performer" for a score between 60 and
less than 90, and "high performer" for a score 90 or above.
HUD's field offices have nearly 800 staff devoted to oversight of
housing authorities and implementation of the full range of the
Department's public housing programs, including PHMAP. Because HUD's
field offices are responsible for implementing PHMAP, HUD expects
them to ensure housing authorities meet PHMAP's requirements and
provide technical assistance when an authority has problems doing so.
Technical assistance can cover a wide variety of activities, such as
a focused review of an authority's compliance with HUD's rules and
regulations, discussions over the telephone, on-site reviews of
HUD-funded modernization work, or suggestions for improving an
authority's occupancy rate and rent collection procedures.
Field offices depend on each authority to submit and certify to the
accuracy of about half the data that lead to the overall PHMAP score;
the balance of the information HUD uses comes from its existing
information system for tracking expenditures from major grants. With
each troubled authority, the act requires HUD to perform an
independent management assessment to determine the causes of an
authority's problems and then enter into a binding memorandum of
agreement stipulating the problems that authority needs to address
and an approach and a timetable to resolve them. Also, when a
troubled authority's new PHMAP score would cause HUD to remove its
troubled designation, HUD's policy has been to require its field
offices to verify the accuracy and completeness of the data the
housing authority submitted. Standard- and high-performing
authorities that fail any of the 12 indicators must submit a plan for
improving their performance in that indicator.
While HUD's primary use of PHMAP has been to identify troubled
housing authorities and target technical assistance to them, the
Congress and HUD have proposed to use PHMAP for other purposes. In
1994, the Senate Committee on Banking, Housing, and Urban Affairs
proposed additional flexibility for housing authorities that had
achieved PHMAP scores over 90. In its fiscal year 1997 budget
request, HUD proposed to give high-performing housing authorities
bonuses based in part on their PHMAP scores.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
Most of HUD's field offices are using PHMAP to identify troubled
housing authorities and target HUD's limited technical assistance
resources. However, the field offices have not been systematically
using the assessment program, as required by statutes and
regulations, to monitor housing authorities' progress in improving
their performance and target technical assistance to them. For
example, the field offices have generally not been meeting the act's
requirement to enter into memorandums of agreement with troubled
authorities, nor have the field offices consistently met HUD's
requirement that housing authorities document plans to correct low
scores in individual performance indicators. Also, the impact of a
1995 reorganization of the field offices' functions and current
departmental downsizing continue to influence some offices' ability
to provide technical assistance.
Performance scores generally have increased during the first 4 full
years of the program. With average scores increasing, the total
number of troubled housing authorities has decreased, and the
greatest proportion of those that are troubled are the smallest
authorities--those managing fewer than 100 units. The proportion of
high-performing authorities has increased steadily from about 33
percent in 1992 to over 50 percent in 1995. High-performing
authorities manage nearly 50 percent of all public housing units.
Periodically, HUD officials provide the Secretary and the Congress
information on the performance of all housing authorities as well as
the number of troubled authorities.
HUD's confirmatory reviews of the information underlying assessment
scores have shown the scores to be inaccurate in half the instances
when such reviews were performed. Regardless of the scores'
accuracy, HUD and public housing industry officials do not believe
that the management assessment program comprehensively assesses how
well local housing authorities manage their properties. This is
because the assessment program does not include indicators to
specifically measure overall housing quality or the quality of
maintenance.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
FIELD OFFICES' USE OF PHMAP
AND RELATED OVERSIGHT TOOLS
HAS BEEN LIMITED
-------------------------------------------------------- Chapter 0:4.1
GAO visited 5 of HUD's 49 field offices and sent a survey to all 49
offices asking about their use of PHMAP. Officials in these offices
generally found PHMAP useful to identify troubled housing authorities
and target limited technical assistance and oversight resources.
However, field offices reported that they have not been
systematically complying with PHMAP's statutory and regulatory
follow-up requirements for all housing authorities.
The extent to which HUD's field offices used PHMAP to provide
technical assistance to housing authorities varies widely. In part,
this difference stems from how different field offices have
interpreted their role in helping authorities improve performance on
PHMAP indicators as well as their overall operations. Some field
offices told GAO that they interpret their role in providing
oversight and technical assistance narrowly, generally limiting their
assistance to advice, information on complying with federal rules and
regulations, and suggestions for solving management problems. Other
offices were more willing to get involved in a housing authority's
operations. For example, staff from one field office spent several
days at a troubled authority to help it set up proper tenant rent
records and waiting lists.
Regarding noncompliance with follow-up requirements, GAO found the
following:
-- In 1992, HUD's field offices entered into 29 percent of the
required memorandums of agreement; by 1995, just 18 percent of
the housing authorities that should have been operating under a
memorandum of agreement actually were. Primarily, field offices
said the reason they did not enter into these required
agreements with troubled housing authorities is that the housing
authorities had already corrected or were in the process of
correcting their management deficiencies.
-- Even though HUD requires an improvement plan to address each
performance indicator an authority fails unless that authority
can correct the deficiency within 90 days, 31 percent of HUD's
field offices had not ensured that housing authorities had
developed such plans.
-- Field offices generally did not meet HUD's requirement to
confirm the accuracy of all PHMAP scores that were high enough
to remove the troubled designation from a housing authority. In
fiscal year 1995, HUD's field offices confirmed fewer than 30
percent of the scores that should have been confirmed. While 13
offices performed none of the required confirmatory reviews,
some of these same offices performed confirmatory reviews of
other standard- or high-performing authorities. Although no
minimum level of activity is required, in 1995 HUD confirmed
just over 6 percent of all scores, with some field offices
performing no confirmatory reviews and others performing 10 or
more.
-- At the five field offices GAO visited, limited use was made of
the independent financial and compliance audits conducted
annually at each authority. Over a year ago, HUD began to
require that these audits certify that the PHMAP data from each
authority were accurate and complete; HUD added this requirement
because it does not have the resources to confirm each score
every year. Nonetheless, few HUD staff in the field offices GAO
visited were aware of this requirement or used the audits to
better focus their oversight and technical assistance.
The technical assistance HUD staff find most effective at improving
the performance of housing authorities is often the type of help
least frequently provided. HUD's field offices told GAO that
technical assistance and oversight are most effective when they
provide it on-site at the local authority. For example, while most
staff in field offices said confirmatory reviews, which must take
place on-site, were one of the most effective ways to provide
technical assistance, discussions over the telephone were the most
common form of technical assistance. Some field offices cited
resource constraints--a lack of staff, travel funds, or expertise--as
the main reason for not meeting follow-up requirements or visiting
housing authorities more often; others opted not to enforce
requirements when they believed the authorities were already
addressing their problems.
SCORES ARE INCREASING, BUT
HUD RECOGNIZES DATABASE
FLAWS LIMIT PROGRAM'S USE
-------------------------------------------------------- Chapter 0:4.2
Average PHMAP scores have increased over the life of the program,
rising from an average of 83 in 1992 to 86 in 1995, the last year of
complete data. In addition, HUD's database of PHMAP scores indicates
the following two trends:
-- The number of high-performing authorities grew each year, from
almost a third in 1992 (33 percent) to over half in 1995 (57
percent).
-- The number of troubled housing authorities has declined from 130
in 1992 to 83 in 1995. However, by 1995, the smallest housing
authorities--those managing fewer than 100 units--accounted for
a greater share of those designated as troubled than when the
program began. In 1995, half of all housing authorities HUD
designated as troubled were small.
GAO found missing, inaccurate, and inconsistent data in HUD's primary
database for storing PHMAP scores. Nevertheless, HUD makes regular,
periodic use of the database to provide information to the Secretary
and the Congress on all housing authorities' scores and the number of
troubled authorities at any given time. However, before providing
this information to others, HUD first manually verifies much of the
data it draws from this system. Senior HUD officials acknowledged
these problems with the database and added that they are currently
working to address data accuracy and reliability problems as well as
improve their ability to correct errors sooner.
THE QUESTIONABLE ACCURACY
AND VALIDITY OF PHMAP SCORES
LIMIT THE PROGRAM'S
USEFULNESS
-------------------------------------------------------- Chapter 0:4.3
PHMAP scores are often changed after HUD confirms the data used to
support the scores. In commenting on this report, HUD said that most
confirmatory reviews are conducted of high-risk housing authorities
whose data are most susceptible to being found inaccurate. Over
half--58 percent--of the changes HUD made to PHMAP scores resulted in
HUD's lowering the score by an average of 14 points; 42 percent of
the changes resulted in HUD's raising the score by an average of 8
points. Typically, HUD changes a PHMAP score after such a review for
several reasons, including the housing authority's failure to report
correctly the required data or its failure to maintain documentation
to support its data.
PHMAP scores are not a generally accepted measure of good property
management. HUD officials, as well as representatives of public
housing industry associations and professional property management
consultants, told GAO that the PHMAP indicators do not assess all
major aspects of a housing authority's performance. For example,
PHMAP does not include an independent on-site inspection of the
condition of an authority's housing, so it does not adequately assess
the quality of modernization work or routine maintenance. These same
HUD officials and industry representatives also told GAO that PHMAP
does not always allow for extenuating circumstances that can lead to
decisions inconsistent with good property management. For example, a
housing authority can improve its PHMAP score on the tenants accounts
receivable indicator by writing off as uncollectible past due rents
from vacated tenants, but PHMAP would not measure how diligent an
effort the authority had undertaken to collect the rent.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
GAO recommends that the Secretary of HUD
-- provide guidance to its field offices that clearly (1)
articulates their minimally acceptable roles regarding oversight
and assistance to housing authorities and (2) emphasizes the
importance of using the results of the independent audits to
better target the Department's limited technical assistance
resources.
Furthermore, because scores are not consistently accurate and PHMAP
does not measure all aspects of property management, GAO recommends
that HUD
-- not consider additional uses for PHMAP, including using scores
as criteria for funding bonuses, until it determines that PHMAP
meets an acceptable level of accuracy and more comprehensively
measures property management performance and
-- require its field offices to confirm the PHMAP scores of housing
authorities with scores low enough that the authorities are at
risk of being designated troubled.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
GAO provided a draft of this report to HUD for its review and
comment. HUD agreed with GAO's recommendations and described the
steps that the Department has begun taking to implement them.
However, HUD expressed concern that the draft report (1) used
potentially inaccurate data from HUD's PHMAP database, (2)
incorrectly assumed that PHMAP was intended to be an all-encompassing
system that measures both management performance and physical housing
conditions, (3) neglected to place PHMAP in a historical perspective
by discussing HUD's previous systems for assessing and identifying
troubled housing authorities, and (4) reached incorrect conclusions
regarding the reliability of all PHMAP scores on the basis of the
results of confirmatory reviews of the high-risk authorities most
susceptible to discrepancies in their PHMAP data.
GAO used the best data available for this review. GAO recognizes and
discusses in the report the inaccuracies in the PHMAP data. Where
HUD was able to provide more accurate data than its PHMAP database
reports, GAO used that data in this report. GAO did not assume that
PHMAP should be a complete measure of both performance and physical
conditions. This report describes the current uses of PHMAP data and
addresses how the program's limitations affect its suitability for
additional purposes. GAO did not provide a historical perspective on
the program because discussing HUD's previous systems for assessing
and identifying troubled housing authorities did not contribute to
the review's objectives of evaluating HUD's use of PHMAP, trends in
scores, or limits on additional uses for the program. Finally, GAO
did not reach a conclusion about the reliability of all housing
authorities' PHMAP scores. The report discusses only the reliability
of PHMAP scores for those housing authorities whose scores are so low
that they may be at risk of being designated troubled. Statements
have been added to the report to clarify this point.
HUD's written comments are presented in appendix V and GAO's
responses are discussed at the end of each chapter as appropriate.
INTRODUCTION
============================================================ Chapter 1
Because public housing represents a $90 billion investment on the
part of the federal government since the program's inception in 1937
and because the Department of Housing and Urban Development (HUD)
currently spends $5.4 billion a year on operating subsidies and
modernization grants for this housing, interest remains keen in
knowing how well local public housing authorities (PHA) are managing
their properties. The PHAs, through which HUD provides these
subsidies and grants, house 3 million low-income people, many of whom
are elderly or disabled. The Congress holds HUD responsible for
ensuring that the authorities provide safe and decent housing,
operate their developments efficiently, and protect the federal
investment in their properties.
The National Affordable Housing Act of 1990 required HUD to develop
indicators to assess the management performance of PHAs.\1 This law
became the framework through which HUD developed one of its primary
oversight tools for housing authorities, the Public Housing
Management Assessment Program (PHMAP). Primarily, PHMAP establishes
objective standards for HUD to evaluate and monitor the management
operations of all PHAs to identify those that are troubled.
According to HUD, PHMAP also allows the Department to identify ways
to reward high-performing PHAs as well as improve the management
practices of troubled PHAs. The program also allows PHAs' governing
bodies, management officials, residents, and the local community to
better understand and identify specific program areas needing
improvement.
To help improve public housing management, the National Affordable
Housing Act of 1990, as amended (the act), required HUD to develop
indicators to assess the performance of PHAs in all the major aspects
of their management operations. The act required HUD to use certain
indicators as well as provided discretion for the Secretary of HUD to
develop up to five additional indicators that the Department deemed
appropriate. HUD implemented PHMAP by using the 12 indicators listed
in table 1.1, the first seven of which are those required by statute.
Table 1.1
Twelve PHMAP Indicators
PHMAP indicator Measurement
---------------------------------------- ----------------------------
1.Vacancy number and percentage Number and percentage of
vacancies, including
progress made within the
previous 3 years to reduce
vacancies
2.Modernization Amount and percentage of
funds unexpended after 3
years
3.Rents uncollected Balance uncollected as a
percentage of total rents to
be collected
4.Energy consumption Increase in annual
consumption
5.Unit turnaround Average time required to
repair and reoccupy vacant
units
6.Outstanding workorders Proportion of maintenance
workorders outstanding
7.Annual inspection and condition of Percentage of units and
units and systems systems not inspected to
determine preventive
maintenance or modernization
needs
8.Tenants accounts receivable Percentage of monies owed to
the authority by current
residents
9.Operating reserves Percentage of operating
reserves maintained by the
authority
10.Routine operating expenses Level of operating expenses
compared to operating income
and federal subsidy
11.Resident initiatives Existence of a partnership
between residents and the
authority to promote
opportunities for self-
sufficiency and other
programs
12.Development Ability to develop
additional public housing
units
----------------------------------------------------------------------
Because some indicators are more important than others in measuring
management performance, HUD assigns them added weight in determining
the overall score.\2 HUD considers the indicators for vacancies,
rents uncollected, annual inspection and condition of units and
systems, and resident initiatives most indicative of good property
management and delivery of services to residents, so each one has a
greater weight than other indicators.
After reviewing existing procedures and extensively consulting with a
group of PHAs, public housing industry groups, private management
firms, resident groups, and HUD staff in field offices, HUD has
significantly revised the PHMAP indicators. HUD's revisions to
PHMAP, published December 30, 1996, eliminated three indicators;
consolidated four other indicators into two; and added one new
indicator, security.\3 These revisions primarily address the
performance indicators on which housing authorities report data, not
HUD's use of PHMAP data.
--------------------
\1 P.L. 101-625, Section 502(a), as amended by the departments of
Veterans Affairs and Housing and Urban Development, and Independent
Agencies Appropriations Act of 1992.
\2 According to a HUD official, these weights reflect the
Department's determination that they are the most important
indicators of good property management.
\3 HUD eliminated the tenants accounts receivable, routine operating
expenses, and development indicators. HUD also consolidated unit
turnaround into the vacancy indicator and energy consumption into the
financial management (formerly operating reserves) indicator.
INDICATOR GRADES DETERMINE THE
PHMAP SCORE, PERFORMANCE
DESIGNATION, REQUIRED
FOLLOW-UP, AND INCENTIVES
---------------------------------------------------------- Chapter 1:1
Annually, PHAs receive a grade of "A" through "F" for each of the
twelve indicators that apply to their operations. HUD uses a formula
that reflects the weights assigned to each indicator, converts
indicator grades into points, totals each PHA's points, and divides
that total by the maximum total the PHA could have achieved to arrive
at a percentage. That percentage, a number between 0 and 100, is the
PHMAP score.
HUD draws data on the performance of a housing authority from two
sources to determine the authority's PHMAP score. First, the housing
authority submits data to HUD for about half of the PHMAP indicators
and certifies that this information is accurate and complete.\4 HUD
assigns grades to each of these indicators according to a comparison
of the authority's data and HUD's criteria for grades "A" through
"F." The balance of the information HUD uses comes from its own
information system for tracking expenditures from major grants. This
system contains the financial and other types of data the field
offices need to grade the remaining indicators for which the PHAs do
not provide data. The field offices use this data and the
PHA-certified data to determine indicator scores, the PHMAP score,
and the PHA's performance designation.
The PHMAP score is HUD's starting point for both the performance
designation it assigns to a PHA and, depending on that designation,
the extent of follow-up required of the PHA to correct deficiencies
identified during the PHMAP assessment. Generally, HUD uses three
designations to describe the performance of PHAs:
-- troubled performers are those scoring less than 60 percent;\5
-- standard performers are those scoring between 60 and less than
90 percent; and
-- high performers are those scoring 90 percent or more.
HUD has the discretion to withhold the troubled designation or award
the high performer designation if a PHA's score is within 10 points
of the threshold for either designation and HUD determines that its
score results from the physical condition and/or neighborhood
environment of that authority's units rather than from the PHA's poor
management practices.
If a housing authority is designated as troubled, it faces several
mandatory follow-up activities and/or corrective actions to improve
performance and remove the troubled designation. Specifically, the
act requires HUD to perform an independent management assessment of
the troubled PHA's overall operations to identify the causes of the
deficiencies that led to its poor PHMAP score. HUD uses private
contractors to perform these independent assessments.
HUD expects the independent assessments to form the basis for the
second requirement for troubled PHAs--the memorandum of agreement
(MOA). A memorandum of agreement is a binding contract between HUD
and a troubled PHA to identify solutions to its management problems
and pursue those solutions in a way that is significant, expeditious,
and lasting. Among other things, HUD requires that the MOA address
the specific responsibilities of HUD and the PHA, the resources each
will commit to resolving the authority's problems, the annual and
quarterly performance targets for improving its performance on PHMAP
indicators, and the incentives for it to meet its performance targets
as well as sanctions for failing to do so. A PHA's initial MOA
generally lasts 18 months so that it can complete a second-year
agreement with HUD, if necessary, before the first expires.\6
HUD's regulations for implementing PHMAP require standard- and
high-performing PHAs to develop improvement plans for every PHMAP
indicator on which the PHA received an "F," unless the PHA can
correct the deficiency within 90 days;\7 HUD may also choose to
require these plans for indicators receiving scores of "D" or "E"
when failure to raise the grade might pose significant added risk.
An improvement plan documents how and when the PHA plans to correct
deficiencies. Although similar in content and scope to a memorandum
of agreement, improvement plans differ in that (1) PHAs develop and
submit them to HUD for approval rather than negotiate them with HUD
officials and (2) they are not a binding contractual commitment
between the PHA and HUD.
When HUD first implemented PHMAP, it offered high-performers a
variety of incentives, primarily regulatory relief from various
reporting requirements. These incentives included less frequent
reviews of changes to a PHA's operating budget and, for those
performing well on the modernization indicator, no prior HUD review
for architects' or engineers' contracts. In addition to regulatory
relief, high-performing PHAs receive a HUD certificate of
commendation and public recognition for their performance.
In its fiscal year 1997 budget request, HUD proposed an additional
PHMAP-based incentive for high-performing PHAs when it sought to
create a $500 million capital bonus fund (as part of the $3.2 billion
it sought for its public housing capital fund).\8 To be eligible for
a bonus, a PHA would have to be a PHMAP high performer and have
undertaken substantive efforts to obtain education and job training
for its residents. However, the Congress chose not to fund the bonus
proposal for public housing or any of HUD's other major programs, in
part because of concerns about HUD's ability to accurately and
reliably track the performance of bonus recipients.
--------------------
\4 PHAs certify the data for the following indicators: vacancy
number and percentage, rents uncollected, unit turnaround,
outstanding work orders, annual inspection and condition of units and
systems, and resident initiatives.
\5 PHMAP includes an additional designation, modernization troubled
(mod-troubled), which can apply to any PHA that scores less than 60
percent on the modernization indicator.
\6 Second and subsequent year agreements are necessary only as long
as the PHA remains troubled.
\7 Mandatory improvement plans would not apply to the modernization
indicator because any PHA receiving an F on this would be designated
mod-troubled.
\8 The public housing capital fund would have consolidated public
housing modernization, development, and several other capital repair
and replacement programs.
HUD'S FIELD OFFICES IMPLEMENT
PHMAP
---------------------------------------------------------- Chapter 1:2
With nearly 800 staff devoted to oversight of housing authorities and
implementation of the full range of HUD's public housing programs,
its field offices have the bulk of the Department's responsibility
for the day-to-day implementation of PHMAP. Field offices' PHMAP
responsibilities include determining the indicator grades and PHMAP
scores, negotiating memorandums of agreement, approving PHAs'
improvement plans, and monitoring their progress in meeting the goals
the MOA or improvement plan set forth. To determine a housing
authority's PHMAP score, a field office relies on that PHA to provide
about half the data that leads to the overall PHMAP score and certify
the data's accuracy. As a result, the overall PHMAP score and
everything it influences--from incentives for high performers to
sanctions for troubled PHAs--are very much a joint effort and a
shared responsibility.
A PHA may also request to exclude or modify the data HUD should
consider in computing its PHMAP score. An exclusion means that the
indicator (or one or more of its components) is entirely excluded
from calculations to determine the PHMAP score. For example, PHAs
with no ongoing modernization or development programs are
automatically excluded from being assessed on those indicators.
Modifying the data for an indicator allows HUD to consider unique or
unusual circumstances by exempting some of the data HUD usually
requires the PHA to consider. The PHA still receives a score for the
indicator, but the score would not reflect the data associated with
the PHA's unique or unusual circumstances. For example, a PHA
operating under a court order not to collect tenants' rent at
specific developments until it corrects deficiencies the court had
identified can seek to exempt those units in its developments from
being considered in its indicator score for rents uncollected. A PHA
always has the right to appeal a field office's decision about
modifications, exclusions, indicator scores, or the performance
designation. However, after those appeals have been exhausted, the
field office certifies the PHA's PHMAP score, assigns a final
performance designation, and proceeds with any required improvement
plans, MOAs, or other necessary follow-up.
When a troubled authority's new PHMAP score is high enough to cause
HUD to remove its troubled designation, HUD's policy is to require
the field office to verify the accuracy and completeness of the new
data submitted by the housing authority. HUD also requires the field
office to conduct a confirmatory review to verify the data the PHA
had certified as well as the accuracy of the data HUD had obtained
from its own information system. HUD's guidance for implementing
PHMAP stipulates that a confirmatory review must take place on-site
at the PHA and cannot be accomplished through remote monitoring.
HUD's field offices may choose to conduct some confirmatory reviews
of standard- and high-performing PHAs' PHMAP certifications. HUD
expects its field offices to choose these PHAs according to the risk
they pose and focus on those with the highest potential for fraud,
waste, mismanagement, or poor performance. Some of the factors HUD
field offices may consider in analyzing the risk associated with a
PHA's PHMAP certification include size (number of units), borderline
troubled designation (5 percent above or below the percentage for the
designation), and negative trends in overall or individual indicator
scores over several years.
In May 1995, HUD expanded the scope of the annual independent audit
each PHA receives in order to improve the Department's ability to
determine whether PHA-certified data are accurate. The annual audit,
conducted pursuant to the requirements of the Single Audit Act,
examines the housing authority's financial statements, internal
controls, and compliance with HUD's rules and regulations. Housing
authorities are responsible for selecting their own auditors and
submitting the results of the audits to their HUD field office.
Field offices are responsible for reviewing the audits to ensure they
meet all of HUD's requirements and, when they have approved the
audit, reimbursing housing authorities for them. In fiscal year
1995, these independent audits cost HUD about $8 million for all
housing authorities.
HUD now requires the independent auditors to determine whether a
housing authority has adequate documentation for the data it submits
to HUD for its PHMAP certification. According to HUD officials,
because the Department's resources are too limited to conduct annual
confirmatory reviews of most housing authorities, they expected to
use the results of these audits to better focus HUD's attention,
oversight, and technical assistance. In addition to paying for the
audits, HUD expects its field offices to use the results as part of a
risk assessment to determine which housing authorities should get the
most sustained attention and technical assistance.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:3
Stressing the need for HUD to hold housing authorities accountable
while making better use of the data that PHMAP produces, the Chairman
of the Subcommittee on Housing and Community Opportunity, House
Committee on Banking and Financial Services, asked GAO to review
HUD's use and implementation of PHMAP. As agreed with the Chairman's
office, we reviewed
-- whether HUD's field offices are using PHMAP and complying with
the program's statutory and regulatory requirements to monitor
and provide technical assistance to housing authorities,
-- whether PHMAP scores have increased and how HUD uses the program
to inform HUD's Secretary and the Congress about the performance
of housing authorities, and
-- whether PHMAP scores are consistently accurate and can be
considered a generally accepted measure of good property
management.
We developed information from several different sources to address
questions concerning the usefulness of PHMAP to HUD and other uses
for which PHMAP may not be appropriate. To determine PHMAP's
usefulness to HUD, we interviewed officials and collected information
on technical assistance activities at both the Department's
headquarters and field offices. At HUD's headquarters, we analyzed a
variety of documents pertaining to PHMAP and discussed the program's
use as a basis for technical assistance with the Offices of the
Deputy Assistant Secretaries under HUD's Assistant Secretary for
Public and Indian Housing.
At HUD's field offices, our approach was twofold. First, we surveyed
them via fax questionnaire to obtain data on the use of PHMAP, such
as the number of confirmatory reviews each field office performs and
how useful such program tools as improvement plans have been.\9 This
data reflect responses from all of HUD's public housing field
offices.\10 Second, we visited five HUD field offices to review their
use of PHMAP in more depth and to supplement the information we had
gathered in our survey.\11 We judgmentally selected the five field
offices because of their geographic distribution, variations in the
number of HUD staff in each office as well as the number of PHAs each
oversees, and variations in average PHMAP scores for the PHAs
reporting to those offices.
To provide information on PHAs' PHMAP scores, we relied on existing
data from HUD sources, including HUD's primary public housing
database, the System for Management Information Retrieval-Public
Housing (SMIRPH). From this database, we extracted the module
containing housing authorities' PHMAP data, including the PHMAP
scores and individual indicator grades. Our analysis covers federal
fiscal years 1992 through 1995 because the first fiscal year in which
the rules governing PHMAP took effect was 1992 and the most recent
year for which all PHMAP scores were complete at the time of our
review was 1995.
We did not systematically verify the accuracy of HUD's data or
conduct a reliability assessment of HUD's database. In performing
our analysis we found erroneous and incomplete information for a few
PHAs, ranging from 1 to 3 percent of the total. We confirmed this
with HUD officials, who attributed the errors to mistakes in data
input or the field office's having entered incomplete scores.
However, because we used these data in context with additional
evidence we obtained directly from HUD's field offices and we did not
focus on the scores of specific PHAs or small groups of PHAs, we
believe our conclusions about overall trends in scores are valid.
Throughout the course of our work, because the number of PHAs
reporting PHMAP scores is too great for us to visit a representative
sample, we consulted with several prominent groups representing the
public housing industry to discuss HUD's uses for PHMAP as well as
their perspectives on the program's ability to measure the
performance of public housing authorities. These groups include the
Council of Large Public Housing Authorities, the National Association
of Housing and Redevelopment Officials, and the Public Housing
Authorities Directors Association.
We provided a draft of this report to HUD for review and comment.
HUD's comments appear in appendix V and are addressed at the end of
each applicable chapter.
We performed our work from January through December 1996 in
accordance with generally accepted government auditing standards.
--------------------
\9 HUD headquarters does not collect or centrally maintain the
information for which we surveyed the field offices.
\10 Although 51 of HUD's 79 field offices have public housing
oversight responsibilities, 2 of those 51 have assumed the workload
of another field office due to attrition or temporary vacancies in
the public housing division. In these cases, the field office
handling the workload provided a combined response reflecting data
for both office's jurisdictions. As a result, our data is drawn from
49 responses, but it does reflect the PHMAP-related activities of all
of HUD's public housing field offices.
\11 We visited HUD's field offices in Birmingham, Alabama; Kansas
City, Missouri; Minneapolis, Minnesota; San Antonio, Texas; and San
Francisco, California.
ALTHOUGH FIELD OFFICES USE PHMAP
TO IDENTIFY TROUBLED PHAS,
COMPLIANCE WITH STATUTORY AND
AGENCY FOLLOW-UP REQUIREMENTS HAS
BEEN LIMITED
============================================================ Chapter 2
HUD's field offices use PHMAP scores for their primary intended
purposes: as a standard, objective means to identify troubled
housing authorities; to compare performance among PHAs; and to
identify when, where, and how to target HUD's limited resources for
technical assistance. However, beyond identifying troubled
authorities and what they need, the amounts and kinds of technical
assistance HUD provides varies because its field offices interpret
their responsibilities differently--some choose to be actively
involved while others adopt a hands-off approach. Furthermore, HUD's
1995 reorganization of its field offices adversely affected some
offices' ability to provide technical assistance while others adapted
to changed expectations and resumed providing as much assistance as
they did before the reorganization.
HUD USES PHMAP TO IDENTIFY
TROUBLED HOUSING AUTHORITIES,
BUT TECHNICAL ASSISTANCE VARIES
---------------------------------------------------------- Chapter 2:1
As part of HUD's oversight of public housing, the PHMAP score is an
important tool for identifying troubled authorities so HUD can focus
technical assistance and monitoring on them. The most common types
of technical assistance that HUD's 49 public housing field offices
provided all PHAs were telephone consultations, training, and
participation in conferences. However, we found differences in how
field offices defined their roles in providing PHAs technical
assistance as well as some innovations in how others provided that
assistance. For example, some field offices have encouraged
high-performing PHAs to provide "peer assistance" to lower
performers. Many of the differences in assistance were due to
variations in field offices' interpretations of their roles and the
impact of HUD's 1995 reorganization of its field offices. HUD
headquarters officials believe that more training for all field staff
and leadership from field office managers would help achieve more
quality and consistency among field offices in providing technical
assistance.
HUD USES PHMAP TO TARGET
PHAS FOR TECHNICAL
ASSISTANCE
-------------------------------------------------------- Chapter 2:1.1
Officials in 40 of HUD's 49 field offices rated PHMAP as being of
"utmost" or "major" importance in identifying which housing
authorities need the most technical assistance. According to field
office staff, PHMAP provides standard indicators to objectively
measure an authority's performance. In addition, some staff said
that because PHAs have a strong aversion to failing performance
scores and try to avoid failure, they are confident that when PHAs
report information that results in low scores or failing grades, the
data and the resulting scores are accurate. Because an accumulation
of low or failing scores results in a PHA's being designated
troubled, HUD staff are confident that those PHAs PHMAP identifies as
the worst-performing housing authorities are accurately designated as
troubled performers.
Some field office staff also use declining PHMAP scores to provide an
early warning of management problems and to identify which PHAs could
need additional technical assistance. In addition, the staff use
PHMAP's 12 individual indicator grades to better focus their limited
technical assistance resources and thereby maximize the benefits PHAs
receive from HUD's assistance. For example, one field office
developed a package of technical assistance for the "resident
initiatives" indicator because many PHAs failed this indicator. The
package of assistance included sample policies and procedures for
operating resident programs. Another field office developed
assistance specifically for small housing authorities because many of
them were having trouble renting their units when they became vacant
(thus failing PHMAP's unit turnaround indicator). Among other
things, that field office provided its small PHAs an extensive list
of suggestions on how and where to better market their units.
Most technical assistance from HUD's field offices consisted of
telephone consultations, training sessions, and industry conferences.
HUD also provided assistance--although limited because of time
constraints--at the time of a PHMAP confirmatory review. During
telephone consultations, several offices we visited answered
questions from housing authority staff and helped the executive
directors of new housing authorities better understand public housing
regulations and operations. Training sessions covered these and
other topics and provided more details than telephone discussions.
In addition, to increase the amount of personal contact they have
with housing authority staff and to provide technical assistance,
field office staff said they regularly participate in conferences
hosted by public housing industry associations.
FIELD OFFICES'
INTERPRETATIONS OF THEIR
ROLE AND THEIR RECENT
REORGANIZATION INFLUENCE THE
LEVEL AND TYPES OF TECHNICAL
ASSISTANCE
-------------------------------------------------------- Chapter 2:1.2
Field offices' interpretations of their obligation to improve the
performance of housing authorities influences the type of technical
assistance they provide. For example, officials in one field office
did not believe that it was HUD's role to manage PHAs' operations.
Instead, they believed that the role of their field office should be
limited to providing information on compliance with federal rules and
regulations and to suggesting solutions to management problems. This
field office avoids showing PHAs how to manage their developments
because the staff believe that they do not have sufficient expertise
and that the housing authorities would view this advice as intrusive.
In contrast, staff at other field offices that we visited believed
they are obligated to tell PHAs what must be done to correct
management deficiencies because HUD is responsible for ensuring that
PHAs use federal funds efficiently and effectively to provide safe,
decent housing. For example, staff from one field office spent
several days at a troubled authority to help it set up proper tenant
rent records and waiting lists.
In addition to differences in how they view their role to directly
assist PHAs, we found differences in the extent to which field
offices use outside resources to help their housing authorities.
Some field offices told us that to compensate for a shortage of
resources from HUD, they help PHAs in their jurisdiction by
encouraging technical assistance from other PHAs rather than
providing it themselves. For example, some of the field offices
arranged for high-performing PHAs to provide peer assistance to
authorities with management problems. One field office persuaded
staff from a high-performing PHA to temporarily manage a small
authority that unexpectedly lost its executive director. Another
field office recruited a high-performing PHA to help another one
develop a system for inspecting its housing units.
In 1995, HUD reorganized the field offices and changed the
responsibilities of the staff who oversee and assist PHAs. Before
the reorganization, most field office staff were generalists and
broadly understood federal housing regulations and PHA operations.
After the reorganization, however, the responsibilities of individual
field office staff became more specialized to focus on the rules and
regulations of specific public housing operations.\1 This
specialization confused some staff in field offices and housing
authorities as well as impaired the ability of some field offices to
provide technical assistance. For example, field office staff we
visited said that some specialists do not have the skills needed to
do their jobs because many of them did not have the work experience
or requisite training for the specialists' positions; the staff also
noted that HUD had not provided sufficient training for the staff to
understand the reorganization and their new responsibilities. The
staff also said that the reorganization was a source of confusion for
PHAs. Before the reorganization, a housing authority could call one
employee at HUD's field office to answer all its questions;
afterward, a housing authority generally needed to call several
different staff at HUD's field office to answer questions.
Adjusting to the reorganization differed across field offices. At
one field office, staff resisted the reorganization because they did
not want to become specialists and they recognized that technical
assistance to the PHAs suffered as a result. For example, the staff
now disagree over who is responsible for overseeing certain PHA
operations. They also have resisted working together to provide
technical assistance and have not been sharing PHMAP information to
develop the best plan for correcting management deficiencies. Other
field offices we visited adapted to the reorganization. Staff in
these field offices worked cooperatively to build on the skills of
the experienced staff. For example, one field office continues to
assign each housing authority to only one staff member who provides
or coordinates all technical assistance to that authority. The
responsible staff member, however, belongs to a team of staff from
all operational areas who work together to solve each PHA's problems.
Officials at HUD headquarters, including the Deputy Assistant
Secretary for Public and Assisted Housing Operations, acknowledged
that some field offices had difficulty adjusting to the
reorganization. They stated that although adequate training was
crucial to the reorganization's success, some field offices either
did not seek it or did not take the need for it seriously, despite
the availability of training funds for field staff. HUD officials
continue to emphasize the importance and availability of training and
expect field office management to assess the staff's skills and
expertise and request the appropriate training. These officials
believe that because of limited staff resources, now and in the
future, the reorganization is the best way for field offices to
provide effective oversight and technical assistance to PHAs.
Furthermore, they believe that managers of the field offices must
take a more active leadership role in directing their staff to work
together.
--------------------
\1 The five areas in which field office staff now specialize are
finance and budget; marketing, leasing, and management; facilities
management; community relations and involvement; and organization,
management, and personnel.
HUD'S INFREQUENT USE OF SOME
OVERSIGHT TOOLS MAY NOT
ADEQUATELY IMPROVE THE
PERFORMANCE OF PHAS OR TARGET
TECHNICAL ASSISTANCE
---------------------------------------------------------- Chapter 2:2
The act and HUD's requirements for how field offices use PHMAP
provide for several tools to guide improvements in a housing
authority's performance and thereby raise its indicator grades and
PHMAP score. These tools include the memorandums of agreement (MOA),
improvement plans, confirmatory reviews, and the annual independent
audits. While such tools as MOAs and improvement plans generally
apply to PHAs designated as troubled or failing specific indicators,
a confirmatory review is mandatory for any PHA coming off HUD's
troubled list and an independent audit is mandatory for all PHAs.
Nonetheless, we found that the compliance of field offices with
statutory requirements and HUD's guidance for using these tools has
been inadequate and infrequent.
Furthermore, HUD has not determined whether these statutory or agency
requirements are effective, adequately improve housing authority
performance, or help the field offices better target limited
technical assistance resources. As a result, HUD has little
information to determine which of these tools best improve a PHA's
performance and which tools its field offices can use most
effectively to offset their declining resources.
FIELD OFFICES MAKE LIMITED
USE OF OVERSIGHT TOOLS
-------------------------------------------------------- Chapter 2:2.1
Over 90 percent of the field offices we surveyed reported that
on-site visits to the housing authorities were one of the most
effective means to ensure compliance with PHMAP requirements and
provide technical assistance. Officials at one field office
responded that PHAs under its jurisdiction believed that on-site
visits from HUD staff to provide technical assistance were essential
to maintaining effective operations. Yet, most field office staff we
visited made fewer personal visits to housing authorities than they
felt were necessary because of limited staff resources and travel
funds. Field office staff told us, for example, that their workload
has increased because their offices have been unable to replace staff
who have left the agency. With less time available for on-site
visits, direct monitoring of the PHAs' performance has occurred less
frequently. In addition, some field office staff said that they
could rarely justify to their management using limited staff and
travel resources to visit a PHA that is more than a 1-day trip from
the office unless that authority's PHMAP score was below 60.
MEMORANDUMS OF AGREEMENT
-------------------------------------------------------- Chapter 2:2.2
Although HUD is required by law to enter into MOAs with troubled
housing authorities to improve management performance, few field
offices have done so. Figure 2.1 shows that the percentage of
troubled PHAs operating under an MOA has been decreasing since 1992.
Furthermore, in fiscal year 1995, only 3 of HUD's 32 field offices
that had troubled PHAs were fully in compliance with the requirement
to enter into an MOA with each troubled authority.\2
Figure 2.1: Percentage of
Troubled PHAs Operating Under a
MOA, Fiscal Years 1992-95
(See figure in printed
edition.)
Source: HUD field offices' responses to GAO's questionnaire.
The primary reason HUD's field offices told us that they did not
enter into these required agreements with troubled housing
authorities is that the PHAs had already corrected or were in the
process of correcting their management deficiencies. However, HUD
headquarters officials told us they did not accept this as a valid
reason for not meeting the requirement and questioned how the field
offices could be sure the housing authorities were no longer
troubled.
--------------------
\2 For fiscal year 1995, the 49 field offices responded that a total
of 71 housing authorities were troubled, but only 13 had MOAs.
However, HUD's PHMAP database reports 150 housing authorities as
troubled in fiscal year 1995. HUD headquarters officials told us
that our survey information was probably more accurate than its
database.
IMPROVEMENT PLANS
-------------------------------------------------------- Chapter 2:2.3
When a PHA fails any of PHMAP's 12 performance indicators, HUD
requires the responsible HUD field office to obtain a plan from that
PHA for improving its performance and to track its progress against
the plan. However, we found that nearly a third--31 percent--of
HUD's field offices had not ensured that local housing authorities
had developed these plans. We also found examples of PHAs' plans
lacking specific strategies and time frames for correcting management
deficiencies. For example, one PHA's plan for a failing "rents
uncollected" indicator simply stated that the housing authority would
start collecting rent. Although field office staff acknowledged that
the PHA also needed to update its standard tenant lease and develop a
rent collection policy to improve this indicator grade, they said
that they had not yet had the time to contact the PHA to revise its
plan.
HUD requires its field offices to monitor the progress of housing
authorities in implementing improvement plans to ensure PHAs meet the
quarterly and annual performance targets in their plans. However,
four of the five field offices we visited told us they do not follow
up with the PHAs to determine the status of improvement plans or
whether the plans had corrected the management deficiencies. Field
office staff said that they did not have time to track the
effectiveness of the plans because their workloads have been
increasing due to decreasing numbers of staff.
HUD headquarters officials confirmed that systematic tracking of the
field offices' success in obtaining improvement plans or executing
MOAs has not been done. They emphasized that responsibility for
implementing PHMAP rests with the field offices and said that limited
efforts were underway to ensure field offices do more to use these
tools and measure their effectiveness. However, they could not tell
us whether troubled PHAs without MOAs had improved their scores and
left the troubled list without such oversight, nor could they tell us
whether improvement plans are instrumental in improving indicator
scores.
FIELD OFFICES CONFIRM FEW
PHMAP SCORES
------------------------------------------------------ Chapter 2:2.3.1
When a troubled housing authority receives a new PHMAP score that is
high enough to remove that designation, HUD requires that the field
office confirm the score's accuracy by verifying that the PHA's
improvements have been effective before removing the troubled
designation. However, we found most field offices are not meeting
this requirement. In 1995, for example, HUD's field offices
confirmed less than 30 percent of the scores that should have been
confirmed. HUD officials acknowledged that the infrequency of
confirmatory reviews by its field offices hampers the program's
credibility and integrity. Because it has done so few confirmatory
reviews, HUD cannot say that most scores are accurate, nor can it say
that most troubled PHAs that raised their scores above 60 really are
no longer troubled. The HUD Inspector General (IG) recently noted
that without more confirmatory reviews, the self-reporting nature of
PHMAP creates a temptation for PHAs to manipulate data to raise their
scores.\3
In fiscal year 1995, 24 of the 49 field offices had housing
authorities with PHMAP scores high enough to remove them from HUD's
troubled list, but only 11 of the 24 field offices performed all or
some of the required confirmatory reviews.\4 The remaining 13 offices
performed none of the required confirmatory reviews. Nonetheless,
some of these same 13 field offices performed discretionary
confirmatory reviews of other housing authorities that had not been
classified as troubled. In one case, a field office had just one
housing authority whose new PHMAP score was high enough to remove its
troubled designation. Although the field office did not perform a
confirmatory review for that authority until the next fiscal year, it
did complete nine confirmatory reviews of standard- or
high-performing housing authorities. HUD headquarters officials told
us that although they encourage the field offices to do as many
additional, discretionary confirmatory reviews as possible, they
expect field offices to complete the mandatory reviews first. They
also told us that limited resources kept them from monitoring the
performance of field offices on these reviews.
In addition to the field offices' lack of compliance with HUD's
requirement for performing confirmatory reviews, few offices are
performing discretionary confirmatory reviews. Over the life of the
program, HUD has confirmed 6.7 percent of all PHMAP scores. Table
2.1 shows that since the program began in 1992, HUD has confirmed no
more than 8 percent of all PHMAP scores in any given year (see table
2.1).
Table 2.1
PHAs Receiving a Confirmatory Review
From a HUD Field Office, Fiscal Years
1992-95
1992 1993 1994 1995
---------------------- ---------- ---------- ---------- ----------
Number of PHAs 2,372 3,071 3,071 3,077
submitting PHMAP
certifications
Number of confirmatory 137 225 241 200
reviews completed
Percentage of PHAs 5% 7% 8% 6%
receiving a
confirmatory review
----------------------------------------------------------------------
To expand on fiscal year 1995 data, nine field offices performed no
confirmatory reviews, over two thirds performed five or fewer, and 4
offices performed 10 or more confirmatory reviews (see fig. 2.2).\5
Figure 2.2: Number of
Confirmatory Reviews Performed
by HUD Field Offices, Fiscal
Year 1995
(See figure in printed
edition.)
Source: HUD field offices' responses to GAO's questionnaire.
Recognizing that PHMAP scores may not be as accurate as they could or
should be to give the program integrity and credibility, HUD has
added new requirements and begun initiatives to improve the accuracy
of the scores and strengthen the program. HUD currently requires its
field offices to confirm the PHMAP scores of housing authorities
whose scores have risen to 60 or above, thereby removing them from
the troubled list. Recently, HUD formed a team of "expert" field
office staff to develop review guidelines and to perform confirmatory
reviews at selected housing authorities whose new PHMAP scores meet
HUD's criteria for a mandatory confirmatory review. HUD officials
expect this team to perform as many as 12 confirmatory reviews in 1
year, during which they will focus primarily on large, high-risk
housing authorities.
--------------------
\3 Limited Review of the Public Housing Management Assessment Program
(Audit Related Memorandum No. 96-PH-101-0801), July 1996, HUD Office
of Inspector General.
\4 In fiscal year 1995, a total of 58 PHAs reported new PHMAP scores
that would have taken them off HUD's troubled list. HUD's field
offices confirmed 15 (26 percent) of these 58 PHMAP scores.
\5 HUD's Oklahoma City Field Office reported having one staff member
and performing no confirmatory reviews in fiscal year 1995. We did
not include that office's response in our analysis, however, because
its responsibilities for overseeing public housing had been
transferred to HUD's Ft. Worth office after the bombing of the
Oklahoma City Federal Building.
FIELD OFFICES ARE NOT
USING INDEPENDENT AUDITS
TO VERIFY DATA PROVIDED
BY PHAS
------------------------------------------------------ Chapter 2:2.3.2
In May 1995, HUD expanded the scope of the mandatory annual financial
audits of PHAs to require that auditors review the records underlying
a PHA's self-reported PHMAP data.\6 HUD expects the financial audits
to verify that the PHAs' data are accurate and complete and that the
PHAs have adequate documentation to support their submissions. HUD
adopted this requirement because the field offices do not have
sufficient resources to confirm each PHA's score every year.
Moreover, HUD officials told us that further departmental downsizing
will limit its field offices' ability to provide meaningful technical
assistance, including confirmatory reviews. As a result, HUD expects
that the PHMAP review in the annual audit can help ensure the
integrity of housing authorities' PHMAP data and should be a valuable
tool for aiding the field offices to identify those housing
authorities most needing technical assistance.
HUD does not consider the auditors' analysis to be a confirmatory
review because the auditors do not verify the information HUD
maintains in its information system. Furthermore, even though the
auditors certify that a housing authority has documentation to
support the data it submitted to HUD, they do not verify that some of
the activities reflected in that data were actually performed by that
authority. For example, while the auditors verify that a PHA has
data indicating it has met the requirements for the indicator on
conducting annual inspections of all of its housing units and major
systems (e.g., heating, plumbing, and electrical), the auditors do
not verify that those inspections actually took place.
Although the independent audit requirement has been in place since
May 1995, few of the staff in the five field offices we visited were
aware of it. Before field offices authorize payment for an annual
audit, HUD headquarters officials said that they expect field offices
to review the audits for quality and completeness and verify that the
audits addressed all appropriate areas of the PHAs' operations,
including the PHMAP. However, field office staff said that they had
not seen an audit of a housing authority that tested the reliability
of its PHMAP submission.\7 HUD also expects the field offices to
consider significant audit findings in deciding which PHAs need
additional oversight or assistance. HUD officials acknowledged,
however, that the independent auditors may need training to better
understand HUD's expectations of them, regulations, and PHMAP system
as well as the operations of PHAs. Similarly, these officials noted
that staff in HUD's field offices need training and guidance in how
to better use the annual independent audit.
--------------------
\6 Annually, HUD requires and pays for a single financial and
compliance audit of every public housing authority.
\7 HUD's IG evaluated some housing authorities' annual independent
audits that included a PHMAP review. The IG found that the auditors
had problems confirming the reliability of the PHMAP submissions
because the PHAs lacked documentation to support the self-reported
indicator grades.
CONCLUSIONS
---------------------------------------------------------- Chapter 2:3
One of the key challenges HUD faces in the coming years is
effectively downsizing the Department while maintaining the needed
level of oversight at public housing authorities. However, HUD is
currently not maintaining a consistent, minimally acceptable level of
oversight at all housing authorities because of the variance in how
field offices interpret their roles to provide that oversight as well
as their lack of systematic compliance with follow-up requirements.
Furthermore, because field offices are not making enough use of the
independent audits' verification of PHMAP data to target their
technical assistance, HUD is not using the resources it has to
effectively determine which housing authorities' scores are most
likely to be inaccurate. As a result, HUD is not ensuring that the
housing authorities most in need of oversight and assistance are
receiving it and thereby improving their performance. Continued
departmental downsizing likely will cause HUD to leverage its
existing resources to achieve a minimally acceptable level of
oversight. This oversight is needed for HUD to be reasonably
confident that all housing authorities are using federal funds
appropriately, managing and maintaining their developments properly,
and reporting accurately their performance information.
RECOMMENDATION
---------------------------------------------------------- Chapter 2:4
To make better use of the limited resources it has to devote to the
oversight of public housing, we recommend that HUD provide guidance
to its field offices that clearly (1) articulates their minimally
acceptable roles regarding oversight and assistance to housing
authorities and (2) emphasizes the importance of using the results of
the independent audits to better target HUD's limited technical
assistance resources.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 2:5
HUD agreed with our findings regarding oversight of public housing
authorities and stated that it has begun taking steps to address this
recommendation. These steps include a wide variety of training and
other activities to (1) explain the revisions HUD is making to PHMAP;
(2) reemphasize the need for and importance of statutory and agency
follow-up requirements, such as memorandums of agreement, improvement
plans, and confirmatory reviews; and (3) update HUD's guidance to its
field offices regarding their PHMAP and other oversight
responsibilities.
ALTHOUGH PHMAP SCORES HAVE RISEN,
HUD RECOGNIZES THAT FLAWS IN THE
PROGRAM'S DATABASE LIMIT ITS USE
============================================================ Chapter 3
According to a HUD database of PHMAP scores, average PHMAP scores
have increased over the life of the program from an average of 83 in
1992 to 86 in 1995 (the last year of complete data). The number of
high-performing housing authorities increased, with more than half of
all authorities designated high performers in 1995, and the number of
troubled authorities decreased. However, the smallest housing
authorities--those with fewer than 100 units--now make up a greater
proportion of those designated troubled than when the program began.
During our analysis of this database, we found omissions of key data,
such as the number of units under a PHA's management and its
performance designation. We also found inconsistencies between PHMAP
scores and the assigned performance designations. Notwithstanding
these weaknesses, the database represents the most complete data
available on PHA performance over time.
MOST PHMAP SCORES ARE
INCREASING AND FEWER HOUSING
AUTHORITIES ARE TROUBLED
---------------------------------------------------------- Chapter 3:1
Nationwide, average PHMAP scores generally increased over the 4 years
of the program for which we analyzed data. By 1995, over half of all
public housing authorities were high performers. Subsequent analysis
showed little regional variation in how well they scored on PHMAP.
While the overall increases in PHMAP scores held true for all sizes
of PHAs, the largest ones had scores consistently lower than the
national average. With average scores increasing, the number of PHAs
with scores low enough for HUD to designate them as troubled also
decreased. The number of troubled authorities reached 83 in 1995,
with half of that total consisting of the smallest housing
authorities (those managing fewer than 100 units).
AVERAGE PHMAP SCORES
INCREASED
-------------------------------------------------------- Chapter 3:1.1
The average PHMAP score for all housing authorities rose from about
83 in 1992 to 86 in 1995. This increase held true for PHAs of all
sizes, although large PHAs--those with more than 1,250
units--consistently scored lower than the national average (see table
3.1). In fiscal year 1995, 151 large PHAs accounted for
approximately 5 percent of all PHAs reporting PHMAP scores, but they
operated nearly 60 percent of all public housing units.
Consequently, while more PHAs had higher scores, more units were
under the control of PHAs with somewhat lower scores.
Table 3.1
Average PHMAP Score by PHA Size
Category, Fiscal Years 1992-95
1992 1993 1994 1995
------------------ ------------------ ------------------ ------------------
PHA size
category
(number
of Number PHMAP Number PHMAP Number PHMAP Number PHMAP
units)\a of PHAs score of PHAs score of PHAs score of PHAs score
--------- -------- -------- -------- -------- -------- -------- -------- --------
No size 4 54 4 43 3 41 7 31
data\b
1-99 1,453 83 1,471 84 1,481 87 1,488 87
100-499 1,241 83 1,262 85 1,266 88 1,269 87
500- 243 83 242 84 243 86 242 86
1,249
1,250 or 149 78 151 80 151 81 151 83
more
All sizes 3,090 83 3,130 84 3,144 87 3,157 86
-----------------------------------------------------------------------------------------
\a All size categories were calculated on the number of managed units
for fiscal year 1995. HUD does not maintain information on the
number of units managed in previous years. To the extent that
specific PHAs reduced or increased the number of units under their
jurisdiction in previous years, the current size categories may not
represent previous years.
\b The database did not contain size information for these PHAs.
Source: GAO's analysis of data from HUD's System for Management
Information Retrieval-Public Housing (SMIRPH) database.
Appendix I provides average PHMAP scores for PHAs for all of HUD's
field offices for fiscal years 1992 through 1995.
THE MAJORITY OF PHAS WERE
HIGH PERFORMERS
-------------------------------------------------------- Chapter 3:1.2
By fiscal year 1995, more than half--about 57 percent--of all public
housing authorities were designated as high performers. As shown in
table 3.2, the number of high performing authorities grew each year,
rising from 1,033 (33 percent) in 1992 to 1,791 (57 percent) in 1995.
Also, by 1995, nearly 50 percent of all public housing units were
under the management of high-performing authorities.
Table 3.2
Number of PHAs by PHMAP Performance
Category, Fiscal Years 1992-95
Number of PHAs
--------------------------------------
Units
(in
Performance category thousand
(score) s)\a 1992 1993 1994 1995
-------------------- -------- -------- -------- -------- --------
Troubled (<60) 214 130 118 101 83
Standard (60-<90) 443 1,927 1,719 1,358 1,216
High (90-100) 656 1,033 1,293 1,685 1,791
All performance 1,313 3,090 3,130 3,144 3,090
categories
----------------------------------------------------------------------
\a HUD's SMIRPH database contains the number of units for only fiscal
year 1995.
Source: GAO's analysis of data from HUD's SMIRPH database.
LITTLE VARIATION AMONG
REGIONS
-------------------------------------------------------- Chapter 3:1.3
Our analysis showed little regional variation in PHMAP scores. The
regional differences we found were slightly greater than those
associated with the size of housing authorities, but no region was
significantly below the national average. Likewise, there was little
variation among the regions in the percentage of troubled PHAs under
their jurisdiction. For example, in fiscal year 1995, 5 percent of
all PHAs nationwide were troubled, but within the 10 regions we
analyzed, the percentage of troubled housing authorities ranged from
2 to 9 percent.
Appendixes I-IV provide detailed information on average PHMAP scores
as well as the number of troubled, standard- and high-performing
PHAs, respectively, for each HUD field office.
PHAS CONSISTENTLY FAILED
SOME INDICATORS
-------------------------------------------------------- Chapter 3:1.4
Despite some improvement in overall scores, some indicators were more
problematic for PHAs than others. As shown in table 3.3, with the
exception of 1 year, PHAs consistently had the most difficulty with
the energy consumption indicator--which had the highest failure rate
for 1992, 1994, and 1995.\1 Similarly, the indicators for unit
turnaround, tenants accounts receivable, and operating expenses
proved troublesome, with 10 percent or more of all PHAs failing them
in 1995.
Table 3.3
Failed Indicators by Percentage of PHAs,
Fiscal Years 1992-95
Percentage of PHAs that failed
----------------------------------------------
PHMAP indicator 1992 1993 1994 1995
---------------------- ---------- ---------- ---------- ----------
1. Vacancy number and 5.5 4.2 3.6 2.9
percentage
2. Modernization 1.9 1.4 1.4 1.5
3. Rents uncollected 4.1 3.1 3.0 2.2
4. Energy consumption 18.0 12.4 14.3 15.3
5. Unit turnaround 16.4 13.6 12.9 11.2
6. Outstanding 3.6 2.1 2.4 1.7
workorders
7. Annual inspection 2.2 2.3 2.1 1.9
and condition of
units and systems
8. Tenants accounts 16.9 15.2 14.2 12.8
receivable
9. Operating reserves 6.3 5.8 4.8 7.4
10. Routine operating 11.2 10.8 10.1 10.0
expenses
11. Resident 13.4 26.7 11.0 5.9
initiatives
12. Development 8.5 6.2 6.2 3.0
----------------------------------------------------------------------
Source: GAO's analysis of data from HUD's SMIRPH database.
A HUD official explained that the high failure rate in 1993 for the
indicator measuring resident initiatives occurred because the PHAs
were not paying attention to this indicator. In 1992, all PHAs
received an automatic "C" for this indicator because HUD had not
provided enough information on the requirements for grades "A"
through "F" until after the assessment period started. This official
said that many PHAs assumed they would receive an automatic "C" the
next year as well, even though HUD had stated in 1992 that the
automatic grade was a one-time occurrence. This official added that
most field offices followed up by providing technical assistance to
the PHAs with failing grades and were able to resolve the problems in
the following year. This appears to be supported by the decline of
the failure rate over the following 2 years to less than 6 percent in
1995.
--------------------
\1 This indicator measures the annual increase in the housing
authority's energy consumption. Housing authorities with no increase
from year to year receive an A for the indicator; those whose
consumption increases receive lower grades. In commenting on this
report, HUD officials noted that in some cases failing this indicator
was related less to PHA performance than to such conditions as
regional weather variations or an inappropriate baseline to measure
this indicator.
SMALLER PHAS WERE MORE
LIKELY TO BE TROUBLED
-------------------------------------------------------- Chapter 3:1.5
While the total number of troubled housing authorities declined--130
were troubled in 1992 compared to 83 in 1995--more of those PHAs were
concentrated among the smallest housing authorities than when the
program began. The percentage of troubled PHAs that were
small--managing fewer than 100 units each--grew from 32 percent of
all troubled authorities in 1992 to 49 percent in 1995 (see fig.
3.1).
Figure 3.1: Number of Troubled
PHAs by Size, Fiscal Years
1992-95
(See figure in printed
edition.)
Note: For each fiscal year, the figures exclude six or fewer PHAs
for which there was no information on size in HUD's database.
Source: HUD's SMIRPH database.
HUD RECOGNIZES DATABASE FLAWS
AND PLANS CORRECTIONS
---------------------------------------------------------- Chapter 3:2
We found missing, inaccurate, and inconsistent data in HUD's SMIRPH
database, the primary database for storing PHMAP scores. A HUD
official attributed these problems to data input problems at the
field offices. Although HUD headquarters makes regular, periodic use
of this database, it must also manually verify much of the
information before providing it to HUD's Secretary, Members of
Congress, and others. HUD's General Deputy Assistant Secretary for
Public and Indian Housing acknowledged that the SMIRPH database, as
currently implemented, does not produce a complete, accurate list of
troubled PHAs and that HUD is in the process of making it more
reliable and useful.
We found that the number of troubled authorities (150) for fiscal
year 1995 that we derived from the database was inaccurate when we
compared it to the number reported (83) as of December 20, 1995, by
HUD's Management Assessment Coordinator. We also found performance
designations that were inconsistent with PHMAP scores. In 1995, for
the 150 PHAs we found to be troubled, HUD had designated 42 as high
performers, 7 as standard, and 51 had no designation. Among
high-performing PHAs in 1995, of the 1,791 PHAs that we found that
had PHMAP scores of 90 or higher, HUD had designated one as troubled,
43 as standard, and 325 had no performance designation. We also
found some omissions in the database. Data, such as the number of
units and performance designations, had not been entered for all
PHAs. For example, we found that the database did not have size
information on 18 PHAs from fiscal years 1992 through 1995. We also
found that no designations had been entered for 132 PHAs with scores
less than 60 and 1,037 PHAs with scores 90 or higher.
HUD's Management Assessment Coordinator stated that these problems
with missing, inaccurate, and inconsistent data occurred because
field offices either (1) did not enter the information at all or (2)
entered it incorrectly. These instances of inconsistent or missing
data suggest that basic system safeguards do not exist to prevent
field offices from making these data entry errors or omitting
essential PHMAP data.
While HUD officials who oversee PHMAP and the Department's field
offices acknowledged problems with the database, they added that the
program's redesign includes changes that will address the problems
with data accuracy and reliability. HUD officials told us they plan
to change procedures for entering information on PHAs into the
database to allow field offices to update PHA data on a real-time
basis and to make immediate corrections when they find errors or
omissions. These procedural changes will also enable HUD
headquarters staff to access field office data directly and allow
ongoing reviews of the information for accuracy and completeness.
HUD officials also believe that the changes will increase control
over the information from the field offices and help ensure that the
information in the SMIRPH database is accurate.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 3:3
HUD expressed concern that our draft report used data from the SMIRPH
database that HUD had not verified for accuracy. HUD noted that it
is making changes to the database that will improve headquarters'
ability to find and correct data errors that have been entered by
staff at its field offices.
To address HUD's concern that we used inaccurate, unverified data
from its database to analyze PHMAP data on housing authorities'
scores by size and region, we recalculated the number of troubled
housing authorities by size category for 1995 using data HUD verified
with its field offices; we also modified this report to reflect a
more accurate and lower number of troubled housing authorities in
1995. Recalculating the number of troubled authorities by size did
not change our conclusion that a greater proportion of the
authorities that HUD verified as being troubled are those with fewer
than 100 units. In fact, while HUD's database indicates that 44
percent of troubled authorities in 1995 were small, HUD's verified
list of troubled authorities indicates 49 percent were small.
Furthermore, although HUD officials told us that a manually-verified
list of troubled authorities for 1992 was not available, they agreed
with our conclusion that the smallest housing authorities make up a
greater proportion of troubled housing authorities in 1995 than in
1992.
Because our draft report presented no analysis of data on a regional
basis (only data as drawn from HUD's database) and because we draw no
conclusions in that regard in this report, we have retained
appendixes I-IV, which show average PHMAP scores and the number of
troubled, standard, and high-performers in HUD's regions. Where HUD
provided us with manually verified data--particularly in appendix II
showing troubled authorities--we have modified the appendixes to
reflect the more accurate data.
THE QUESTIONABLE ACCURACY OF
PHMAP'S SCORES AND THE PROGRAM'S
VALIDITY LIMIT ITS USEFULNESS
============================================================ Chapter 4
Our review and those of others indicate that PHMAP scores are often
inaccurate, imprecise, and must be changed when HUD verifies the data
that public housing authorities have submitted to support their
scores. Furthermore, professional property managers and others in
the public housing industry question whether PHMAP can capture all
aspects of management operations. Although HUD has taken some steps
to help ensure that future scores are more accurate than they have
been over the program's first 4 years, these steps will be
resource-intensive and do not address all of the program's
limitations. In the past, both HUD and the Congress have proposed
additional uses for PHMAP, such as deregulating and awarding bonuses
to PHAs with high PHMAP scores. However, until greater confidence
exists that individual scores are accurate and HUD brings greater
validity to PHMAP as a comprehensive measure of management
operations, such additional uses for the program may not be
appropriate.
ACCURACY OF SCORES AND VALIDITY
AS A MANAGEMENT ASSESSMENT TOOL
LIMIT USES FOR PHMAP
---------------------------------------------------------- Chapter 4:1
After performing on-site reviews of selected PHAs to confirm the
accuracy of their PHMAP scores, HUD's field offices changed half of
the scores. In commenting on this report, HUD indicated that most
confirmatory reviews involved high-risk PHAs, whose PHMAP data have
been most susceptible to being found inaccurate. In similar reviews,
HUD's independent assessment contractors as well as HUD's IG found
that many scores or grades for specific indicators were inaccurate.
To better identify PHAs that need oversight and technical assistance,
HUD staff often supplement their decision-making with other measures
of management problems to get a more complete picture of an
authority's performance. Professional property managers and industry
representatives agreed that more information is needed than PHMAP
provides to give a complete picture of how well a PHA's management is
performing.
AFTER CONFIRMATORY REVIEWS,
PHMAP SCORES CHANGE
SIGNIFICANTLY
-------------------------------------------------------- Chapter 4:1.1
After performing confirmatory reviews of 200 PHAs in fiscal year
1995, HUD's 49 field offices changed 98 PHMAP scores (see table 4.1).
Table 4.1
Changes in PHMAP Scores After HUD's
Field Offices Performed Confirmatory
Reviews in Fiscal Year 1995
Number of housing
authorities
Change in (percentage of Average change
PHMAP score total) in points
------------------------------ ------------------ ------------------
No change 96 (49) 0
Scores lowered 57 (29) -14
Scores raised 41 (21) +8
----------------------------------------------------------------------
Note: The field offices had not reported the final PHMAP scores for
6 out of the 200 confirmatory reviews because the reviews' results
were being finalized at the time the offices responded to our
questionnaire. As a result, this table reflects results from 194 of
the 200 confirmatory reviews performed in fiscal year 1995.
In several cases, the changes HUD made to PHMAP scores also meant HUD
would have to change the performance designation of those PHAs. For
example, HUD
-- lowered the scores of 14 PHAs enough to designate them as
troubled,
-- raised the scores of 4 troubled PHAs to 60 points or higher, and
-- raised the scores of 10 standard-performing PHAs to 90 or
higher.
Both of HUD's independent assessment contractors as well as HUD's IG
have reviewed PHMAP data to confirm the accuracy of PHAs' scores.
For example, in 1993, the IG confirmed the scores of 12 housing
authorities. As a result of this review, the IG concluded that the
PHMAP scores for 9 of the 12 PHAs should be lowered because 3 of them
fell below 60, a score which should have warranted the troubled
designation. In a second report on PHMAP, the IG reported that six
of HUD's field offices reduced over half of the scores they reviewed.
Similarly, one of HUD's independent assessment contractors reported
that for the 30 assessments it has performed at troubled housing
authorities, it found 21 indicator grades and/or PHMAP scores that
were inaccurate. Over 50 percent of the contractor's assessments
resulted in lowering the indicator grades to an "F." The contractor
most often lowered the indicators used to measure outstanding
workorders and annual inspections of housing conditions and systems.
Several reasons explain why HUD and others changed so many PHMAP
scores after performing a confirmatory review. Some field office
staff said these scores changed because the PHAs did not understand
all the requirements of PHMAP and therefore misreported their data.
They also told us that PHMAP is particularly difficult for smaller
housing authorities whose limited staff can find HUD's paperwork
requirements overwhelming.
HUD staff do not believe many PHAs intentionally try to deceive the
Department by reporting false PHMAP information. Instead, they, as
well as the contractor staff, said that the PHAs often have
insufficient documentation to support the data they must submit to
the field offices or do not understand how HUD wants them to report
the information. For example, while a PHA may report the average
number of days their housing units have been vacant, the PHA may not
have the tenant files to document when the previous tenants moved out
and when the new tenants' leases took effect. Without supporting
documentation or evidence of a system to track unit turnaround, HUD
assigns an "F" to this indicator. Similarly, a PHA may be providing
support programs for its residents, but fail to understand that its
board of commissioners must approve those programs to receive a
passing grade on PHMAP's indicator for resident initiatives.
Typically, when HUD's field office staff find examples, such as
these, during a confirmatory review, they use the correct data to
recalculate the housing authority's grade for each of the affected
indicators.
HUD AND INDUSTRY
PROFESSIONALS SUPPLEMENT
PHMAP WITH ADDITIONAL
FACTORS TO EVALUATE
MANAGEMENT PERFORMANCE
-------------------------------------------------------- Chapter 4:1.2
HUD's field office staff did not use PHMAP alone to assess the
management performance of its public housing authorities. Although
they agreed that PHMAP accurately identifies troubled authorities,
several staff said that they consider other factors besides PHMAP
indicators to supplement their decision-making for the other
authorities they oversee. They said that some PHAs with scores over
90 have management problems that the program's indicators do not
measure. Other factors used by some HUD staff to identify the
potential for management problems at standard- and high-performing
authorities include
-- the failure of a PHA to implement consistent and effective
operating policies and procedures,
-- the frequency of changes in the executive leadership and the
continued interference into a PHA's daily operations by its
board of commissioners,
-- the number and the type of telephone calls received from a PHA's
residents and staff, and
-- any adverse news stories about a PHA.
Staff at the five field offices we visited said that they believed
some housing authorities with high PHMAP scores were not operating
their housing programs efficiently or effectively. These field
offices differed, however, in how they treated those PHAs. Staff at
two field offices told us that although they use the scores to
determine which PHAs need on-site reviews, they would not let a high
score prevent them from visiting an authority they believed had
serious management problems.
The HUD IG also questioned whether or not PHMAP scores accurately
measure the management performance of public housing authorities.
The IG's reviews of high- and standard-performing PHAs found
instances of fraud and program abuse. For example, the IG reported
that the executive director of a high-performing PHA had charged over
$62,000 in ineligible expenses, including excessive compensatory
time, unsupported travel costs, and health and insurance benefits for
his divorced spouse. Another PHA executive director falsified PHMAP
data to obtain a high-performing designation. After reviewing the
operations of a standard-performing PHA, the IG also cited numerous
program abuses and mismanagement. The IG concluded that although
PHMAP could be a useful tool to assess PHAs, the program was too
unreliable for HUD to make oversight decisions.
Other public housing professionals--property managers and those
representing industry associations--agreed that more information is
needed than PHMAP provides to give a complete picture of how well a
PHA is managed. For example, they noted that PHMAP does not
automatically include an on-site observation and inspection of a
PHA's housing developments. One association noted that while a PHA
could improve its PHMAP score by simply writing off more past due
rents from former tenants as uncollectible to improve its grade on
the indicator for rents uncollected, its PHMAP score would not
measure how diligent an effort it had undertaken to collect the rent.
Another industry association official knew of several examples of
PHAs that were making good property management decisions, such as
choosing to perform deferred maintenance when a unit became vacant
rather than rent it immediately, that ironically led to lower PHMAP
scores. Citing a similar situation, HUD has agreed that occasionally
the best decision for a PHA is to take an action that yields a lower
PHMAP score, and that the score should not be the sole driving force
influencing a PHA's decisions.
THE CONGRESS AND HUD HAVE
PROPOSED TO USE PHMAP AS A
BASIS FOR DEREGULATION AND
FUNDING BONUSES
---------------------------------------------------------- Chapter 4:2
While HUD's primary use of PHMAP has been to identify troubled
housing authorities and target technical assistance to them, the
Congress and HUD have proposed to use this program for other
purposes. In 1994, the Senate Committee on Banking, Housing, and
Urban Affairs proposed some deregulation and additional flexibility
for those authorities that had achieved PHMAP scores of 90 or above.
In addition, in its fiscal year 1997 budget request, HUD proposed to
give high-performing PHAs bonuses based in part on their PHMAP
scores. Because PHMAP scores do not always measure the true
management performance of the PHAs, the benefits of these proposals
need to be weighed against the possibility of granting undeserved
flexibility and awards.
To encourage individual PHAs to be more innovative, the Banking
Committee proposed limited deregulation and additional flexibility
for high-performing PHAs in two ways. First, it proposed permitting
a PHA that generates income over a certain level to exclude that
income from calculations of its need for a subsidy from HUD to
operate and manage its properties.\1 At that time, each dollar of
extra income that a PHA generated reduced its subsidy by a dollar,
thereby creating a disincentive to generate additional income from
sources other than rent. Second, the Committee proposed to waive all
but a few key regulations--such as nondiscrimination, equal
opportunity, and tenant income eligibility--so high-performing PHAs
could have more flexibility to bring innovative solutions to local
problems and achieve more efficient operations.
In its fiscal year 1997 budget request, HUD proposed to award $500
million to high-performing PHAs as bonuses based, in part, on their
PHMAP scores. As we reported in our testimony in June 1996 and as we
found in the course of our work on this report, HUD does not confirm
the scores of high performers and generally accepts them.\2 In our
June 1996 testimony, we recommended that the Congress consider not
appropriating the bonus funding until HUD develops adequate
performance measures and supporting information systems. The HUD
appropriations bill which the Congress approved and the President
signed did not contain funding for performance bonuses.
The three associations representing the public housing industry and
the professional property managers that we interviewed all opposed or
had strong reservations about using PHMAP scores for purposes other
than identifying troubled housing authorities and targeting technical
assistance to them. They also believed that other uses would be
inappropriate because of the limited number of confirmatory reviews
the field offices perform and the proportion of PHMAP scores that
have been changed after a review. Two of the associations did not
believe that PHMAP scores adequately measured the management
performance of housing authorities because they thought some PHAs
that received high scores did not provide their residents with
decent, safe housing. The professional property management firm that
independently verified some scores also agreed that the usefulness of
these scores is limited. Because this firm has recommended lowering
many scores after an independent assessment, the firm lacks
confidence in the scores' accuracy and does not believe that the
program provides enough information about the management performance
of PHAs for HUD to make effective funding decisions.
--------------------
\1 Housing authorities receive operating subsidies from HUD each year
to make up the difference between the rent they are allowed to charge
their tenants and the expected costs of operating their developments.
\2 Housing and Urban Development: Comments on HUD's FY 1997 Budget
Request (GAO/T-RCED-96-205, June 17, 1996).
CONCLUSIONS
---------------------------------------------------------- Chapter 4:3
In recent years, both the Congress and HUD have proposed additional
uses for PHMAP, such as bonuses to reward those housing authorities
with the highest scores. While PHMAP has provided a quantifiable
means to assess the management performance of housing authorities,
the scores are not sufficiently accurate for detailed comparisons of
performance. Although HUD is currently working to enhance the
accuracy of these scores, they do not yet provide a comprehensive,
generally accepted way to assess the performance of PHAs. To be
useful for other purposes, not only would these scores have to be
more accurate, but the program would have to be expanded to provide a
more comprehensive measure of public housing authorities' management
operations.
Because HUD does not frequently confirm most scores--confirmatory
reviews have focused on troubled PHAs--HUD does not know how many
authorities are not receiving the proper designation. When HUD does
confirm scores, it changes half of them--and more than half of these
changes result in HUD's lowering the score. We found that when HUD
lowers a PHMAP score, it does so by an average of 14 points. If this
average change held true for housing authorities in general, then HUD
may not be properly designating as troubled those authorities
currently scoring between 60 and the low 70s whose scores should be
lower. As a result, those authorities are not receiving the
oversight and technical assistance HUD should be providing to improve
their performance.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 4:4
We recommend that until it establishes a cost-effective means to
ensure consistently accurate scores, HUD should
-- not consider additional uses for PHMAP, including using its
scores as criteria for funding bonuses, until it determines that
PHMAP meets an acceptable level of accuracy and more
comprehensively measures property management performance and
-- require its field offices to confirm the PHMAP scores of housing
authorities with scores low enough that they are at risk of
being designated troubled.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 4:5
HUD agreed with our findings and recommendations. When we met with
HUD officials, including the General Deputy Assistant Secretary for
Public and Indian Housing, to discuss a draft of this report, they
told us that the Department is no longer considering additional uses
for PHMAP, such as using scores as criteria for funding bonuses.
Even in the absence of using PHMAP for such purposes, we believe that
it is important that HUD works to ensure scores are more consistently
accurate and have, therefore, retained this recommendation. HUD has
begun taking steps to address our recommendation that it confirm
PHMAP scores of those housing authorities that are at risk of being
designated troubled but expressed concern that it may not have
sufficient resources to fully implement this recommendation.
HUD expressed three concerns relating to the information and
conclusions presented in this chapter of our report. HUD believed
that this chapter (1) assumes that PHMAP was intended to be an
all-inclusive assessment system for property management, (2) does not
place PHMAP in a historical perspective, and (3) reaches incorrect
conclusions regarding the overall reliability of PHMAP scores.
We do not believe that we characterize PHMAP's purpose as being an
all-inclusive measure of property management. Our discussion of the
program does not state that this is the purpose of PHMAP. Rather,
the report discusses how the program's limitations--including its
intentional design not to be a complete performance measure--affect
its suitability for additional purposes, such as those proposed in
recent years by HUD and the Congress. HUD agreed that there is a
perception that PHMAP is an all-encompassing system to assess the
performance of PHAs and stated it is taking steps to address this
misperception. Seeking to clarify the program's purpose, HUD added
language to its recently revised interim PHMAP rule (published in the
December 30, 1996, Federal Register), that the program's indicators
reflect performance in only specific areas.
HUD correctly states that this report does not provide a historical
perspective of PHMAP by discussing previous HUD systems for assessing
and identifying troubled housing authorities. We believe that such
information would not contribute substantially to our report's three
objectives to evaluate HUD's use of the current program, provide
trends in PHMAP scores from fiscal years 1992 through 1995, and
discuss limitations in the program's design and implementation that
affect its usefulness for purposes other than identifying troubled
housing authorities and targeting assistance to them. Therefore, we
have not added the historical information HUD suggested to the
report.
Finally, HUD is concerned that we have incorrectly reached
conclusions about the reliability of all PHMAP scores based on the
results of confirmatory reviews of high-risk authorities. HUD noted
that the accuracy of the scores of these PHAs does not necessarily
represent the accuracy of all PHMAP scores because the data provided
by these PHAs are most susceptible to being inaccurate. Our report
did not reach a conclusion about the reliability of all housing
authorities' scores because of the changes that resulted from
confirmatory reviews. This report discusses the reliability of PHMAP
scores for housing authorities whose scores are low enough that they
may be at risk of being designated troubled. We have added language
to the report to clarify this point.
AVERAGE PHMAP SCORE BY GEOGRAPHIC
REGION, FISCAL YEARS 1992-95
=========================================================== Appendix I
Average PHMAP score
----------------------------------------------------------------------
Region 1992 1993 1994 1995
----------------- ---------------- ---------------- ---------------- ----------------
=========================================================================================
Great Plains 82.2 83.1 86.9 82.9
Des Moines 84.4 82.3 88.9 92.1
Kansas City 81.7 81.5 86.9 88.3
Omaha 84.0 88.4 88.3 90.4
St. Louis 78.9 78.7 83.6 53.1
=========================================================================================
Mid-Atlantic 80.1 81.3 83.1 85.7
Baltimore 81.8 84.3 82.8 84.8
Charleston 76.0 79.7 82.2 86.1
Philadelphia 80.9 81.4 85.9 87.4
Pittsburgh 80.1 80.5 81.3 83.2
Richmond 84.5 84.7 84.2 88.7
District of 72.5 71.7 73.9 76.3
Columbia
=========================================================================================
Midwest 83.6 86.1 88.8 88.9
Chicago 75.4 77.2 82.9 83.9
Cincinnati 77.6 78.8 84.8 87.1
Cleveland 77.9 80.8 83.9 85.5
Columbus 80.7 85.1 86.3 90.7
Detroit 82.4 84.0 87.6 87.9
Grand Rapids 85.5 86.3 91.2 90.2
Indianapolis 82.6 87.3 89.6 89.5
Milwaukee 90.2 92.5 93.1 91.5
Minneapolis 86.5 90.1 90.6 90.7
=========================================================================================
Northwest/Alaska 89.9 92.9 92.9 92.6
Anchorage 77.8 86.1 94.9 99.2
Portland 90.6 93.9 94.1 95.4
Seattle 89.6 92.0 91.4 89.4
=========================================================================================
New York/New 81.8 84.9 88.2 89.7
Jersey
Buffalo 84.6 86.9 90.0 91.2
New York 79.6 85.0 83.4 90.5
Newark 80.9 83.7 88.9 88.4
=========================================================================================
New England 83.4 85.3 89.3 89.2
Boston 83.6 82.9 88.0 90.4
Hartford 74.4 76.9 83.7 78.3
Manchester 91.9 94.1 95.1 94.0
Providence 79.9 87.3 89.9 92.4
=========================================================================================
Pacific/Hawaii 85.4 85.4 86.9 87.2
Honolulu 81.5 68.9 73.8 70.6
Los Angeles 89.1 90.0 90.4 91.0
Phoenix 85.4 83.3 88.2 83.8
Sacramento 86.1 81.9 75.5 85.5
San Francisco 82.9 84.9 87.4 87.7
=========================================================================================
Rocky Mountains 87.8 87.9 86.8 91.5
Denver 87.8 87.9 86.8 91.5
=========================================================================================
Southeast/ 82.9 85.4 87.9 84.6
Caribbean
Atlanta 81.7 83.7 85.5 87.9
Birmingham 85.8 87.3 89.7 90.8
Columbia 84.2 86.5 91.7 94.0
Greensboro 82.5 85.6 88.8 89.9
Jackson 82.6 86.6 86.4 87.4
Jacksonville 79.6 83.3 86.0 83.3
Louisville 86.9 88.4 91.0 60.6
Knoxville 85.4 87.7 91.9 90.1
Nashville 77.1 82.4 85.4 85.3
Caribbean 44.5 36.7 48.2 32.0
=========================================================================================
Southwest 80.3 80.8 84.0 85.7
Albuquerque 74.8 70.4 72.7 83.7
Beaumont 79.1 78.4 83.7 84.1
Ft. Worth 80.5 79.5 82.9 85.9
Houston 80.1 78.3 80.8 82.7
Little Rock 86.3 90.5 91.6 91.5
New Orleans 76.5 78.6 81.3 79.5
Oklahoma City 80.0 81.1 84.0 86.1
San Antonio 79.8 79.9 85.5 87.0
-----------------------------------------------------------------------------------------
Source: GAO's analysis of data from the Department of Housing and
Urban Development's (HUD) System for Management Information
Retrieval-Public Housing (SMIRPH) database.
NUMBER OF TROUBLED PHAS BY
GEOGRAPHIC REGION, FISCAL YEARS
1992-95
========================================================== Appendix II
Number of troubled PHAs
----------------------------------------------------------------------
Region 1992 1993 1994 1995
----------------- ---------------- ---------------- ---------------- ----------------
=========================================================================================
Great Plains 10 13 12 7
Des Moines 1 3 1 1
Kansas City 1 3 5 2
Omaha 0 1 1 1
St. Louis 8 6 5 3
=========================================================================================
Mid-Atlantic 7 7 11 6
Baltimore 0 0 1 0
Charleston 1 0 0 0
Philadelphia 2 2 4 3
Pittsburgh 2 3 4 2
Richmond 1 1 1 0
District of 1 1 1 1
Columbia
=========================================================================================
Midwest 24 16 12 12
Chicago 12 10 7 5
Cincinnati 1 0 0 0
Cleveland 2 0 1 1
Columbus 0 0 0 0
Detroit 4 5 3 2
Grand Rapids 1 0 0 0
Indianapolis 2 0 0 2
Milwaukee 0 0 0 1
Minneapolis 2 1 1 1
=========================================================================================
Northwest/Alaska 0 0 0 1
Anchorage 0 0 0 0
Portland 0 0 0 0
Seattle 0 0 0 1
=========================================================================================
New York/New 10 5 6 3
Jersey
Buffalo 0 1 1 0
New York 4 2 3 1
Newark 6 2 2 2
=========================================================================================
New England 12 5 2 5
Boston 2 0 0 0
Hartford 7 5 2 4
Manchester 0 0 0 1
Providence 3 0 0 0
=========================================================================================
Pacific/Hawaii 2 2 2 4
Honolulu 0 0 0 0
Los Angeles 0 0 0 0
Phoenix 1 1 1 2
Sacramento 0 0 0 0
San Francisco 1 1 1 2
=========================================================================================
Rocky Mountains 2 4 8 0
Denver 2 4 8 0
=========================================================================================
Southeast/ 30 30 19 58
Caribbean
Atlanta 9 12 7 5
Birmingham 1 1 0 0
Columbia 2 2 0 0
Greensboro 3 3 0 1
Jackson 2 0 1 1
Jacksonville 2 4 3 3
Louisville 2 1 2 1
Knoxville 0 0 0 1
Nashville 7 4 4 2
Caribbean 2 1 1 1
=========================================================================================
Southwest 33 36 29 28
Albuquerque 4 9 8 3
Beaumont 3 4 0 1
Ft. Worth 7 7 8 6
Houston 1 1 0 1
Little Rock 3 1 0 1
New Orleans 12 7 9 12
Oklahoma City 1 1 0 0
San Antonio 2 6 4 4
=========================================================================================
All regions 130 118 101 150
-----------------------------------------------------------------------------------------
Source: GAO's analysis of data from HUD's SMIRPH database.
NUMBER OF STANDARD-PERFORMING PHAS
BY GEOGRAPHIC REGION, FISCAL YEARS
1992-95
========================================================= Appendix III
Number of standard-performing PHAs
----------------------------------------------------------------------
Region 1992 1993 1994 1995
----------------- ---------------- ---------------- ---------------- ----------------
=========================================================================================
Great Plains 235 213 157 141
Des Moines 30 31 18 12
Kansas City 99 95 62 60
Omaha 67 47 40 34
St. Louis 39 40 37 35
=========================================================================================
Mid-Atlantic 121 115 89 78
Baltimore 15 13 12 13
Charleston 25 25 22 20
Philadelphia 36 33 17 11
Pittsburgh 26 27 20 18
Richmond 13 12 13 11
District of 6 5 5 5
Columbia
=========================================================================================
Midwest 304 246 215 205
Chicago 73 67 61 61
Cincinnati 6 6 6 4
Cleveland 12 13 9 9
Columbus 22 12 13 9
Detroit 24 19 19 17
Grand Rapids 44 40 23 28
Indianapolis 27 20 17 13
Milwaukee 36 30 24 24
Minneapolis 60 39 43 40
=========================================================================================
Northwest/Alaska 24 13 10 10
Anchorage 1 1 0 0
Portland 11 5 4 3
Seattle 12 7 6 7
=========================================================================================
New York/New 107 93 67 54
Jersey
Buffalo 31 28 19 15
New York 18 15 14 6
Newark 58 50 34 33
=========================================================================================
New England 88 84 61 51
Boston 40 49 31 22
Hartford 21 17 15 18
Manchester 10 5 4 4
Providence 17 13 11 7
=========================================================================================
Pacific/Hawaii 42 46 31 27
Honolulu 2 2 2 2
Los Angeles 8 9 9 8
Phoenix 8 8 4 3
Sacramento 4 7 6 3
San Francisco 20 20 10 11
=========================================================================================
Rocky Mountains 49 44 31 35
Denver 49 44 31 35
=========================================================================================
Southeast/ 459 421 334 310
Caribbean
Atlanta 110 110 103 94
Birmingham 80 73 60 49
Columbia 19 19 9 7
Greensboro 63 58 43 31
Jackson 30 27 25 25
Jacksonville 49 37 36 47
Louisville 52 46 23 20
Knoxville 22 18 9 9
Nashville 34 32 25 27
Caribbean 0 1 1 1
=========================================================================================
Southwest 497 445 363 331
Albuquerque 26 18 17 18
Beaumont 56 55 48 42
Ft. Worth 130 128 97 86
Houston 16 16 16 11
Little Rock 56 39 28 31
New Orleans 69 62 54 53
Oklahoma City 78 74 65 51
San Antonio 66 53 38 39
=========================================================================================
All regions 1,927 1,719 1,358 1,242
-----------------------------------------------------------------------------------------
Source: GAO's analysis of data from HUD's SMIRPH database.
NUMBER OF HIGH-PERFORMING PHAS BY
GEOGRAPHIC REGION, FISCAL YEARS
1992-95
========================================================== Appendix IV
Number of high-performing PHAs
----------------------------------------------------------------------
Region 1992 1993 1994 1995
----------------- ---------------- ---------------- ---------------- ----------------
=========================================================================================
Great Plains 102 122 179 200
Des Moines 18 15 30 36
Kansas City 36 39 70 75
Omaha 31 50 57 63
St. Louis 17 18 22 26
=========================================================================================
Mid-Atlantic 42 48 74 90
Baltimore 3 5 6 6
Charleston 6 7 10 12
Philadelphia 12 15 30 37
Pittsburgh 8 6 13 17
Richmond 12 14 13 16
District of 1 1 2 2
Columbia
=========================================================================================
Midwest 200 266 303 316
Chicago 13 21 30 34
Cincinnati 2 3 3 5
Cleveland 5 6 9 9
Columbus 2 12 11 15`
Detroit 18 23 25 28
Grand Rapids 31` 36 53 48
Indianapolis 12 21 24 26
Milwaukee 59 65 71 71
Minneapolis 58 79 77 80
=========================================================================================
Northwest/Alaska 34 45 48 47
Anchorage 0 0 1 1
Portland 18 24 25 26
Seattle 16 21 22 20
=========================================================================================
New York/New 46 65 90 106
Jersey
Buffalo 20 22 31 36
New York 10 15 15 25
Newark 16 28 44 45
=========================================================================================
New England 67 78 102 112
Boston 24 17 34 44
Hartford 5 11 15 12
Manchester 33 38 39 38
Providence 5 12 14 18
=========================================================================================
Pacific/Hawaii 35 31 46 49
Honolulu 0 0 0 0
Los Angeles 14 13 13 14
Phoenix 6 6 10 11
Sacramento 3 0 1 4
San Francisco 12 12 22 20
=========================================================================================
Rocky Mountains 64 67 77 84
Denver 64 67 77 84
=========================================================================================
Southeast/ 281 355 459 487
Caribbean
Atlanta 51 79 91 101
Birmingham 63 70 85 96
Columbia 19 19 31 34
Greensboro 31 36 54 65
Jackson 19 24 27 27
Jacksonville 22 35 41 30
Louisville 52 59 81 85
Knoxville 10 14 23 23
Nashville 14 19 26 26
Caribbean 0 0 0 0
=========================================================================================
Southwest 162 216 307 342
Albuquerque 6 9 13 16
Beaumont 11 11 22 27
Ft. Worth 36 38 68 83
Houston 3 3 4 8
Little Rock 51 70 82 78
New Orleans 16 29 34 34
Oklahoma City 23 27 37 51
San Antonio 16 29 47 45
=========================================================================================
All regions 1,033 1,293 1,685 1,833
-----------------------------------------------------------------------------------------
Source: GAO's analysis of data from HUD's SMIRPH database.
(See figure in printed edition.)Appendix V
COMMENTS FROM THE DEPARTMENT OF
HOUSING AND URBAN DEVELOPMENT
========================================================== Appendix IV
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix VI
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION
Lawrence J. Dyckman, Associate Director
Eric Marts, Assistant Director
Carol Anderson-Guthrie
Curtis Groves
Bill MacBlane
Luann Moy
Terri Russell
*** End of document. ***