[Review of the fiscal year 1999 performance reports and fiscal year 2001 performance plans for the U.S. Department of the Interior]
[From the U.S. Government Printing Office, www.gpo.gov]

Title: ï¿½ï¿½ï¿½

 
Date:  June 30, 2000

**********DISCLAIMER********** 
This file contains an ASCII representation of an OIG report. No attempt has been made to display graphic images or illustrations. Some tables may be included, but may not resemble those in the printed version. A printed copy of this report may be obtained by referring to the PDF file or by calling the Office of Inspector General, Division of Acquisition and Management Operations at (202) 219-3841. 
****************************** 

June 30, 2000
  
Honorable Fred Thompson
Chairman, Committee on
Governmental Affairs
United States Senate
Washington, D.C.  20510
  
Dear Mr. Chairman:
  
This report presents the results of our review of the fiscal year 1999 performance reports and fiscal year 2001 performance plans for the U.S. Department of the Interior and its bureaus and offices. 

The objective of our review was to analyze the reports and plans for the purpose of offering our comments and suggestions for improvements and to respond to the six questions contained in your May 10, 2000 letter to me about the strategies, goals, and measures in the fiscal year 1999 performance reports. (No. 00-I-533)
  
Our review was based on information contained in the performance plans and reports for fiscal year 1999 and the performance plans for fiscal year 2001 for the Departmental Management Offices, the Bureau of Indian Affairs, the Bureau of Land Management, the Bureau of Reclamation, the Minerals Management Service, the National Park Service, the Office of Insular Affairs, the Office of Surface Mining Reclamation and Enforcement, the U.S. Fish and Wildlife Service, and
the U.S. Geological Survey.  The Department and its bureaus and offices reported 261 goals\measures in their fiscal year 1999 performance reports and 188 in the fiscal year 2001 performance plans. We coordinated our efforts with the General Accounting Office (GAO) and did not examine the 30 goals\measures and related performance accomplishments reviewed by GAO.  Our review covered the remaining 231 goals\measures in the fiscal year 1999 reports and 2 goals\measures that were included in the fiscal year 1999 plans but not discussed in the reports.
  
Overall, we concluded that the performance goals\measures were appropriate for and relevant to the accomplishment of Department and bureau missions\programs objectives.  However, we believe that the goals and measures could be improved by eliminating or modifying goals that are not program or function-related, establishing goals for all significant programs and functions, describing the method used to establish target levels of accomplishment, indicating the relevance of performance measures, and establishing appropriate measures as performance indicators.  We also concluded that 5 of the 10 bureaus and Departmental Management Offices included goals in their fiscal year 1999 performance plans that addressed all of their applicable management challenges, the Department reported that it met 145 of the 231 (63 percent) performance goals\measures and adequately explained why 82 of the remaining 86 goals\measures were not achieved; the reported performance data for 161 of the 231 (70 percent) goals\measures were reliable assuming that the verification and validation processes described by the bureaus were implemented and adequate controls were instituted to ensure the integrity of the data; the agencies identified steps to correct identified shortcomings in the data; the bureaus provided an adequate explanation of why goals\measures included in the fiscal year 1999 plans were modified or excluded from the report; and the fiscal year 2001 plans are generally an improvement over their fiscal year 1999 plans.
  
Our general observations, responses to the six questions, and assessments of individual bureaus and offices performance plans and reports are presented in the body of the report.  Department officials provided comments to a draft of this report, which we considered and incorporated into the final report as appropriate.
  
If you or any members of your staff have any questions about this report please contact me or Ms. Mary Adler, Deputy Inspector General, at (202) 208-5746.
  
Sincerely,
  
Earl E. Devaney
Inspector General  
  
cc: John Berry, Assistant Secretary Policy, Management and Budget
Susan Proper, Minority Staff Counsel, 
Committee on Governmental Affairs United States Senate
Norma Campbell, Director, Office of Planning and Performance Management                            

CONTENTS Page
  
GENERAL OBSERVATIONS 1
  
RESPONSES TO QUESTIONS FROM THE U.S. SENATE
COMMITTEE ON GOVERNMENTAL AFFAIRS 4
  
SPECIFIC COMMENTS ON PLANS AND REPORTS . . . . . . . . . . . . . .            7

DEPARTMENTAL MANAGEMENT OFFICES 7
BUREAU OF INDIAN AFFAIRS 13
BUREAU OF LAND MANAGEMENT 22
BUREAU OF RECLAMATION 32
MINERALS MANAGEMENT SERVICE 48
NATIONAL PARK SERVICE 52
OFFICE OF INSULAR AFFAIRS 60
OFFICE OF SURFACE MINING 67
U.S. FISH AND WILDLIFE SERVICE 73
U.S. GEOLOGICAL SURVEY 80
  
GENERAL OBSERVATIONS
  
In our analyses of the Department of the Interior's (DOI) performance reports and plans, we evaluated the performance goals and measures and data validation and verification procedures and, where appropriate, offered suggestions for improvements.  Because the reports used the terms goals and measures  inconsistently, we identify all performance goals and measures as "goals" in this report. No. 00-I-533      

Performance Goals

Overall, we found that performance goals were appropriate for and relevant to the accomplishment of bureau mission\program objectives.  Also, in general, we consider the bureaus' fiscal year 2001 plans to be an improvement over their fiscal year 1999 plans with respect to the elimination of nonessential goals and establishment of more mission-related  goals.  However, we believe that DOI could improve the goals in its performance plans by:
  
-  Eliminating or modifying goals that are not program or function-related.  In a few cases, bureaus established goals that were not related to programs but to approaches or strategies for achieving program results.  For example, one of four goal categories established by National Park Service (NPS) is "ensure organizational effectiveness." (A similar goal was included in Bureau
of Reclamation's (BOR) performance plan.)  This goal category includes items such as "workforce stewardship" (that is, employee job satisfaction) and workforce diversity.  Also, Fish and Wildlife Service's (FWS) fiscal year 2001 plan contains a secondary goal, "partnerships in accountability" (under its "Partnerships in Natural Resources" mission goal), the objective of which is "to
ensure that the Service have [sic] in place processes, procedures, and controls . . . to ensure fiscal accuracy and accountability to the public."  We do not consider organizational effectiveness to be a distinct performance goal because, as Bureau of Land Management (BLM) states in its performance plan, it is a "'means' rather than an outcome" and "only indirectly related to mission."

Bureaus also classified as goals data collection efforts or other strategies for accomplishing program objectives.  For example, FWS classified grants management as a goal, the objective of which was to automate information on grants.  Maintaining grants information is an activity that enables FWS to award grants for projects and programs that support FWS mission objectives.  BLM classified as a goal the condition assessment of priority sub-basins.  The performance of condition assessments is an activity that enables BLM to identify sub-basin areas in need of restoration, with restoration being the program for which performance measures should be established.  Also, NPS classified as a goal the inventorying of archeological sites, an activity that supports another NPS goal, improving the condition of archeological sites. 
  
- Establishing goals for all significant programs and functions.  In some cases, bureaus did not establish performance goals for all major programs and functions.  For example, NPS, which had performance goals for its historic structures, cultural landscapes, and archeological sites, did not have a performance goal related to the maintenance or construction of its basic
infrastructure of buildings, structures, roads, and other facilities.  In addition, although NPS had a goal on visitor satisfaction it did not have a goal specific to its concessions program, which provides essential goods and services to park visitors.  The Office of Surface Mining Reclamation and Enforcement (OSM), which has a statutory requirement to collect fees based on the type and
amount of coal production, also did not have a performance goal related to its fee collection program.  Annually, OSM collects fees of about $277 million from coal operators.  The fees are the sole funding source for OSM's Abandoned Mine Lands Program.
  
Performance Measures
  
In general, we found that the bureaus established appropriate measures or performance indicators and that baseline data in the fiscal year 2001 plan were more complete than data in the fiscal year 1999 plan.  The measures, however, in many cases were not related to the achievement of program objectives but rather to the completion of tasks or activities.  As such, the measures may not provide meaningful information to evaluate mission accomplishments. We believe that DOI
could improve the measures in its performance plans  by:
  
- Describing the method used to establish target levels of accomplishment.    In general, the bureaus did not describe their rationale for targeting annual performance at a particular level. However, some bureaus  for some measures, did provide an explanation for the targeted performance levels.  OSM, for example, planned to reclaim 8,100 acres in fiscal year 2000 and 9,100 acres in fiscal year 2001.  In the text of its 2001 plan, OSM stated that it had requested
additional funding of $15.3 million, "which will result in an additional 1,000 acres being reclaimed."  Also, we found that some bureaus set fluctuating values for their performance goals, but did not explain the reason for the fluctuations.  NPS, for example, having restored 14.7 percent of its disturbed lands in fiscal year 1999, established a fiscal year 2000 goal of restoring 16.8 percent of disturbed lands, and a 2001 goal of  restoring 2 percent of disturbed lands.  NPS provided no explanation for the fluctuating targeted level of accomplishment other than a note stating "the baseline was changed for the fiscal year 2001 goal." 
  
- Indicating the relevance of the performance measure.  In quantifying accomplishments, the bureaus, in general, did not provide information to show the relevance or value of the targeted level of performance.  For example, OSM stated that it would reclaim a specific number of acres and fund a specific number of projects to accomplish its goal of environmental restoration. 
However, OSM's measure did not provide for prioritizing the accomplishment of reclamation projects based on the significance of the problems (health and safety projects should be given priority) and did not provide information on the total number of acres or projects in need of restoration. 
  
- Establishing appropriate measures as performance indicators.  In some cases, bureaus did not establish appropriate measures for their performance goals.  For example:
  
-- The U.S. Geological Survey (USGS) has a program activity "Hazards" the objective of which is "to minimize the loss of life and property."  However, the measures for achieving long-term and annual performance goals for this program activity do not describe the outcome in terms of conducting science or providing technologies that reduce risk, increase preparedness, or mitigate damage.  Instead, USGS's hazard-related goals pertain to the maintenance of scientific
equipment or the delivery of technical\scientific products.  
  
-- The Bureau of Indian Affairs (BIA) has a long-term goal of improving the safety and functionality of BIA schools and facilities and measures the accomplishment of this goal in terms of the "number of schools on the priority list receiving replacement construction" and the number of facility improvement and repair projects awarded.  The measures, however, do not provide information on whether schools are safer or more functional.
  
-- The Office of Insular Affairs (OIA) had a goal of improving the infrastructure and government systems and operations of insular area governments.  For this goal, OIA established a performance measure of completing multi-year capital plans for each insular area government. 
We consider effective implementation and not completion of the plan to be the measure of program accomplishment. 
  
-- BOR had goals of increasing water availability and improving water quality.  BOR, however, had no performance measures which showed the increased water availability or the progress in improving water quality.
  
Data Reliability and Verification
  
Most bureaus and offices described the methods or proposed methods that would be used to validate or verify performance data.  In some cases, a detailed explanation was provided that clearly stated the actions that would be taken to verify data, the individuals responsible for the verification, and the internal controls that might be implemented to ensure data reliability. However, some bureaus that had significantly revised their performance plans had not established performance baselines and, therefore, did not address the issue of data validation and verification. Also, although most bureaus' fiscal year 1999 reports disclosed data collection shortcomings and difficulties encountered in obtaining reliable data, only FWS included a separate section in its
fiscal year 2001 plan that described data limitations for each performance measure.             

How valid and reliable is the data by which the agency judged its performance? Where data shortcomings exist, did the agency acknowledge them and indicate what steps it will take to correct them?
  
Since we did not audit the baseline measures or the reported accomplishments, we cannot make an independent assessment of the reliability and validity of the bureaus' and offices' data. However, we assessed the bureaus' and offices' validation and verification processes as described in their 1999 and 2001 plans in terms of whether the processes provided reasonable approaches that might produce reliable data.  Based on these assessments, we consider the 161 of 231 goals to be based on processes that might produce reliable data if sufficient controls are implemented to ensure the integrity and consistency of the data collection.  Also, bureaus and offices identified limitations on the data reliability of 104 goals and for 98 of these goals, the bureaus and offices described the actions that would be taken to improve data reliability.  In our assessment we considered the discontinuation of a goal to be an action taken to address data reliability.
  
Where an agency has not met a performance goal, does the report adequately explain why and describe a strategy to meet the goal in the future?
  
Of 86 goals that were reported as not being met, the bureaus provided an adequate explanation for 82.
  
Where a goal from the fiscal year 1999 performance plan is not covered in the performance report or has changed, did the agency adequately explain why?
  
Of the 231 goals reviewed, all were in the fiscal year 1999 performance plans but 2 were not discussed in the reports.  The omission of the two goals was not adequately explained.  Thirteen goals in the plans were changed in the reports and only one of these changes was adequately explained.
  
What improvements has the agency made in its performance plan for Fiscal Year 2001 that are relevant to the above changes?
  
Our responses to this question are presented in our specific comments on the bureau and offices' plans and reports presented in the next section of the report.
  
SPECIFIC COMMENTS ON PLANS AND REPORTS
  
DEPARTMENTAL MANAGEMENT OFFICES (DMO)
  
DMO organized its fiscal year 1999 performance plan and report into 6 goal categories with 7 long-term goals and 8 annual goals and substantially revised its fiscal year 2001 plan into 4 long-term goals with10 performance indicators.  The 10 performance indicators in the fiscal year 2001 plan are essentially the same as the 7 long-term goals in the fiscal year 1999 plan with 3 additional indicators added.  DMO did not describe the Departmental offices included or excluded from its plans and report.  However, in the text of the plans' individual goals, reference was made to 7 of the 14 Departmental offices listed in DOI's budget justification. Typically, the plans provided for the reporting of results in terms of numbers or percentages.
  
Fiscal Year 1999 Performance Plan and Report
  
The fiscal year 1999 plan and report contained goal categories including: workforce of the future; the year 2000 computer problem; reliable and accurate financial information; faster, better, more cost-effective goods, services, and facilities; facilities maintenance and capital improvements; and waste management.  The plan's long-term and annual performance goals consisted of improving the diversity of DOI's workforce; developing a new training program; achieving unqualified audit opinions on financial statements; remediating critical systems for the year 2000 date change; increasing purchase card transactions; improving accountability over museum objects; completing initial 5-year plans that identify the priority of facility maintenance needs; and establishing a baseline of environmental audit activities.  

We consider seven of the eight annual goals to be appropriate indicators of the overall mission goals or goal categories with which they are allied.  One annual goal, cataloguing museum collections, we consider to be inappropriately classified as "better, faster, more cost-effective goods, services and facilities," because the item does not pertain to the delivery of goods,
services, or facilities.  We also found that DMO revised the goal of annually increasing the percentage of catalogued museum objects to inventorying 38.2 million museum objects in the fiscal year 1999 performance report. 
  
We cannot comment on whether DMO's fiscal year 1999 performance plan provided an
adequate basis for evaluating DMO's mission or strategy because we did not review or audit to determine whether DMO did consider seven of the other Departmental offices.
  
Except for the two goals previously discussed, we consider most annual performance goals and targets for accomplishment of performance goals to be appropriate and results oriented.  In most cases, DMO established a performance baseline that provided a reasonable approach to measuring performance and accomplishments.  We cannot comment on whether the baseline data are valid
or whether the performance targets (that is, the numbers, amount, or percentage of achievement) are appropriate or should be measured in greater or lesser amounts.
  
DMO was consistent in that, with the exception of two goals, the goal categories, long-term goals, and performance targets used in its 1999 performance plan were included in the 1999 performance report.  DMO said that it achieved all but one of its fiscal year 1999 goals.  Regarding the one unmet goal, achieving an unqualified opinion on the financial statements, DMO described the actions it planned to take to meet its objective. 
  
Fiscal Year 2001 Performance Plan
  
In its fiscal year 2001 performance plan, DMO restated its long-term goals as (1) lead people to succeed, (2) provide the services and technology to manage, (3) ensure financial and managerial accountability, and (4) provide safe and high quality places of work.  DMO also added three new goals which relate to DMO-wide operations, including:1) completion of an information technology
infrastructure protection plan; 2) completion of information technology architecture; and 3) resolution of material weaknesses and management risks.  
  
DMO's fiscal year 2001 plan includes the fiscal year 1999 goal of establishing a DOI-wide baseline of environmental audit activities and programs.  For this goal, we suggest that DMO establish a method or system for data accumulation, documentation, validation, and verification.  The current plan states that bureaus differ in their capabilities to collect data and provides no explanation as to how DMO will ensure that environmental data are consistent and reliable.
  
In its fiscal year 2001 plan, DMO reclassifies inventorying museum property and credit card transactions as performance indicators under the long-term goal, "provide the services and technology to manage."  Again, we find the museum property "indicator" inappropriately classified because the identification of museum objects is an asset accountability issue and not a services\technology management issue.  If DMO intends to increase accountability over artifacts
by "inventorying" or "cataloguing" the objects,  we suggest that DMO clarify  how it will verify and validate the accomplishment of this goal.  Regarding the fiscal year 2001 goal on purchase card transactions, we found that the goal does not include indicators or measures to show that credit cards are used only for authorized and necessary purchases an\or that the purchases are transacted
in compliance with regulations.  Also, the plan does not provide for bureau services of internal controls to detect and prevent fraud, waste, or abuse in credit card use.  We suggest that the plan provide for the bureaus to assume responsibility for validating purchase card payments and not rely on OIG for validation, as provided for in DMO's fiscal year 2001 Plan.

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of
implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan. 


D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency  adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were
adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

BUREAU OF INDIAN AFFAIRS
  
BIA's fiscal year 1999 performance plan and report  had 8 mission goals, 19 long term goals, and 45 annual performance goals, while the fiscal year 2001 performance plan had 8 mission goals, 17 long term goals, and 44 annual performance.   The performance targets for these goals typically consisted of units, such as number of houses repaired or fishing sites maintained, or
percentages. 
  
   
Fiscal Year 1999 Performance Plan and Report
  
The eight mission goals in the fiscal year 1999 performance plan  appropriately addressed BIA's major programs. 
  
While BIA's Fiscal Year 1999 performance report did not specifically identify the sources of data or the data verification and validation methodology for its 30 performance goals\measures that we reviewed (we excluded 2 mission goals, consisting of 15 annual performance goals that were reviewed by the GAO), its fiscal year 2001 performance plan includes a section "Data Verification and Validation" which provides a general discussion of BIA's performance data gathering and reporting process.  It also indicates that its primary customers are Individual Indians and Tribal organizations and that information collection requirements limit what and when BIA can require the customer to provide data.  This limits the BIA's data collection activities for 12 of the 30 goals\measures. 
  
We are not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of BIA's data collection, verification, and validation processes for each of BIA's 30 performance goals\measures.  However, we consider 18 of the 30 goals\measures to
have processes that would produce reliable data if adequate controls are implemented to ensure the integrity of the data throughout the process.
      
We believe that the specific annual performance goals\measures are indicators of certain levels of program activities and that the method of reporting the results in an appendix does not communicate what outcomes were actually accomplished by performing those activities.  Many of the BIA's goals\measures are activities which obligate funds through contracts, compacts, grants, or purchase orders.  BIA's measurement of performance generally stops with obligating action, we believe that the plan should include measures that demonstrate whether the intended outcome from the obligations were realized and to what extent these outcomes accomplished the mission goal and long-term goal. In addition, a number of long-term goals are vague because baselines for measuring improvement have not been established and\or communicated in the plan. 
  
Of the "Ten Management Challenges" identified  in December 1998  by the OIG, four, were applicable to the BIA.  BIA's fiscal year 1999 plan include goals\measures addressing "Maintenance" and "Financial Management," but did not have any goals\measures regarding "Waste Management " or "Revenue Collections."    BIA estimated in 1997 that its cleanup liability for known sites was $66 million and that it needs approximately $100 million for studies and evaluations to identify other sites and to determine associated estimates of cleanup costs. We
believe that BIA should consider whether a performance measure should be established under the Mission Goal of Resources Management to address this matter   Also, BIA revenue collections for fiscal year 1999 were more than $200 million.  We believe that BIA should consider whether this matter should be addressed by a performance measure under the Mission Goal of Administration and Support Services.
  
Fiscal Year 2001 Performance Plan
  
BIA's fiscal year 2001 performance was changed significantly form the Fiscal Year 1999 performance plan.  BIA eliminated or revised many goals\measures in order to improve the performance plan for fiscal year 2001.  Overall, the fiscal year 2001 plan was an improvement.  

However, the fiscal year 2001 plan still includes a significant number of goals and measures that do not communicate program outcomes.
*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were
adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H. 

BUREAU OF LAND MANAGEMENT
  
BLM organized its fiscal year 1999 and fiscal year 2001 performance plan into eight goal categories, each of which had subsets of annual goals (48 in 1999 and 20 in 2001).  The annual goals, in turn, had one or more performance targets to measure accomplishments.  These targets consisted of units, such as acres, number of sites; or percentages.
  
Fiscal Year 1999 Performance Plan and Report
  
BLM's Fiscal Year 1999 Performance Plan and Report generally provided an adequate basis for evaluating BLM's accomplishment of its essential mission goals.  One significant exception was that a number of BLM's goals were activity rather than program based and thus may not provide a meaningful performance measure.  For most goals and measures, BLM reported that it established a performance baseline for each annual goal and that the basis for the baseline was described in a separate document.  We cannot comment on whether the baseline data are valid or whether the annual goals (in terms of numerical amount) are appropriate because we have not reviewed the supporting document.
  
BLM's 1999 Plan provided overall but not detailed information on methods BLM planned to use to validate data. BLM said that there were data limitations in collecting data that was used to document the achievement of performance goals.  
  
In its fiscal year 1999 performance report, BLM either met or exceeded 26 of  48 goals. BLM adequately explained why 22 goals were not met and provided a strategy to ensure that the goals will be met in the future or deleted it, or revised the goal.  

One long-term goal target in the fiscal year1999 Plan was changed in the  performance report without any explanation.  BLM changed the goal for "acres of vegetation are improved" from 316,000 acres to 250,000 acres (although the Report stated that 253,771 acres were improved).  

Fiscal Year 2001 Performance Plan

In its fiscal year 2001 Plan, BLM added one mission goal, deleted one mission goal, deleted 30 annual goals, and added or modified other annual goals.  As a result of theses changes, the 2001 plan is a significant improvement over the 1999 plan in that it contains more annual goals to measure program performance.  Because BLM has a complex mission and is responsible for managing 264 million acres of public land and 70 million acres of subsurface interests on privately owned lands, we consider the task of establishing meaningful goals and measures to be a difficult one for BLM.
  
The 2001 plan provided detailed information on the data that would be used to measure performance and in most cases described the data collection process and the methods used to validate the data.  
  
Based on our review, we believe that BLM could improve its fiscal year 2001 plan to convey program objectives and the accomplishments of performance measures as follows: 
  
- Provide additional information to more fully explain its goals and measures and the significance of each goal measure.
  
- Describe the total program area for which a measure has been established.  For example, one fiscal year 2001 annual goal is to correct physical safety hazards at 10 abandoned mine sites and clean up 156 hazardous material sites.  The plan did not identify the total number of sites that have been identified as needing improvements.
  
- Establish more precise and limited goals to measure program performance. For example, one fiscal year 2001 goal is to "complete 91% of actions on existing energy and mineral leases, permits, and claims consistent with established land health standards to minimize future liabilities."  This goal encompasses oil and gas leases which generate about $517 million of Federal revenue, coal leases
which generate about $305 million of Federal revenue, and production of other
revenue-producing minerals (such as gold, silver, and copper).  Because each of these resources requires separate oversight and program management activities, we believe that a separate measure is needed to monitor performance in each area.  
  
- Add annual goals or measures for some programs.  For example, BLM has two goals directly related to its cultural resource program.  While these goals address critical elements and known problem areas within the program, BLM has no goal, measure, or strategy for inventorying BLM lands to identify all culturally significant sites.  Because OIG, in a 1999 audit report, stated that BLM had comprehensively surveyed less than 5 percent of its lands for the presence of cultural sites, we believe that BLM needs to include in its performance plan a process for identifying its culturally significant sites. 

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of
implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency adequately explained why it excluded or changed the goal\measure. 

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were adequate controls to ensure the integrity of the data throughout the process.  In addition, the FY 1999 plan referenced a 130-page document which stated BLM included detailed information on data specifications for each goal.  OIG did not review this document.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.  BLM's Fiscal Year 2001 performance plan acknowledges that BLM has shortcomings in collecting data and explains how it is trying to overcome this problem.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

BUREAU OF RECLAMATION
  
BOR's fiscal year 1999 performance plan and report had 15 mission goals, 47 annual performance goals, and 47 performance indicators.  Its fiscal year 2001 plan had 3 mission goals, 27 annual goals, and 48 performance indicators (some annual goals had 2 or more indicators).  The indicators typically consisted of units, such as number of projects or reviews; single tasks, such as issuing
guidelines; or percentages.
   
Fiscal Year 1999 Performance Plan and Report
  
BOR's fiscal year 1999 performance plan included 15 mission goals and 47 annual performance goals which covered all of BOR's major programs.  However, several of the mission goals were overlapping and\or did not address major program areas or program outcomes.  For example, the plan included  the mission goal "Facility Maintenance and Rehabilitation" which addressed all BOR facilities but also included a separate goal, "Enhance Recreational Opportunities Through State
And Local Partnerships," which addressed the maintenance of recreation facilities.  

Several annual goals had performance indicators which measured the accomplishment of processes and activities (outputs) rather than the accomplishment of the missions (outcomes).  For example, BOR established annual goals\performance indicators such as drafting proposed surplus and shortage guidelines for the Colorado River; increasing the number of tribes receiving BOR
Technical assistance to 10; publishing the results of 72 research projects; reviewing and commenting on 100 percent of water conservation plans submitted for review; and preparing 5 regional reports on the quality, accuracy, and accessibility of BOR's water quality reporting systems.  Generally, the plan's performance indicators did not measure BOR's progress in accomplishing mission goals.

For example, there were six annual goals addressing the mission goal "Increase Water Availability," but none of the goals actually measure the increase in water availability.  (BOR included a goal in its FY 2001 Plan which measures increases in water availability, but it is limited to the CALFED Bay-Delta Program.)  Similarly, none of the four annual goals under the mission goal
"Maintain and protect water quality" provided baseline information on water quality or any performance measurements demonstrating the progress in accomplishing the mission goal.

The plan did not include sufficient baseline data or other information for us to determine whether performance indicators were appropriate.  For example, the plan did not discuss why the performance indicator for the goal "Provide Power" increased from $2.18\Mwh in fiscal year 1998 to $2.57\Mwh in fiscal year 1999.

We are unable to provide independent assurance regarding the reliability of BOR's performance data because neither the plan nor the report described the verification and validation processes for each performance measure, and we have not assessed the adequacy of BOR's data collection, verification, and validation processes or tested the accuracy of the reported data.  However, the fiscal year 2001 does describe the validation and verification process for each indicator,
Including those for the 12 goals carried over from the fiscal year 1999 plan, and it appears that these processes would produce reliable data assuming that adequate controls are in place to ensure the integrity and consistency of the data throughout the process. We also determined that the performance data for 10 additional goals in the fiscal year 1999 plan was likely to be reliable because the goal was simply to complete a task such as issuing guidelines.

Fiscal Year 2001 Performance Plan 

We believe that BOR's fiscal year 2001 plan is a significant improvement, in both format and content, over its fiscal year 1999 plan.  BOR decreased the number of mission goals from 15 to 3, which eliminated overlapping goals and provided a clearer link to its budget categories. 

BOR also decreased the number of annual performance goals and measures from 47 to 27 by eliminating some goals and measures and combining others.  For example, BOR eliminated 21 goals which were one-time efforts that were completed in fiscal year 1999 or expected to be completed in fiscal year 2000; 8 goals which it determined were unmeasurable or did not provide a meaningful measurement of achievement of a mission goal; and 3 goals that were process oriented.  BOR
adequately described its basis for eliminating these goals.  The 27 goals in the fiscal year 2001 plan include 15 new goals and 12 goals from the fiscal year 1999 plan, some of which had modified performance measurements.

BOR's mission and long-term goals on managing, developing and protecting water and related resources and on operating, maintaining and rehabilitating facilities generally cover BOR's major programs and functions.  However, its third mission goal, "Advance Reclamation's Organizational Effectiveness" includes some activities such as customer service benchmarking evaluating
workforce capability, and workforce representation (diversity) that support the accomplishment of the other goals.  Also, none of the eight annual performance goals provide measures demonstrating that organization effectiveness has improved as a result of those activities.

While the Fiscal Year 2001 Plan contains more outcome measures than the Fiscal Year 1999 Plan, it still contains a relatively large number of output measures which do not fully demonstrate that BOR has achieved its mission or long-term goals.  For example, to address its long-term goal of improving water quality affected by BOR projects, the fiscal year 2001 Plan included the following annual goal:  Conduct water quality assessments at 10 BOR reservoirs and stream segments, implement 3 operational changes and structure modifications, and provide technical assistance to states and local watershed groups.  Although these activities may contribute to accomplishing the goal, they do not provide a measurement of water quality improvement.  Overall, many of the goals and measures in the plan while measuring outputs rather than outcomes, appear to represent critical initial steps in accomplishing BOR's long-term goals, including the establishment of procedures and baseline information.  We would expect that once these activities are accomplished, more outcome-oriented goals and measures can be established.

An additional improvement in the fiscal year 2001 plan is the inclusion of a brief description, for each performance measure, of BOR's data collection methodology, sources, and limitations and its data verification and validation process.  The plan also indicates that BOR has issued a guide on procedures and definitions for reporting on the fiscal year 2001 annual performance goals. 
BOR's verification and validation procedures should ensure the reliability of the reported performance information if adequate control procedures are in place to ensure the accuracy and consistency of the data.

Regarding the key management challenges applicable to BOR (Revenue Collections and Waste Management), the fiscal year 2001 plan did not sufficiently addresses these issues.  Our prior audits have identified several issues related to and sound business practices with respect to cost recovery on BOR's water projects, and several of the related recommendations have not yet been fully resolved and\or implemented.  While the fiscal year 2001 plan contains some goals related to these issues ("Complete long-term water contract renewals" - to ensure full collection of contract obligations; "Improve incidental resource management," (which addresses prior OIG recommendations) and "Improve management of indirect costs") we believe that the plan could be improved by including a goal which addresses the overall issue of ensuring that BOR employs sound business practices in recovering the costs of its projects, including the timely resolution of all related audit issues.  With respect to the Waste Management challenge, the annual related goal in the fiscal year 1999 plan has been discontinued and the fiscal year 2001 Plan does not include an annual goal which specifically addresses this issue.  

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of implementing a Trust Management.  Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.


D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were
adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

MINERALS MANAGEMENT SERVICE
  
The Minerals Management Service (MMS) organized its fiscal year 1999 and fiscal year 2001 performance plans into GPRA mission goal categories (8 in 1999 and 7 in 2001), each of which has subsets of long-term goals (14 in 1999 and 9 in 2001) and annual goals (22 in 1999 and 9 in 2001).  The annual goals, in turn, had one or more performance targets to measure accomplishments.  The targets typically consisted of barrels, cubic measurements, dollars, or percentage.
  
Fiscal Year 1999 Performance Plan and Report
  
MMS's fiscal year 1999 performance plan generally provided an adequate basis for evaluating MMS's accomplishment of its two primary missions: offshore minerals management and royalty management.  For most goals, MMS established a performance baseline that provided a reasonable approach to measuring the accomplishment of mission objectives.  However, we cannot comment on whether the baseline data are valid or whether the annual goals for measuring accomplishments are appropriate because information on the development of the baselines and the numerical goals for annual accomplishments were not in the Plan or Report.  Because the General Accounting Office reviewed 5 mission goals and 12 annual goals, we limited our review to the remaining 4 mission goals and 10 annual goals.
  
MMS's 1999 Plan provided overall but not detailed information on the methods MMS planned to use to validate data.  MMS did not acknowledge any data limitations.
  
Fiscal Year 2001 Performance Plan
  
In its 2001 Plan, MMS explained why it eliminated or revised 1999 goals, stating that these goals primarily depend on outside (market) conditions which are beyond MMS's control.  For the 2001 Plan, MMS identified seven mission goals with nine related annual goals.  Of the seven mission goals, three were the same as those in its fiscal year 1999 plan; three were rephrased from fiscal year 1999, and one was new. Of the nine annual goals, five were rephrased from fiscal year 1999 and four were new.  The 2001 plan provided detailed information on the data that would be used to measure performance and in most cases described the data collection process and the methods used to validate the data.  Overall, the mission and annual goals in the fiscal year 2001 plan provide a better framework for measuring performance than those in MMS' fiscal year 1999 plan.

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of
implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

NATIONAL PARK SERVICE

NPS organized its fiscal year 1999 and 2001 performance plans into four goal categories, each of which had subsets of annual goals (32 in 1999 and 34 in 2001).  The annual goals, in turn, had one or more performance targets to measure accomplishments.  These targets typically consisted of units, such as number of square feet or number of sites, or percentages.
  
Fiscal Year 1999 Performance Plan and Report
  
NPS's fiscal year 1999 performance plan provided an adequate basis for evaluating NPS's accomplishment of its mission.  NPS had four goal categories: preserving park resources, providing for public enjoyment and visitor experience of the parks, and strengthening and preserving natural and cultural resources and enhancing recreational opportunities.  The fourth goal category, ensuring
organizational effectiveness, contains program activities that support the accomplishment of the three other goal categories, but it does not represent a core NPS program or mission in itself. 

For example, by obtaining information (Goal IVa1. Data Systems), recruiting a diverse workforce (Goal IVa4. Workforce Diversity), and increasing financial resources (Goals IVb2.: Donations and Grants, IVb3: Concession Returns, and IVb4: Fee Receipts), NPS is better able to accomplish its mission.  Because the items in goal category 4 are important management functions, we suggest that
NPS and DMO consider including these goals in the Department's performance plan as cross- cutting program objectives for the management of the Department in areas such as information resources, human resources, and financial resources.

Although the goal categories, in general, were appropriate, we believe that some long-term goals could be eliminated and restated as strategies for accomplishing the goals.  For example, the Ib series of long-term goals (Natural Resource Inventories and Cultural Resource Baselines) are activities that enable NPS to accomplish its Ia series of long-term goals (Disturbed Lands, Historic structures, Museum Collections, etc).  We also suggest that NPS reconsider the inclusion of goals Ia3, Air Quality, and Ia4, Water Quality, in its plan because the achievement of these goals largely depends on conditions outside the control of NPS officials.  Also, long-term goals IIIa3, User Satisfaction, and IIIb2, Community Satisfaction, relate to the relationships that NPS maintains with partners that promote NPS program objectives.  While we recognize that NPS accomplishes its mission through its affiliations with outside organizations, we believe that the satisfaction of the public served by these affiliations and not the satisfaction of the affiliated parties is the program objective.
  
Most performance targets for accomplishment of performance goals are appropriate and, in most cases, NPS established a performance baseline that provided a reasonable approach to measuring its accomplishments.  We cannot comment on whether the baseline data are valid or whether the performance targets (that is, the numbers, amount, or percentage of achievement) are appropriate or should be greater or lesser amounts because, in general, NPS did not provide information on why targets were set at specific levels. 

NPS's 1999 plan provided overall but no detailed information on the methods that NPS planned to use to validate data.  In its 1999 report, NPS indicated that in some instances the data were inconsistent or that NPS needed to further refine its baseline measures.
  
NPS was consistent in that the goal categories, long-term goals, and performance targets used in its 1999 plan were included in the 1999 report.  Detailed information on NPS's 1999 plan and report are provided in the Attachment.

Fiscal Year 2001 Performance Plan
  
In its 2001 Plan, NPS explained why it eliminated or revised 1999 goals.  Of three long-term goals reportedly revised, water quality (Ia4), workforce stewardship (1Va2), and construction and maintenance backlog (IVa9), NPS revised two of the goals to provide a different measure or baseline.  The backlog goal was eliminated because the target objective had been achieved.  Also, the 2001 plan contained five new long-term goals: vital signs, geologic resources (Ia9 and Ib4), aquatic resources, and environmental leadership.  Although these goals support the accomplishment of program objectives, we suggest that they be eliminated as goals and incorporated into the plan as strategies or performance parameters.

Neither the 1999 plan nor the 2001 plan provided long-term goals for two core NPS programs: providing sufficient, safe, and well-maintained park facilities and structures to meet park user needs and providing sufficient, well-maintained, and reasonably priced recreational visitor services (typically through concessions operations).  Although these goals might be measured as part of Goal IIa1, Visitor Satisfaction, we believe that they merit separate performance measures to enable management to better monitor and evaluate these essential operations.
  
The 2001 plan generally provided detailed information on the data that would be used to measure performance and in some cases described the methods used to validate the data.  Although NPS in most cases described the data collection process, it generally did not describe the controls that would be implemented to ensure that the data were complete, representative, and accurate.  

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of
implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.  The Park Service's Fiscal Year 1999 performance report does not discuss data shortcomings; however, the Fiscal Year 2001 performance plan does describe data limitations, particularly the need to develop baseline data.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H. 

OFFICE OF INSULAR AFFAIRS
  
The Office of Insular Affairs (OIA) fiscal year 1999 performance plan included four goal categories: improve infrastructure, improve governmental systems and services, improve governmental relations, and manage special problems.  There were 6 long-term and 13 annual performance goals.  The fiscal year 2001 performance plan included three goal categories: 

Improve infrastructure, improve governmental systems and services, and  improve governmental relations. 

For 2001, OIA had five long-term and five annual goals.
                                                    
Fiscal Year 1999  Performance Plan and Report
  
Of the 13 annual performance goals in OIA's  1999 performance plan, 9 related directly to the management challenge "Financial Management" for the insular area governments.  However, only 11 of the 13 annual performance goals were reported  in OIA's fiscal year 1999 performance report.  OIA  did not report on the two performance goals associated with the goal category "manage special problems."  We believe that these two performance goals did not relate to OIA's overall mission to assist the islands in developing more efficient and effective government by providing financial and technical assistance and to serve as a focal point for the management of Federal-island relations by developing and promoting appropriate Federal policies.  One goal related to the implementation of the second year of the brown tree snake program and the other related to the implementation of the fifth year of an interagency effort to address labor,
immigration, and law enforcement problems in the Commonwealth of the Northern Mariana Islands. 
  
According to its fiscal year 1999 performance report, OIA did not achieve any of the reported 11 fiscal year 1999 performance goals and measures and OIA provided valid and reliable data to judge performance for only two goals.  These goals related to the establishment of multi-year capital plans and to the completion of financial improvement plans for each of the insular area governments.  For the goal related to improving governmental relations, OIA discussed the
Method for validation and verification and stated that the method required a quantifiable survey of the insular area governments.  OIA, however, conducted no survey to validate fiscal year 1999 data. OIA also did not meet the goal that all insular area governments complete audited financial statements and provided no discussion of its planned method of validating performance in this area. The
remaining seven performance goals and measures were discontinued and no information was provided on the validity and reliability of the related data.  OIA stated that it discontinued these seven performance goals and measures largely because it had no control over the outcomes, which were dependent on the insular area governments.      

  
OIA discontinued the fiscal year 1999 goal that insular governments will achieve institutional changes 10 percent above the 1998 baseline report cards, consistent with "operations and maintenance improvement action plans" because of the high cost of data collection, which was done by an independent contractor.  OIA stated that some improvement in institutional changes in 1999 over 1998 were achieved, but not at the stated 10 percent level.  While we agree with the rationale for discontinuing this specific goal, we believe that some concrete method of measuring the "outcome" of achieving the overall program activity -- "Improving Infrastructure" -- should be developed.  Audit history has shown that the insular area governments have a relatively poor record of improving infrastructure on a sustained and long-term basis, despite OIA grants for capital infrastructure improvements.  The existence of capital plans and the completion of grant projects do not necessarily signify that infrastructure improvements have been achieved.

The discontinued fiscal year 1999 goal that OIA would work with each insular area government to identify priorities and develop and implement long-term plans to improve expertise, operating efficiency, and institutional capacity in health care, education, public safety, economic development and statistical data was dropped because OIA had no control over the accomplishment of this goal. OIA said that technical assistance efforts would be continued to assist the insular area governments to make meaningful changes in these areas.  While we agree with the rationale for discontinuing this specific goal, we believe that some concrete method of measuring the "outcome" of achieving the overall program activity -- "Improving Government Systems and Services" -- should be developed. 
Audit history has shown that the insular area governments have a very poor record of improving overall financial management systems and the delivery of basic government services on a sustained and long-term basis, despite OIA grants for financial management improvements and other technical assistance.  The existence of financial management improvement plans and the completion of technical assistance grant projects do not necessarily signify that government systems and services have been improved.  

Fiscal Year 2001 Annual Performance Plan
  
OIA decreased to five the number of performance goals and measures in its fiscal year 2001 performance plan because of the long-term perspective of OIA's technical assistance activities.  OIA explained that it had decided "to focus on those goals and objectives that are achievable by OIA in the short term, are less dependent on the insular governments, and are less affected by external factors."  OIA stated that this revision would result in a more "output"-focused approach rather than "outcomes" approach.  OIA's fiscal year 2001 performance plan includes two goals related to improving infrastructure.  The first relates to completion of multi-year capital plans for each insular area government.  While this is a valid goal and the performance measure would be simple to validate, the performance measure should require that at least one additional capital plan is "implemented" not "completed" because completion of a plan does not necessarily mean that it has been or will be implemented.  The second goal is to increase grant productivity by increasing the ratio of projects completed to projects started and reducing the average completion time of projects.  While this is a valid goal and the performance measures would be simple to validate,
A better measure might be to determine and quantify the extent to which individual grant projects achieved their original stated purposes\goals, rather than determining whether the grants were completed in a timely manner.  OIA's fiscal year 2001 performance plan includes two goals related to improving government systems and services.  The original fiscal year 1999 goal, which was to have financial management improvement plans completed for each of the seven insular area governments, was not met because financial and time constraints prevented timely completion of the plans.  OIA revised this goal, stating that all financial improvement plans would be completed by fiscal year 2005.  While this is a valid goal and the performance measure would be simple to validate, the performance measure should require that at least one additional financial improvement plan is "implemented" not "completed" because completion of a plan does not necessarily mean that it has been or will be implemented.  The second goal was to increase grant productivity by increasing the ratio of projects completed to projects started and reducing the average completion time of
projects.  While this is a valid goal and the performance measures would be simple to validate, a better measure might be to determine and quantify the extent to which individual grant projects achieved their original stated purposes\goals, rather than simply determining whether grants were completed and the timeframes in which they were completed.

See notes at end of table:
A
B
C
D
E
F
G
H
I

GPRA Mission Goals: (1) Assist the Islands in developing more efficient and effective government by providing financial and technical assistance, and
(2) serve as a focal point for the management of federal-island relations by developing and promoting appropriate federal policies.

Fiscal Year 1999 Performance Goal\Measure Fiscal Year 1999 goal achieved?
If not, report adequately explains why and describes a strategy to meet goal in the future? Goal relates to agency's key  management challenges? Fiscal Year 1999 performance plan goal excluded from report? Fiscal Year 1999 performance plan goal changed in report? Adequate explanation of why Fiscal Year 1999
performance plan goal was excluded or changed in report? Agency's performance plan and\or report adequately described the data verification and validation process? Agency acknowledges shortcomings in its data? Agency indicates what steps it will take to correct shortcomings in its data? Selected Comments on Agency Goals\Measures and validation methods

Program Activity 1: Improve Infrastructure

Goal was discontinued in Fiscal Year 2000.

Program Activity 2: Improve Governmental Systems and Services

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of
implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency  adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were
adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

OFFICE OF SURFACE MINING RECLAMATION AND ENFORCEMENT

OSM organized its fiscal year 1999 and fiscal year 2001 performance plans into four goal categories, each of which had subsets of long-term goals.  The long-term goals, in turn, had an annual goal and a target (23 in fiscal year 1999).  These targets typically consistent of percentages, acres or projects.  In its fiscal year 2001 performance plan, OSM has two goal categories, each of which has two annual goals (also referred to as performance measures) and a target for each. 
  
Fiscal Year 1999 Performance Plan and Report

OSM's fiscal year 1999 performance plan generally provided an adequate basis for evaluating OSM's accomplishment of its two primary missions: regulating current coal mining operations and reclaiming\restoring land previously degraded by past coal mining.  For most goals, OSM established a performance baseline that provided a reasonable approach to measuring the accomplishment of mission objectives.  However, we cannot comment on whether the baseline data are valid or whether the annual goals (in terms of numerical amount) are appropriate
Because the plan and report did not contain information on how the baselines were developed.  Also, most annual goals and related targets measured activities rather than program results.
  
OSM's 1999 plan provided overall but not detailed information on the methods that OSM planned to use to validate data.  Although OSM did not discuss any data limitations, the data may not be reliable in all cases.  Specifically, OIG's financial statements audit for fiscal year 1999 disclosed errors in OSM's reporting of performance data regarding the "percentage of sites that are free of off-site impacts to 94%" goal.  The audit report stated that OSM did not consistently gather data for off-site impacts and did not include data on mine sites located on Federal lands. 
  
In its fiscal year 1999 performance report, OSM met or exceeded 14 of 23 annual goals. OSM adequately explained why eight goals were not met and provided a strategy to ensure that the goals will be met in the future or stated that the goals were discontinued.  For one goal that was not met, OSM did not provide a strategy to achieve the goal.  In addition, we found two unexplained differences between the fiscal year 1999 plan and the report.  In one instance, the report
Included performance information on a goal that was not in the plan.  In the other instance, the measurement for the goal "number of acres released from Phase III Performance Bonds" was shown as 100,000 acres in the plan and 90,000 acres in the report.  

Fiscal Year 2001 Performance Plan
  
In its 2001 plan, OSM said that it eliminated or revised 1999 goals primarily to convert the goals to a "process efficiency index" (OSM's term, which was not defined), a performance statistic, a customer satisfaction index, or a workload statistic.  For the 2001 Plan, OSM identified two mission goals with four related annual goals.  The two mission goals were the same as those in its fiscal year 1999 plan and report.  Also, four annual goals were the same as those in its fiscal year 1999 plan, except that the measurement (e.g. number of acres) changed.  The 2001 plan provided detailed information on the data that would be used to measure performance and in most cases described the data collection process and the methods that would be used to validate the data. 
Overall, the mission and annual goals in the fiscal year 2001 plan are an improvement over the fiscal year 1999 goals.

Based on our review of its plan, we believe that OSM could improve its report as follows:

- Provide sufficient information to fully explain the goals and the significance of the goals.
  
- Describe the total program areas for which measures have been established.  For example, one fiscal year 2001 annual goal, under the environmental restoration mission goal, is to reclaim 9,100 acres annually.  The performance report and plan, however,  do provide information on the total number of acres that need to be restored.  In addition, another annual goal for fiscal year 2001, under environmental restoration, is to provide funding for 46 new cooperative Acid Mine Drainage Projects under the Clean Streams Initiative. The Report and plans do not identify the scope or priority of the projects and how many projects are needed.
  
- Add annual goals\measures.  For example, one annual goal for fiscal year 2001 under the environmental restoration mission goal is to reclaim 9,100 acres annually.  The enabling legislation that created OSM established priorities for reclaiming abandoned coal mine lands and also authorized use of funds to mitigate effects on noncoal mining activities after the priorities are completed within a particular state.  We believe that at a minimum there should be goals\measures which address reclamation of the highest priority coal projects including those for protecting public health, safety, general welfare, and property from extreme danger (priority 1); protecting public health, safety, and general welfare from adverse effects (priority 2); and restoring land and water
resources and environment (priority 3).  

*The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency  adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.  We determined that OSM had not addressed shortcomings in its data.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

U.S. FISH AND WILDLIFE SERVICE
  
FWS organized its fiscal year 1999 and fiscal year 2001 performance plans into four goal categories, each of which had subsets of annual goals (15 in 1999 and 17 in 2001).  The annual goals, in turn, had one or more performance targets to measure accomplishments.  These targets typically consisted of units, such as the number of acres or sites, or percentages.
  
Fiscal Year 1999 Performance Plan and Report
  
FWS's fiscal year 1999 performance plan provided an adequate basis for evaluating FWS's accomplishment of its essential mission goals: sustainability of Fish and Wildlife Populations; habitat conservation; and public use and enjoyment.   FWS identified 15 goals and 35 measurements within these mission areas.  For most goals, FWS established a performance baseline that provided a reasonable approach to measuring the accomplishment of mission objectives. 
However, we cannot comment on whether the baseline data are valid or whether the numerical goals for measuring accomplishment are appropriate.  Also, data may not be complete or reliable because data collection methods may not have been developed.  For example, FWS established a performance measure for interpretive, educational, and recreational visits to national wildlife refuges and national fish hatcheries based on the number of visits.  However, FWS currently does not compile visitation data at fish hatcheries and the plan did not describe the method used to capture visitation statistics on wildlife refuges.
  
FWS's  1999 plan provided overall but not detailed information on the methods FWS planned to use to validate data.  FWS acknowledged that there were data limitations for some of the measures including factors such as weather, the use of volunteers to capture data, subjective assessments, and the cost of attaining accurate information to measure performance.  
  
In its fiscal year 1999 performance report, FWS said that it either met or exceeded the measures for 12 of the 15 goals. FWS adequately explained why the measures for the three goals were not met and provided a strategy to achieve the performance goals in the future.
  
Fiscal Year 2001 Performance Plan
  
In its 2001 plan, FWS explained why it eliminated or revised 1999 goals.  The 2001 plan added one mission goal and four performance goals, revised four existing goals, and eliminated three goals.  These changes were made to more fully reflect  FWS's commitment to its partners' efforts to conserve fish and wildlife and their habitats (new mission goal), to provide a different measure
or baseline (revised goals), or to delete objectives that were achieved or that had been merged into other goals (eliminated goals).
  
The 2001 plan generally provided detailed information on the data that would be used to measure performance and in most cases described the data collection process and the methods used to validate the data.  However, FWS did not describe the controls in place or that will be implemented to ensure that the data are complete, representative, and accurate.  For example, the 2001 plan does not provide a discussion of the controls that are or will be in place to ensure that the reported acres under the Habitat Conservation mission goal have been restored or enhanced.  *The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of implementing a Trust Management Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.


D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report. 

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency  adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were
adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.  The Fiscal Year 1999 performance report does not discuss data shortcomings; however, the Fiscal Year 2001 performance plan does describe data limitations, particularly the need to develop baseline data.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.

U.S. GEOLOGICAL SURVEY

USGS's fiscal year 1999 performance plan and report had two mission goals, two long term goals, and ten annual performance goals\measures.  The fiscal year 2001 performance plan had two mission goals, two long term goals, and twelve annual performance goals\measures.  The measures typically consisted of units of activity, such as number of meetings held or hazardous monitoring networks maintained. 

Fiscal Year 1999 Performance Plan and Report.
  
The two mission and long-term goals in the fiscal year 1999 performance plan appropriately address USGS's major programs. The specific annual performance goals\measures are adequate indicators of certain program activities.  However, the report does not communicate what was actually accomplished by performing those activities.  For example, six hazards monitoring networks were maintained.  We consider the term "maintained" to be vague as to what was achieved. For instance the networks could have been maintained to be operational 99 percent of
the time or the equipment could be maintained but not operational.  Although the plan describes the networks as consisting of flood, earthquake, volcano, landslide, geomagnetic, and an integrated monitoring network, the report does not address what benefits were achieved by the operation of the networks, whether lives were saved by early warning or response, whether property damage
was minimized by early warning, or describe how network operations achieved the overall performance objective.

USGS's fiscal year 1999  report did not specifically identified fiscal year the sources of data or the data verification and validation methodology for its 10 performance goals\measures. However, in the USGS fiscal year 2001 performance plan, Section III "Additional GPRA Information," the reader finds Subsection 3.4, "Data Verification and Validation" where there is a general discussion
of USGS's performance reporting system for all fiscal year 1999 performance data.
  
We are not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of the USGS's data collection, verification, and validation processes for each of the USGS's ten performance goals\measures.  However, the fiscal year 2001 plan does describe the validation and verification processes for each performance goal\measure.  In our opinion, these processes would produce reliable data if adequate controls are implemented to ensure the integrity of the data.
  
Fiscal Year 2001 Performance Plan.
  
USGS's fiscal year 2001 performance plan continued the 10 performance measures from fiscal year 1999 with only minor change in the basis for measuring a specific data element, which we considered to be a minor adjustment to the performance reporting.  For fiscal year 2001 USGS did include a customer satisfaction measure for each of the two long-term goals.  However, it
Did not provide a definition of the new measure; specifics concerning the data collection methodology, sources and limitations; or the validation process as it did for each of its ten other performance measures. *The Office of the Special Trustee for American Indians is responsible for the "Management of Indian Trust Funds."  OST did not prepare a performance plan for Fiscal Year 1999 or produce a performance report.  OST is in the process of implementing a Trust Management
Improvement Project and is under a Court Order to produce a quarterly report of its progress on the High Level Implementation Plan.

D. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were excluded from the report.

E. We compared the agency's Fiscal Year 1999 performance plan and Fiscal Year 1999 performance report to determine if any planned goals\measures were changed in the report.

F. If the agency excluded or changed any of its Fiscal Year 1999 planned goals\measures, indicate whether the agency adequately explained why it excluded or changed the goal\measure.

G. We were not able to provide assurance regarding the validity and reliability of data because we have not assessed the adequacy of data collection, verification, and validation processes for each of the agency's Fiscal Year 1999 performance goals\measures.  However, we attempted to determine whether the agency's verification and validation process as described in its performance plan and\or report could produce reliable data, assuming that there were
adequate controls to ensure the integrity of the data throughout the process.

H. In some instances an agency may have identified shortcomings or limitations in its data collection processes.

I. Having identified shortcomings or limitations in its data collection processes, did the agency indicate what steps it would take to correct\overcome the problems.  A Y response indicates that the agency has identified such steps or discontinued the performance goal\measure in its Fiscal Year 2001 performance plan. Responses to Column I should reconcile to the number of Y responses in Column H.