Millennium Challenge Corporation: Vanuatu Compact Overstates
Projected Program Impact (11-JUL-07, GAO-07-909).
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) for foreign assistance. Congress has
appropriated almost $6 billion to MCC. As of March 2007, MCC had
signed almost $3 billion in compacts with 11 countries, including
a 5-year, $65.7 million compact with Vanuatu. MCC states that the
Vanuatu compact will have a transformational effect on the
country's economy, increasing per capita income and GDP and
benefiting 65,000 poor, rural people. GAO examined (1) MCC's
methods of projecting economic benefits, (2) MCC's portrayal and
analysis of the projected benefits, and (3) risks that may affect
the compact's impact. GAO reviewed MCC's analyses and met with
officials and business owners in Vanuatu as well as with other
donors.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-909
ACCNO: A72527
TITLE: Millennium Challenge Corporation: Vanuatu Compact
Overstates Projected Program Impact
DATE: 07/11/2007
SUBJECT: Budgeting
Cost analysis
Cost control
Economic growth
Economically depressed areas
Financial statements
Impacted areas
Income statistics
Program evaluation
Federal corporations
Federal aid to foreign countries
Risk management
Cost estimates
Vanuatu
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-909
* [1]Results in Brief
* [2]Background
* [3]Description of Vanuatu
* [4]Vanuatu's Economy
* [5]MCC's Compact with Vanuatu
* [6]MCC Projected Compact's Impact Using Estimates of Benefits,
* [7]MCC Calculated Benefits, Costs, and Beneficiaries
* [8]Benefits
* [9]Costs
* [10]Beneficiaries
* [11]MCC Used Projected Benefits and Costs to Determine ERR and E
* [12]MCC's Data Do Not Support Its Portrayal of Compact Benefits
* [13]Portrayal of Per Capita Income Benefit Suggests Larger Effec
* [14]Portrayal of GDP Benefit Differs from Underlying Analysis
* [15]Portrayal of Poverty Reduction Does Not Quantify Benefits to
* [16]Estimates of Compact Benefits Are Generally Reasonable, but
* [17]Several Risks May Lead to Reduced Project Benefits
* [18]Construction Costs May Exceed Contingencies
* [19]Compact Benefits Are Likely to Accrue More Slowly Than Proje
* [20]Project Maintenance after Compact Expiration Cannot Be Ensur
* [21]Induced Benefits Require Tourism Providers and Agricultural
* [22]Efficiency Gains May Not Cause Measurable Change in Per Capi
* [23]Conclusions
* [24]Recommendations
* [25]Agency Comments and Our Evaluation
* [26]Compact Impact on Incomes and Growth
* [27]Compact Beneficiaries
* [28]Program Risks
* [29]Per Capita Income Impact
* [30]Calculation of Beneficiaries
* [31]GAO Comments
* [32]GAO Contact
* [33]Staff Acknowledgments
* [34]Order by Mail or Phone
Report to the Chairman, Committee on Foreign Affairs, House of
Representatives
United States Government Accountability Office
GAO
July 2007
MILLENNIUM CHALLENGE CORPORATION
Vanuatu Compact Overstates Projected Program Impact
GAO-07-909
Contents
Letter 1
Results in Brief 3
Background 6
MCC Projected Compact's Impact Using Estimates of Benefits, Costs, and
Catchment Area 12
MCC's Data Do Not Support Its Portrayal of Compact Benefits 17
Several Risks May Lead to Reduced Project Benefits 24
Conclusions 29
Recommendations 30
Agency Comments and Our Evaluation 30
Appendix I Objectives, Scope, and Methodology 35
Appendix II Prior Development Assistance to Vanuatu 38
Appendix III MCC Estimates of Per Capita Income Impact and Poor, Rural
Beneficiaries 40
Appendix IV MCC Cost Estimates for the Vanuatu Compact 47
Appendix V Illustrative Alternative Calculations of Vanuatu Compact Impact
49
Appendix VI Comments from the Millennium Challenge Corporation 52
GAO Comments 57
Appendix VII GAO Contact and Staff Acknowledgments 63
Tables
Table 1: Average Annual Contributions of Top Five Donors to Vanuatu in
2004 and 2005 38
Table 2: Number of Poor, Rural Beneficiaries of Vanuatu Compact, as
Calculated by MCC 43
Table 3: Effect of Calculation and Data Errors on Beneficiary Population
in Efate and Santo 44
Table 4: Effect of MCC Assumptions on Beneficiary Population in Efate and
Santo 44
Table 5: Summary of Compact Impacts as Presented by MCC, with GAO
Recalculations 49
Table 6: Summary of Compact ERR, GDP, and Per Capita Income impacts under
Alternative Scenarios of Accounting for Risks to Benefits 50
Table 7: Summary of Selected Compact Project Impacts under Alternative
Scenarios Accounting for Risks to Benefits 51
Figures
Figure 1: Location of Vanuatu 7
Figure 2: Development and Implementation of Vanuatu Compact 9
Figure 3: MCC Vanuatu Projects by Size and Location 11
Figure 4: MCC Statement of Impacts in March 2006 Congressional
Notification 12
Figure 5: MCC's Logic Model for the Vanuatu Compact 14
Figure 6: MCC Benefits Calculations as Source of Statements about Impact
of Vanuatu Compact 17
Figure 7: Vanuatu Compact's Projected Impact on Real Per Capita Income
According to MCC Statement and MCC Data Relative to 2005 Per Capita Income
19
Figure 8: MCC Methodology for Projecting Vanuatu Compact's Impact on Real
Per Capita Income 20
Figure 9: Vanuatu GDP Growth with and without MCC Compact 21
Figure 10: MCC Analysis of Distribution of Vanuatu Compact Benefits 22
Figure 11: MCC Compact's Projected Impact on Vanuatu Per Capita Income
with and without Adjusted Population Estimates 41
Figure 12: MCC Compact's Projected Impact on Vanuatu Per Capita Income in
Real Terms Relative to Per Capita Income without MCC 42
Figure 13: MCC Efate Ring Road Catchment Area 45
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
Abbreviations
ADB Asian Development Bank
CEO Chief Executive Officer
ERR economic rate of return
GDP gross domestic product
IMF International Monetary Fund
MCA Millennium Challenge Account
MCC Millennium Challenge Corporation
PWD Public Works Department
United States Government Accountability Office
Washington, DC 20548
July 11, 2007
The Honorable Tom Lantos
Chairman
Committee on Foreign Affairs
House of Representatives
Dear Mr. Lantos:
In January 2004, Congress established the Millennium Challenge Corporation
(MCC) to administer the Millennium Challenge Account (MCA) for foreign
assistance. MCC's mission is to reduce poverty by supporting sustainable,
transformative economic growth in developing countries that create and
maintain sound policy environments. MCC carries out this mission by
funding projects or activities in developing countries that have
demonstrated a commitment to ruling justly and democratically, encouraging
economic freedom, and investing in people. Congress appropriated almost $6
billion for fiscal years 2004 to 2007 to MCC, and the President has
requested an additional $3 billion in MCC funding for fiscal year 2008.
As of March 2007, MCC had signed compacts with 11 countries, totaling
approximately $3 billion, including a 5-year, $65.7 million compact with
the Pacific island nation of Vanuatu.^1 Although MCC's compact with
Vanuatu is its smallest compact monetarily, it provides by far the largest
amount relative to the country's population and gross domestic product
(GDP).^2 In a process known as due diligence, MCC analyzed Vanuatu's
proposal for compact assistance to determine the compact's expected
economic rate of return (ERR)^3 and impact on poverty reduction and
economic growth. After completing its due diligence analysis, MCC stated
in its compact and in its notification to Congress that it expects the
Vanuatu compact to have a transformational impact on Vanuatu's economic
development--an effect MCC defines as "a dramatic and long-lasting impact
on poverty reduction through sustainable economic growth."^4 MCC states
that its compacts will provide or contribute to a transformational impact
in 5 of its 11 compacts.^5
^1An MCC compact is an agreement between the U.S. government, acting
through MCC, and the government of a country eligible for MCA assistance.
^2MCC's $65.7 million compact with Vanuatu provides $317 per capita; in
contrast, MCC's $547 million compact with Ghana--its largest
compact--provides $25 per capita. The amounts provided per capita by the
11 compacts signed to date range from $6 for Madagascar to $317 for
Vanuatu.
^3Project cash flows are determined by comparing program spending against
future expected increases in value added or income. The internal rate of
return is calculated for these cash flows to summarize the economic
impact. MCC refers to this internal rate of return as the economic rate of
return.
At your request, we reviewed MCC's economic analyses of the Vanuatu
compact. Specifically, we examined
o MCC's methods of projecting the compact's economic benefits and
methods of calculating the benefits,
o MCC's portrayal and analysis of the projected benefits, and
o risks that could affect the compact's impact on poverty
reduction and economic growth.
We reviewed MCC's record of due diligence and supplemented this
review with interviews with MCC officials to identify MCC's logic,
data, methods, and assumptions for determining the compact's
projected costs and benefits, and to identify the projected
effects on GDP, per capita income, and poverty. We evaluated MCC's
statements about the compact's impacts in its notification to
Congress, in the Vanuatu compact, and in its "investment memo"^6
by comparing these statements with the underlying analyses and
data used to support them. We could not validate most of
^4Millennium Challenge Account, Best Practices in Compact Development
(Washington, D.C.: 2006).
^5For example, in Nicaragua, MCC expects that the compact will transform
project areas into an engine of economic growth; in El Salvador, MCC
states that the compact provides an historic opportunity to transform the
country's economic development; and in Armenia, MCC is undertaking road
and irrigation projects to transform the economic performance of Armenia's
agricultural sector.
^6The "investment memo" is an MCC internal document prepared by MCC's
compact assessment team and submitted to MCC's investment
committee--consisting of MCC's Chief Executive Officer (CEO), vice
presidents, and other senior officials. The committee reviews the memo and
decides whether to recommend proceeding to compact negotiations.
MCC's underlying data and assumptions because the data were not
available or could not be checked within the time frames of our
engagement. We examined MCC's methodologies and checked the
analyses for calculation errors. Further, we identified risks to
MCC's compact results based on our review of MCC's internal
documentation, donor reporting, and academic development
literature. We interviewed Vanuatu and MCC officials and contacted
the contractor that assisted in MCC's analyses. We also
interviewed interested parties such as tourism and agriculture
business owners in Vanuatu. We focused our analysis and field work
on MCC's three transportation infrastructure projects on Vanuatu's
two most populous islands, Santo and Efate, which represent 56
percent of compact cost. We modeled several areas of project risks
to illustrate their maximum impact on the economic analyses of
ERR, GDP, and per capita income. In modeling these risks, we used
the data from MCC's economic analyses; we did not validate these
data. We conducted our review from August 2006 through May 2007 in
accordance with generally accepted government auditing standards.
Results in Brief
MCC projected the impact of the Vanuatu compact by estimating the
program's benefits, costs, and beneficiaries and calculating the
compact's effect on per capita income, GDP, and poverty reduction.
According to MCC, transportation infrastructure improvements will
provide direct benefits from construction spending in the local
economy, reduced transportation costs, and improved services. The
improved infrastructure will also provide induced benefits from
growth in Vanuatu's tourism and agriculture sectors. MCC estimated
the value of these benefits over a 20-year period, beginning in
full in 2008 or 2009 and growing each year. MCC developed its
project cost estimates based on existing cost estimates prepared
for the government of Vanuatu and for another donor. To determine
the number of poor, rural beneficiaries, MCC defined a catchment
area--the geographic area in which benefits may be expected to
accrue--using maps of Vanuatu and data from the most recent
Vanuatu census. Using its projected benefit and cost data, MCC
calculated (1) the compact's ERR, comparing projected benefits to
projected costs; (2) per capita income, determining the total
benefits and dividing the total value by Vanuatu's baseline
population; and (3) the compact's effect on Vanuatu's GDP by
computing the total benefits added to the economy.
MCC's portrayal of the Vanuatu compact's impact does not reflect
the data and analysis underlying its projections of the compact's
benefits. MCC states that as a result of the compact, per capita
income will increase by approximately $200, or 15 percent, by 2010
and $488, or 37 percent, by 2015. This statement suggests that per
capita income in 2010 and 2015 will be, respectively, 15 percent
and 37 percent higher than without the compact. However, MCC's
underlying data show that these figures represent the sum of gains
in per capita income for individual years relative to 2005 rather
than actual, or net, annual gains as of 2010 or 2015. For example,
MCC sums the per capita income gains relative to 2005 for 2006 to
2010, averaging 3 percent, as 15 percent in 2010, without stating
that these gains are cumulative. Our analysis of MCC's data shows
that actual gains in per capita income, relative to income in
2005, would be $51, or 3.9 percent, in 2010 and $61, or 4.6
percent, in 2015. Likewise, MCC states that Vanuatu's GDP will
increase by "an additional 3 percent a year." However, MCC's
underlying data and calculations show that while the level of
Vanuatu's GDP will grow by 6 percent in 2007, the economy's growth
rate in subsequent years continues at approximately 3 percent,
MCC's assumed rate without the compact. Regarding MCC's portrayal
of the compact's impact on poverty reduction, MCC's publicly
available documents state that the compact is expected to benefit
approximately 65,000 poor, rural inhabitants "living nearby and
using the roads to access markets and social services." However,
MCC's underlying documentation shows that these 65,000
beneficiaries will not receive the majority of the benefits but
will instead share the 43 percent of the compact's monetary
benefits that MCC expects to go to the local population; the
remaining 57 percent will go to beneficiaries such as tourism
services providers, transport companies, and local businesses.
Finally, although MCC's estimates of compact benefits and
beneficiaries are generally reasonable, some of the calculations
and assumptions it used are problematic. Correcting MCC's
calculations of per capita income benefits slightly reduces the
estimated benefit, and correcting MCC's calculations and fully
discounting some of its assumptions regarding the count of
beneficiaries would reduce estimated beneficiaries on Santo and
Efate by about one-third.
We identified five key risks that could affect the Vanuatu
compact's projected impact on poverty reduction and economic
growth.
o The contingencies included in MCC's calculations of construction
costs may not be sufficient to cover potential cost overruns. We
received MCC cost estimate documentation for 5 of MCC's 11
construction projects showing that MCC's estimates for these 5
include design contingencies of 20 percent; however, a previous
study found average cost overruns of more than 20 percent in
transportation infrastructure projects in other countries.
Further, the risk of excessive cost overruns is significant in a
small country such as Vanuatu. Any construction cost overrun could
cause MCC to reduce the compact's scope and therefore its
benefits.
o Although the compact's benefits are projected to begin shortly
after completion of the projects, some of these benefits will
likely accrue more slowly. For example, according to agricultural
and timber producers, their businesses will likely respond
gradually to any increased market opportunities.
o Whereas the benefit projections assume continued maintenance of
completed projects, MCC's ability to ensure such maintenance will
end in 2011. Moreover, previous donors to Vanuatu have found
Vanuatu's record of maintaining donor projects to be poor. Reduced
maintenance would lead to reduced benefits from the project.
o The projected induced benefits from expanded tourism and
agriculture depend on businesses and rural inhabitants responding
to opportunities created by improved infrastructure.
o Efficiency gains that MCC counts as direct benefits, such as
time saved in transit, may not be put to economic use and result
in increased per capita income, as MCC projects.
Our analysis of these areas of risk illustrates the extent to
which MCC's benefit projections are dependent on assumptions of
immediate realization of benefits, successful long-term
maintenance, realization of induced benefits, and benefits from
efficiency gains. Accounting for these risks can reduce overall
compact ERR from 24.2 percent, as projected by MCC, to between 5.5
percent and 16.5 percent, and some projects may have a negative
ERR.^7
To help better express and determine the impact of its compacts,
this report recommends that the Chief Executive Officer (CEO) of
MCC (1) revise the public reporting of the projected impact of the
Vanuatu compact, (2) assess whether similar statements in other
compacts accurately reflect underlying data, and (3) improve MCC's
economic analysis by phasing costs and benefits and more fully
accounting for risks to project benefits.
^7We modeled risk scenarios by first assuming the phasing of costs and
benefits. We then combined this phasing with three additional models that
assumed a lack of maintenance, a lack of induced benefits, and a lack of
monetized efficiency gains. See appendix V for more information about our
methodology.
We provided a draft of this report to MCC. In commenting on the
draft, MCC did not directly acknowledge our recommendations. MCC
acknowledged that its use of projected cumulative compact impact
on income and growth was misleading, but asserted that it had no
intention to mislead and that its portrayal of projected compact
benefits was factually correct. MCC questioned our finding that
its underlying data and analysis do not support its portrayal of
compact benefits and our characterization of the program's risks.
In response to MCC's comments, we clarified the terms and
presentation of our analysis of MCC's statements and data and
provided additional information on the basis for our findings. We
also provided technical corrections where appropriate.
Background
Description of Vanuatu
Vanuatu consists of 83 islands spread over hundreds of miles of
ocean in the South Pacific, 1,300 miles northeast of Sydney,
Australia (see fig. 1). About 39 percent of the population is
concentrated on the islands of Santo and Efate. Vanuatu's capital,
Port Vila, is on Efate, and Vanuatu's only other urban center,
Luganville, is on Santo. The country has three official
languages--English, French, and Bislama--but more than 100 other
dialects are also spoken. Traditional custom chiefs have a
significant role in the state particularly in rural areas. Civil
unrest has caused occasional disruptions; for example, riots
erupted in Port Vila over the misappropriation of assets from the
Vanuatu National Provident Fund in 1998, and an ethnic conflict on
Efate led Vanuatu's parliamentary government to declare a state of
emergency in March 2007.
Figure 1: Location of Vanuatu
Vanuatu's Economy
In the past decade, Vanuatu's real GDP growth averaged 2 percent,
although more rapid population growth led to a decline in per
capita GDP over the same period. Average growth of real GDP per
capita was negative from 1993 to 2005. In its economic analyses,
MCC used a baseline 2005 Vanuatu per capita income of $1,326. An
estimated 40 percent of Vanuatu's population of about 207,000 has
an income below the international poverty line of $1 per day.
Agriculture and tourism are the principal productive sectors of
Vanuatu's economy, contributing approximately 15 percent and 19
percent to GDP, respectively. Although agriculture represents a
relatively small share of Vanuatu's overall economy, approximately
80 percent of Vanuatu's residents live in rural areas and depend
on subsistence agriculture for food and shelter, selling surplus
commodities to generate cash for school fees, transportation,
consumer goods, and services. Copra (dried coconut) is the main
cash crop; kava and cacao are also grown. The tourism sector is
dominated by expatriates of foreign countries living in Vanuatu,
who also predominate in other formal sectors of the economy such
as plantation agriculture and retail trade. According to the Asian
Development Bank (ADB), tourism is one of the most promising
sectors in Vanuatu in terms of its potential for earning foreign
exchange and generating employment. According to a survey
conducted by the Vanuatu National Statistics office, the largest
markets for Vanuatu tourism are Australia and New Zealand. Despite
prior donor efforts, private sector development in Vanuatu's
tourism and agricultural sectors faces challenges, including
political uncertainty, and high costs of doing business. (See app.
II for a summary of donor efforts and Vanuatu's development
challenges.)
MCC's Compact with Vanuatu
On May 6, 2004, MCC determined that Vanuatu was eligible to submit
a compact proposal for Millennium Challenge Account (MCA)
funding.^8 Vanuatu's proposal identified transportation
infrastructure as a key constraint to private-sector development.
Based on the analysis performed during approximately 5 months of
due diligence,^9 MCC signed a 5-year, $65.7 million compact with
the government of Vanuatu on March 2, 2006, with entry into
force^10 on April 28, 2006.^11 In keeping with its statutory
requirements, MCC submitted a congressional notification of the
compact signing on March 7, 2006.
^8The Millennium Challenge Act of 2003 requires MCC to determine whether
countries are eligible for MCA assistance each fiscal year. Countries with
per capita income at or below a set threshold may be selected as eligible
for assistance if they meet MCC indicator criteria and are not statutorily
barred from receiving U.S. assistance. MCC uses 16 indicators divided into
three categories: Ruling Justly, Encouraging Economic Freedom, and
Investing in People. To be eligible for MCA assistance, countries must
score above the median relative to their peers on at least half of the
indicators in each category and above the median on the indicator for
combating corruption. GAO, Millennium Challenge Corporation: Compact
Implementation Structures Are Being Established; Framework for Measuring
Results Needs Improvement, [35]GAO-06-805 (Washington, D.C.: July 28,
2006).
Figure 2 illustrates the chronology of the development and
implementation of the Vanuatu proposal and compact.
Figure 2: Development and Implementation of Vanuatu Compact
^9Prior to the due diligence process, a transaction team conducts a
preliminary assessment of each proposal and reports its findings in an
internal "opportunity memo" to the MCC investment committee. If the
opportunity memo is approved, the team launches a detailed due diligence
review, using MCC's published proposal guidelines and criteria to analyze
project costs and benefits and assess the proposal's consultative process,
rationale, environmental and social impact, and sustainability. The
transaction team makes recommendations based on its assessment of the
proposal in an investment memo to the MCC investment committee.
^10MCC and the country's accountable entity must complete supplemental
agreements, including a disbursement agreement and a procurement
agreement, before the compact enters into force and funds are disbursed.
^11By U.S. law, all compacts must end within 5 years of entry into force.
The $65.7 million Vanuatu compact includes $54.5 million for the
rehabilitation or construction of 11 transportation infrastructure assets
on 8 of Vanuatu's 83 islands, including roads, wharves, an airstrip, and
warehouses (see fig. 3). The compact also includes $6.2 million for an
institutional strengthening program to increase the capacity of the
Vanuatu Public Works Department (PWD) to maintain transportation
infrastructure.^12 The remaining $5 million is for program management and
monitoring and evaluation. More than half of the compact, $37 million, is
budgeted for three road projects on Santo and Efate islands. On both
islands, the compact will upgrade existing roads, while on Santo the
compact also includes five new bridges on an existing road. To oversee and
manage the compact programs, the Vanuatu government has established
MCA-Vanuatu, an independent entity housed within the Vanuatu Ministry of
Finance and Economic Management.
^12The institutional strengthening program includes $5.74 million for
equipment purchases; of this amount, $1.4 million is provided directly to
PWD and the remainder will purchase equipment for the use of the MCC
construction contractor, to be turned over to the PWD in specified
condition 4 years later.
Figure 3: MCC Vanuatu Projects by Size and Location
The compact and the congressional notification state that the compact will
have a transformational impact on Vanuatu's economic development,
increasing average per capita income by approximately $200--15 percent--by
2010 and increasing total GDP by "an additional 3 percent a year." The
investment memo further quantifies the per capita income increase as
$488--37 percent--by 2015. The compact and the congressional notification
further state that the compact will provide benefits to approximately
65,000 poor, rural inhabitants (see fig. 4).
Figure 4: MCC Statement of Impacts in March 2006 Congressional
Notification
MCC Projected Compact's Impact Using Estimates of Benefits, Costs, and Catchment
Area
To project the Vanuatu compact program's benefits, costs, and
beneficiaries and calculate the compact's impact on per capita income,
GDP, and poverty reduction, MCC made site visits to Vanuatu and reviewed
available documents to gather needed data. MCC determined that investments
in transportation infrastructure will lead to increases in incomes and
used the available data and made assumptions to determine the compact's
benefits and costs. MCC also estimated the number of beneficiaries within
a defined catchment area--the geographic area in which benefits may be
expected to accrue. MCC then used the benefits and costs to calculate
summary statistics about the compact's ERR and effects on Vanuatu's GDP
and per capita income.
MCC Calculated Benefits, Costs, and Beneficiaries
To prepare its economic analyses of the compact program's benefits, costs,
and number of beneficiaries, MCC sent a team of employees and contractors
to Vanuatu. The contractor MCC retained had previously served as a
consultant to the ADB in Vanuatu and had prepared a multiyear study of
projects proposed to the ADB as part of its Vanuatu Outer Islands
Infrastructure Development Project in 2000 to 2003. Seven of MCC's 11
transportation infrastructure projects were initially developed as part of
the ADB study. While in Vanuatu, the MCC team met with officials
representing other donors, Vanuatu's Department of Economic and Sectoral
Planning, the National Statistics Office, and the Vanuatu PWD, among
others. The MCC team also met with a number of interested and informed
parties representing the transportation, tourism, and agriculture sectors
and reviewed a number of documents to gather the information needed for
its economic analyses.^13
Benefits
MCC's compact aims to benefit poor, rural agricultural producers and
providers of tourism-related goods and services by reducing transportation
costs and improving the reliability of access to transportation services.
MCC's logic model for the compact posits that improvements in
infrastructure will lead to (1) direct benefits from increased
transportation reliability, construction spending, and reduced
transportation costs and (2) induced benefits from greater tourism and
agricultural trade. The direct and induced benefits in turn lead to
increases in per capita income and GDP and reduction in poverty (see fig.
5).
^13MCC used Vanuatu statistical surveys such as the Vanuatu 1999 Household
Income and Expenditure Survey, 1993 Agricultural Survey, 2004 Tourist
Survey, and a monthly Hotel Occupancy Survey as baseline data sources.
Some data were updated to 2005 values from earlier data based on growth
factors. MCC assessed the data it used as the best available, but noted
that some data have shortcomings. Other data--such as the amount of
spending by expatriates in Vanuatu, and the percentage of tourist spending
that remains in the country--were not available, and MCC had to estimate
their value. Since signing the compact with Vanuatu, MCC has funded
additional surveys to improve the quality of baseline data and track key
compact performance indicators.
Figure 5: MCC's Logic Model for the Vanuatu Compact
MCC calculated several different direct and induced benefits for the
compact's projects over a 20-year period. Direct benefits include the
local value added of construction spending in the economy, reduced
spoilage of agricultural goods, time saved in transit on the improved
roads, and reduced user costs for operators of ships, aircraft, and
vehicles.^14 Induced benefits are those projected to result from Vanuatu's
response to new economic opportunities in tourism and agriculture. For
example, MCC assumes that the projects will cause Vanuatu tourism to grow
at 7 percent per year instead of the recent rate of 5 percent; tourists
will increase their daily spending; new hotel rooms will be constructed
and hotel occupancy will increase; and crop, livestock, and fisheries
production will increase. Benefits from the completed projects are counted
as beginning in full in 2008 or 2009 and are assumed to increase a minimum
of 3 percent every year thereafter. MCC expects the benefits of the
compact to flow from different sources, depending on the project and its
location. In Efate, the Ring Road is expected to provide direct benefits
from decreased road user costs and induced benefits through tourism and
foreign resident spending. In Santo, MCC anticipates benefits from all of
these efforts, as well as the induced benefit of increased agricultural
production. On other islands, where tourism is not as developed, the
benefits primarily derive from user cost savings and increased
agriculture.^15
^14MCC determined reduced road user costs by means of (1) traffic counts
prepared by Vanuatu's PWD and MCA-Vanuatu for the MCC and (2) an
estimation of road user costs per kilometer based on the Highway Design
and Maintenance Standards Model (HDM)-III developed by the World Bank.
HDM-III is a software package that provides the economic decision criteria
for evaluating road construction or maintenance strategies. MCC determined
the costs to road users by multiplying the average annual daily traffic on
the road by the length of the road and by the average road user cost per
kilometer. Road user cost includes vehicle operating costs and the value
of passengers' time. The difference in user cost before and after
construction provides the benefit.
Costs
To calculate compact construction and maintenance costs, MCC used existing
data from previous contractor studies and data from the Vanuatu PWD. In
MCC's economic model, construction costs are assumed to be incurred in the
first year after compact signing. Although listed on the cost side of the
model, MCC also counted the effect of the construction spending on the
economy as a benefit in the year it takes place.^16 According to MCC's
contractor, it used two different primary sources in developing the MCC
cost estimates:
o The contractor used its own 2003 analysis for the ADB projects
and updated its analysis to account for inflation and changes in
project scope.
o The contractor used a cost analysis prepared by a different
contractor for the government of Vanuatu, dated approximately
2004, as a basis for MCC's cost estimates for the Santo and Efate
projects. MCC's contractor then prepared a new estimate because
MCC's projects had a different scope of work. We requested from
MCC a copy of the original report to the government of Vanuatu.
According to MCC officials, MCC did not have it and the government
of Vanuatu was not willing to provide the report to MCC for our
review. These Santo and Efate projects represent 56 percent of
compact cost.
^15Although not included in the economic analyses, other benefits may
accrue to Vanuatu from the MCC compact. Increased economic activity in
tourism may have spillover benefits for other sectors of the economy, and
the welfare of Vanuatu's citizens may improve due to increased access to
health care and educational opportunities.
^16MCC assumed that 16 percent of the total cost of construction would be
captured in the local economy and counted this as a benefit.
MCC assumed a cost for continued annual maintenance based on past
estimated maintenance costs provided by the PWD. MCC applied
assumptions and estimated maintenance costs based on incomplete
information, because the PWD did not track maintenance costs in
its budget. For some projects, MCC assumed that maintenance costs
would decrease once the construction project was complete. MCC
also estimated and included the cost of a periodic major
reinvestment for some projects. For example, on the Efate Ring
Road, MCC assumed a cost for rehabilitation in 2017 and 2026.
Beneficiaries
MCC assessed the number of poor, rural beneficiaries by
determining the catchment area--the geographic area in which
benefits may be expected to accrue. For example, MCA-Vanuatu
officials told us that they defined the catchment area in Efate as
consisting of villages within 3 miles of the rehabilitated Efate
Ring Road. MCC used Vanuatu maps to identify villages in the
catchment area and used the 1999 Vanuatu National Population and
Housing Census to determine the number of persons living in those
villages. In all, MCC calculated that approximately 65,000 poor,
rural people on the eight islands would benefit from MCC projects.
MCC Used Projected Benefits and Costs to Determine ERR and Effects
on GDP and Per Capita Income
MCC used its projection of costs and benefits over a 20-year
period as the basis for calculating three summaries of the
compact's impact: its ERR, effect on per capita income, and effect
on GDP (see fig. 6). The compact's ERR reflects the ratio of the
benefits of the compact in relation to its costs, expressed as a
percentage. For the Vanuatu compact, MCC reported an overall
compact ERR of 24.7 percent over 20 years.^17
^17In its final April 2006 economic analysis, MCC adjusted this
calculation downward slightly to 24.2 percent.
Figure 6: MCC Benefits Calculations as Source of Statements about
Impact of Vanuatu Compact
MCC also prepared a sensitivity analysis to assess how a range of
possible outcomes would affect compact results. MCC's tests
included
o a 1-year delay of the start date for accrued benefits;
o a 20 percent increase of all costs;
o a 20 percent decrease of all benefits; and
o a "stress test," with a 20 percent increase of all costs and a
20 percent decrease of all benefits.
MCC tested a best-case scenario based on a 10 percent increase in
benefits and a 10 percent decrease in costs. For the overall
compact, MCC calculated a best-case ERR of 30.2 percent and
worst-case of 13.9 percent. MCC also used the cost and benefit
data for the compact projects to determine the projects' expected
impacts on Vanuatu's per capita income and GDP.
MCC's Data Do Not Support Its Portrayal of Compact Benefits
The anticipated benefits of the Vanuatu compact that MCC stated in
March 2006 in the compact and in its Congressional Notification do
not accurately reflect MCC's supporting data. MCC projects an
increase in per capita income of $200 by 2010, but this increase
is cumulative; MCC's underlying analysis shows a projected
increase of $51 in 2010. Similarly, while MCC's statement about
the compact's transformational effect on GDP could be interpreted
as increasing the growth rate, the projected effect on GDP
represents an increase to the level, not the growth rate, of GDP.
The supporting data show the growth rate remaining roughly the
same as it would be without the compact. In addition, the 65,000
rural poor cited by MCC may share less than half of the compact's
benefits with others in Vanuatu who are not poor or not rural. In
addition, although MCC's estimates of project benefits and
beneficiaries are generally reasonable, MCC made some calculation
errors and questionable assumptions in developing these estimates.
Correcting MCC's calculations of the per capita income benefit
slightly reduces the benefit to $49, or 3.7 percent, in 2010;
correcting calculations and fully discounting some of MCC's
assumptions regarding beneficiaries would reduce the estimate of
beneficiaries on Santo and Efate by about one-third.
Portrayal of Per Capita Income Benefit Suggests Larger Effect
Than Data Support
MCC's portrayal of the compact's projected impact suggests a
greater effect on Vanuatu's per capita income than its analysis
supports. In its publicly available documents such as the Vanuatu
compact, and the congressional notification, issued in March 2006,
MCC states that the transportation infrastructure project "is
expected to have a transformational impact . . . increasing
average income per capita (in real terms) by approximately $200,
or 15 percent of current income per capita, by 2010." In addition,
MCC's investment memo states that the compact will cause per
capita income to increase by $488, or 37 percent, by 2015.
MCC's statements suggest that as a result of the program, average
incomes in Vanuatu will be 15 percent higher in 2010 and 37
percent higher in 2015 than they would be without the compact.
However, MCC's underlying data show that these percentages
represent the sum of increases from per capita income in 2005 that
MCC projects for each year. For example, according to MCC's data,
Vanuatu's per capita income in a given year between 2006 and 2010
will range from about 2 percent to almost 4 percent higher than in
2005; however, MCC sums these percentages as 15 percent, without
stating that this percentage is a cumulative increase from 2005.
Our analysis of MCC's data shows that actual gains in per capita
income, relative to income in 2005, would be $51, or 3.9 percent,
in 2010 and $61, or 4.6 percent, in 2015 (see fig. 7).
Figure 7: Vanuatu Compact's Projected Impact on Real Per Capita Income
According to MCC Statement and MCC Data Relative to 2005 Per Capita Income
Note: MCC's statement: "Increasing average income per capita (in real
terms) by approximately $200 or 15 percent of current income per capita by
2010" and by $488--37 percent--by 2015.
Figure 8 further illustrates MCC's methodology in projecting the compact's
impact on per capita income levels for 2010 and 2015.
Figure 8: MCC Methodology for Projecting Vanuatu Compact's Impact on Real
Per Capita Income
Portrayal of GDP Benefit Differs from Underlying Analysis
Like its portrayal of the per capita income benefit, MCC's portrayal of
the program's GDP impact differs from that supported by the underlying
data. In the compact and the 2006 Congressional Notification, MCC states
that the compact will have a transformational effect on Vanuatu's economy,
causing GDP to "increase by an additional 3 percent a year." Given the GDP
growth rate of about 3 percent that MCC expects in Vanuatu without the
compact, MCC's statement of a transformational effect suggests that the
GDP growth rate will rise to about 6 percent. However, MCC's underlying
data show that although Vanuatu's GDP growth rate will rise to about 6
percent in 2007, in subsequent years the GDP growth rate will revert to
roughly the rate MCC assumes would occur without the compact,
approximately 3 percent (see fig. 9). Although MCC's data show that the
compact will result in a higher level (i.e., dollar value) of GDP, the
data do not show a transformational increase to the GDP growth rate.
Figure 9: Vanuatu GDP Growth with and without MCC Compact
Notes:
According to MCC, "GDP is expected to increase by an additional 3 percent
a year as a result of the MCA program."
According to MCC data, the compact will have a small impact on GDP growth
rate in later years. In 2010 to 2015, the GDP growth rate resulting from
the compact will be 3.1 percent, compared with 3 percent without the
compact.
Portrayal of Poverty Reduction Does Not Quantify Benefits to Rural Poor
MCC's portrayal of the compact's projected impact on poverty does not
identify the proportion of the financial benefits that will accrue to the
rural poor. In the compact and the Congressional Notification, MCC states
that the program is expected to benefit "approximately 65,000 poor, rural
inhabitants living nearby and using the roads to access markets and social
services." However, in its underlying documentation, MCC specifies that 43
percent of the monetary benefits are expected to go to the local
population. The remaining 57 percent of the benefits are expected to
accrue to other beneficiaries, including expatriate tourism services
providers, transport providers, government, and local businesses (see fig.
10). Given that many of MCC's benefits flow to the expatriate-dominated
tourism sector, MCC makes an appropriate assumption that the local
population will not receive all the benefits. However, the compact,
congressional notifications, and other publicly available MCC documents do
not provide this information.
Figure 10: MCC Analysis of Distribution of Vanuatu Compact Benefits
Note: MCC defines "local population" as comprising local producers, local
consumers, and inhabitants of rural communities.
Although MCC expects that 43 percent of monetary benefits will go to the
local population, MCC does not establish the proportion of
local-population benefits that will go to the rural poor. Because MCC
defines the local population as "local producers, local consumers and
inhabitants of remote communities," the 65,000 poor, rural beneficiaries
that MCC projects may share local-population benefits with those who are
urban and are not poor.
Estimates of Compact Benefits Are Generally Reasonable, but Some Calculations
and Assumptions Are Problematic
MCC's documentation states that it used conservative assumptions in
developing its benefit estimates. Although our fieldwork and meetings in
Vanuatu generally affirmed MCC's assumptions about benefits, many
participants in our Vanuatu discussion groups noted that the positive
impacts of the MCC compact are contingent on other factors, such as the
development of tourist facilities and activities. However, our review of
MCC's analyses identified some calculation errors in its determination of
the compact's impact on per capita income and estimation of the number of
compact beneficiaries, as well as questionable assumptions in regard to
the beneficiary population. (See app. III for a detailed discussion of
these calculation errors and the effect of MCC's assumptions.)
o Calculation errors. To determine per capita income effects, MCC
incorrectly adjusted the value of a benefit stream for inflation
that was already presented in real (i.e., inflation-adjusted)
terms. In addition, MCC failed to account for population growth in
projecting per capita income effects from the compact. Although
these errors largely cancel each other, correcting them reduces
MCC's projection of the per capita income benefit slightly, from
$51 to $49 in 2010 and from $61 to $57 in 2015. Correcting
calculation errors significantly increases the number of rural
beneficiaries; MCC's estimate of compact beneficiaries in Santo
and Efate did not take into account population growth since the
1999 census.
o Assumptions. Our analysis of MCC's data for Santo and Efate
indicates that MCC's count of beneficiaries may be overestimated.
MCC's count of poor, rural beneficiaries includes all rural
inhabitants in the catchment area, indicating that MCC assumes
that all rural inhabitants are poor. However, according to ADB
reporting, rural poverty in Vanuatu is widespread but not
universal. The poverty level, defined as having an income of one
U.S. dollar per day, is 51 percent in rural areas.^18 Further, in
defining the catchment areas in Efate and Santo, MCC assumed that
residents of villages near existing paved portions of the Ring
Road not improved by MCC, as well as those on off-shore islets,
would benefit fully from the compact.
Correcting calculation errors and fully discounting MCC's
assumption including these residents in the catchment area would
reduce the beneficiary count on Efate and Santo by 32
percent--from 26,553, as stated by MCC, to 18,070--indicating that
MCC may overestimate the compact's beneficiaries.
^18In its Vanuatu Monitoring and Evaluation Plan, MCC is using the poverty
measure proposed by Vanuatu, the fraction of individuals with monthly cash
income less than 20,000 vatu (approximately $185 U.S. dollars or $6 per
day).
Several Risks May Lead to Reduced Project Benefits
We identified five key risks that may affect the Vanuatu compact's
projected impact on poverty reduction and economic growth. First,
the contingencies included in MCC's calculations of construction
costs may not be sufficient to cover average transportation
project overruns. Second, although the compact's benefits are
projected to begin shortly after completion of the projects, some
of these benefits are likely to accrue more slowly. Third, while
the benefit projections assume continued maintenance of completed
projects, MCC's ability to ensure such maintenance will end in
2011, and Vanuatu's record of road maintenance is poor. Fourth,
the projected induced benefits from expanded tourism and
agriculture depend on businesses and rural inhabitants responding
to opportunities created by improved infrastructure. Fifth,
efficiency gains, such as time saved in transit, may not result in
increased per capita income, as MCC projects. Our analysis of
these areas of risk illustrates the extent to which MCC's
projections of benefits are dependent on assumptions of immediate
realization of benefits, successful long-term maintenance,
realization of induced benefits, and benefits from efficiency
gains.^19
Construction Costs May Exceed Contingencies
Although MCC considered the risk of construction cost increases,
the contingencies used in its calculations may not be sufficient
to cover actual construction costs. We received documentation from
MCC's contractor of its MCC cost estimates for only 5 of MCC's 11
construction projects. For these five, it used a design
contingency of 20 percent.^20 In its due diligence book, MCC
states that its cost estimates include physical contingencies,
with an average value of 15 percent, and price contingencies, with
an average value of 12 percent. However, cost overruns of more
than 20 percent occur in many transportation projects.^21 For
example, a study of more than 250 transportation projects in
Europe, North America, and elsewhere found that costs for all
projects were 28 percent higher, on average, than forecasted at
the time of decision to build, while road projects averaged
escalations of 20.4 percent.^22 Further, as MCC's analysis notes,
the risk of excessive cost overruns is significant in a small
country such as Vanuatu, because (1) few comparable projects have
been undertaken in Vanuatu and, consequently, little previous cost
information is available and (2) no local contractors are capable
of undertaking a project of this size, and foreign contractors may
apply large margins to cover unknown factors.^23 Any construction
cost overrun must be made up within the Vanuatu compact budget by
reducing the scope, and therefore the benefits, of the compact
projects.^24 Reduced project benefits would in turn reduce the
compact's ERR and effects on per capita income and GDP.
^19We modeled risk scenarios with alternative methods of looking at
compact benefits that assume phasing of costs and benefits and phasing
coupled with lack of maintenance, lack of induced benefits, and lack of
monetized efficiency gains. See appendix V for more information about our
methodology.
^20For one of these five, the $400,000 South West Bay airstrip project,
the estimate also included an additional 10 percent construction
contingency.
In addition, because MCC was unable to provide the data that it
used to produce the estimates for the Santo and Efate projects,
the project costs may be at further risk from unchecked
assumptions and data (see app. IV).
^21GAO recently reported this occurring as part of U.S. Agency for
International Development (USAID)-funded road construction in a
tsunami-affected area of Indonesia. In Indonesia, construction cost per
mile increased by 75 percent, USAID reduced the length of road to be built
by more than one third and the agency may extend the planned completion
date by 5 months. GAO, Foreign Assistance: USAID Signature Tsunami
Reconstruction Efforts in Indonesia and Sri Lanka Exceed Initial Cost and
Schedule Estimates and Face Further Risks, [36]GAO-07-357 (Washington,
D.C.: February 28, 2007).
^22Bent Flyvbjerg, Mette Skamris Holm, and Soren Buhl, "Underestimating
Costs in Public Works Projects: Error or Lie?," Journal of the American
Planning Association, Vol. 68, No. 3 (2002), cited in GAO, Highway and
Transit Investments: Options for Improving Information on Projects'
Benefits and Costs and Increasing Accountability for Results,
[37]GAO-05-172 (Washington, D.C.: January 24, 2005).
^23MCC cites the "design-construct" contract proposed for the MCA program,
which will include design and construction of all the projects as one
package, as key to mitigating this risk. However, MCC's analysis also
recognized that nonconstruction-related issues (such as access to parts of
the project site) have the potential to delay the contractor and increase
costs and that such issues can be significant for major road upgrade
projects where the competing interests of the contractor, adjacent
villages, and the general public must be balanced. MCC's analysis states
that, to help manage the risk of project-related disputes and delays, MCC
plans to have experienced consultants work with local PWD staff who have
an understanding of the social and cultural issues.
^24According to the compact, the government of Vanuatu must pay any
environmental mitigation and remediation costs in excess of the budget.
Compact Benefits Are Likely to Accrue More Slowly Than Projected
Although MCC's analysis assumes compact benefits from 2008 or
2009--shortly after the end of project construction--we found that
benefits are likely to accrue more slowly. Our document review and
discussions with tourism services providers and agricultural and
timber producers suggest that these businesses will likely react
gradually to any increased market opportunities resulting from
MCC's projects, in part because of constraints to expanding
economic activity.^25 For example:
o According to tourism officials and business owners, several
types of activities need to progress to enable industry growth.
Factors needed to foster tourism growth in Vanuatu include
marketing Vanuatu as a tourist destination; promoting different
types of products to different markets; improving domestic and
international air and sea access; developing or upgrading the
electric power, water, and road infrastructures; and recruiting
and training workers. Tourism service providers in Santo and Efate
identified improving air capacity as an important need.
o According to an official from the fisheries department in Santo,
besides rough roads, the lack of ice-making equipment and lack of
feed are important barriers to developing Santo's small-scale
aquaculture.
o Timber production expands slowly; for example, a timber company
owner in Santo stated that newly planted trees could not be
harvested for about 15 years.
o In both Efate and Santo, feeder roads are in worse condition
than the project roads and are critical to agriculture
shipments.^26
These constraints suggest that future benefits related to tourism
and agriculture will be phased in once transportation
infrastructure is improved. Moreover, MCC assumes all construction
costs will be incurred in the first year, instead of phasing these
costs over the multiyear construction schedule. Our analysis shows
that if costs are phased over 3 years and benefits are phased over
5 years, overall compact ERR declines from 24.2 percent, as
projected by MCC, to 16.5 percent (see app. V).
^25Benefits from construction activities may also be reduced by a delayed
procurement. MCA-Vanuatu officials initially told us they anticipated
issuing an invitation for bid to contractors by the end of February 2007.
As of May 2007, the invitation had not yet been issued. MCC currently
expects construction to begin in 2008, further reducing the likelihood of
benefits starting in 2007 as MCC anticipated in its analyses.
^26It is unclear whether responsibility for feeder roads lies with the
government at the central or provincial level; neither takes
responsibility. The Santo PWD expects to focus on maintaining feeder roads
once the East Coast Road is done.
Project Maintenance after Compact Expiration Cannot Be Ensured
Uncertainty about the maintenance of completed transportation
infrastructure projects after 2011 may affect the compact's
projected benefits. According to World Bank and ADB officials,
continuing donor involvement is needed to ensure the maintenance
and sustainability of completed projects. In addition, during our
visits to Efate and Santo, tourism and agriculture business
representatives cited continued road maintenance as a critical
concern. However, although MCC has budgeted $6.2 million for
institutional strengthening of the Vanuatu PWD, MCC has no means
of ensuring the maintenance of completed projects after the
compact expires in 2011; the Millennium Challenge Act limits
compacts to 5 years.
Although the conditions precedent^27 to the Vanuatu compact
require the government's commitment to ongoing project
maintenance, World Bank and ADB officials told us, based on their
experience with Pacific countries, that covenants such as these
are difficult to enforce after the project is completed and often
are not effective in ensuring sustainability. According to Vanuatu
government officials, funds from vehicle registration and related
fees are available for maintenance of the completed projects;
however, the government has not dedicated these funds for this
purpose. In addition, according to donor reporting, the government
has failed to sustain maintenance of previous donor projects,
primarily because of a lack of funds and a shortage of resources,
skills, and capabilities. Some of the department's equipment on
Santo Island is more than 27 years old, and newer equipment
remains idle for long periods because spare parts are scarce;
overall, the department lacks adequate equipment for Santo's
1,000-kilometer road network. Although the compact provides funds
for equipment and for technical assistance to increase capacity,
MCC cannot ensure the maintenance of infrastructure after the
compact ends.
^27Conditions precedent are specific steps that must be completed by the
compact country prior to MCC's providing a disbursement. According to MCC
officials, the government has thus far met or exceeded the requirements of
these conditions.
Poor maintenance performance will reduce the benefits projected in
the MCC compact.^28 Our analysis shows that with phasing of costs
and benefits, no large periodic maintenance expenditures, and
inadequate maintenance performance, overall compact ERR decreases
from 24.2 percent, as projected by MCC, to 16.5 percent with
phasing and to 13.8 percent without maintenance.^29
Induced Benefits Require Tourism Providers and Agricultural
Producers to Respond to Opportunities
The compact's induced benefits depend on the response of Vanuatu
tourism providers and agricultural producers. However, constraints
affecting these economic sectors--such as the air-passenger
capacity limitations that affect Vanuatu's tourism--may prevent
the sector from expanding as MCC projects, affecting the
reliability of MCC's estimate of program benefits.
Some development assistance organizations do not include induced
benefits in calculations of ERR. In particular, World Bank
officials told us they have not considered induced benefits in
economic analyses of infrastructure for 10 years, because they
deem such benefits too conjectural and subject to manipulation.^30
For example, if a projected road is assumed to transform an area's
agricultural production, it is easy to show large benefits and
high ERR.^31
^28In technical comments on a draft of this report, the World Bank noted
that initial construction quality and subsequent traffic can affect a
road's projected benefits and, if not included in the investment analysis,
can have important implications for the actual ERR that may be equal to,
or larger than, the impact of poor maintenance. The World Bank also noted
that weak governance and political corruption can also have a large impact
on a road's sustainability.
^29This simulation results in a modest impact on ERR because it reflects
both a reduction in future benefits and a reduction in future costs for
road maintenance.
^30In technical comments on a draft of this report, the World Bank noted
that it measures induced benefits directly, comparing base traffic levels
with expected growth in demand for transport as a result of investments.
This method results in relatively straightforward and transparent
calculations. The World Bank observed that models using other, more
indirect approaches to estimating traffic, such as examining the impact of
increased agricultural production, require careful consideration and that
it has generally used such methods less often over the last 10 years
because the results are sometimes unclear and sensitive to assumptions.
Typically, it uses such methods only when a road project is expected to
have significant restructuring effects--such as a completely new road or
bridge in an urban environment--that could drastically change land use
patterns.
Limited response to the compact by tourism providers and
agricultural producers would have a significant impact on compact
benefits. For example, our analysis shows that the phasing of
costs and benefits reduces the overall compact ERR from 24.2
percent to 16.5 percent, and the omission of induced benefits
(i.e., no producer response) further reduces the compact ERR to
5.5 percent. Two projects would have a negative ERR.
Efficiency Gains May Not Cause Measurable Change in Per Capita Income
MCC counts efficiency gains--such as time saved because of better
roads--as compact benefits. However, although efficiency gains
could improve social welfare, they may not lead to changes in per
capita income or GDP or be directly measurable as net additions to
the economy.
Excluding efficiency gains from ERR calculations reduces the
compact's overall ERR. For example, our analysis shows that the
phasing of costs and benefits and the omission of efficiency gains
from the Vanuatu ERR calculation causes the overall compact ERR to
decline from 24.2 percent to 16.5 percent with phasing and to 11.8
percent without calculating efficiency gains. This analysis also
results in a negative ERR for two projects (see app. V).^32
Conclusions
MCC obtained the input of knowledgeable stakeholders in projecting
the benefits, costs, and number of beneficiaries of its compact
with Vanuatu, which addresses one of Vanuatu's primary constraints
to economic growth. However, MCC's public statements about these
benefits--particularly its projection of the compact's effect on
per capita income--suggest greater impacts than MCC's underlying
data and analysis support. MCC's statements can be understood only
by reviewing supporting source documents and spreadsheets, which
are not publicly available. These gaps could lead to unrealistic
expectations about the compact's effect within Vanuatu; for
example, by suggesting that per capita incomes will increase so
quickly, MCC suggests that its compact will achieve sustainable
growth in a way that other donors to Vanuatu have not been able to
achieve. Further, these gaps between MCC's statements and
underlying analysis raise questions about other compacts'
projections of transformative impact on country economies or
economic sectors. Without an accurate representation of the
compacts' projected benefits, the extent to which the compacts
further MCC's goals of poverty reduction, economic growth, and
transformative development cannot be accurately evaluated.
Further, MCC's economic analyses for the Vanuatu compact did not
fully consider the phasing of costs and benefits, such as the time
required to improve the infrastructure and for the economy to
respond to the opportunities from the improved infrastructure.
Additionally, MCC's analysis did not fully account for risks that
could substantially reduce compact benefits.
^31However, the lack of consideration of induced benefits is not without
its flaws. World Bank officials told us that in the fast-growing economy
of Vietnam, the World Bank funded a road on the basis of current use of
traffic without projecting the gains to the economy. The road is now very
congested, and the project was underinvested. According to the World Bank,
consideration of projected traffic increases--that is, induced
traffic--are taken into account only when the project is not justified on
the basis of existing traffic alone. However, rapid traffic growth is
considered exceptional.
^32The ERR in this case does not take into account opportunity costs and
is therefore an internal rate of return.
Recommendations
We recommend that the CEO of MCC take the following actions:
o revise the public reporting of the Vanuatu compact's projected
impact to clearly represent the underlying data and analysis;
o assess whether similar statements in other compacts accurately
reflect the underlying data and analysis; and
o improve economic analysis by phasing the costs and benefits in
compact ERR calculations and by more fully accounting for risks
such as those related to continuing maintenance, induced benefits,
and monetized efficiency gains as part of sensitivity analysis.
Agency Comments and Our Evaluation
MCC provided written comments regarding a draft of this report. We
have reprinted MCC's comments, with our responses, in appendix VI
and incorporated technical comments from MCC where appropriate.
Although MCC did not directly acknowledge our recommendations, it
questioned our finding of a gap between its portrayal of the
compact's benefits and its underlying analysis. In addition, MCC
responded to our discussion of beneficiary numbers and program
risks. Following is a summary of MCC's comments and our evaluation
of these comments.
Compact Impact on Incomes and Growth
MCC acknowledged that its use of projected cumulative compact
impact on income and growth was misleading, but it asserted that
(1) it had no intention to mislead and (2) its portrayal of
projected compact benefits was factually correct and consistent
with its underlying data. Our report does not state or imply that
MCC intended to mislead; we have not determined why MCC made its
statements portraying projected benefits on income and growth.
Further, MCC's portrayal of the projected benefits could be
considered factually correct and consistent with the underlying
data only if the reader knows that the portrayal represents
cumulative income and growth over 5 years. However, MCC's public
documents and communications to Congress never describe the
benefits it projects as being cumulative over 5 years. At issue in
this finding are both the facts represented by MCC's data and
calculations and MCC's representation of these data and
calculations in its public statements. As our report notes, MCC's
statements can be understood accurately only by reviewing
supporting source documents and spreadsheets, which are not
publicly available.
In addition, MCC's portrayal of the compact's effect in cumulative
terms is not consistent with its description of per capita income
in Vanuatu in annex I of the compact, which states that per capita
income in Vanuatu declined by 15.4 percent between 1994 and 2003.
This calculation is consistent with the World Bank's data, which
show Vanuatu's per capita income in 2003 as approximately 15.4
percent less than its per capita income in 1994--a comparison of
2003 and 1994 income levels, not a reflection of cumulative
decline. If the methodology MCC uses to describe past per capita
income in Vanuatu is used to describe the compact's projected
future impact, then the projected increase in Vanuatu's per capita
income in 2010 should be portrayed as 3.9 percent, not the
cumulative 15 percent that MCC presents in its congressional
notifications and public documents.
MCC also commented that its data on the compact's effect on
Vanuatu's GDP are consistent with its assessment that GDP will be
perpetually 3 percent higher with the MCC investment than without
it. We agree that MCC's data reflect this, but MCC's portrayal of
this benefit in its public statements implies an effect on the
growth rate, rather than the level, of Vanuatu's GDP. MCC couples
its portrayal of the GDP benefit with a description of the
compact's impact as "transformational"--that is, as MCC defines
the term, having "a dramatic and long-lasting impact on poverty
reduction through sustainable economic growth." However, although
MCC's data show projected benefits to Vanuatu's economy, including
an increase in the level of GDP, the projected change to the
overall Vanuatu economy, as reflected by the GDP growth rate, is
smaller than 3 percent. In 2010-2015, for example, the GDP growth
rate increase resulting from the compact will be 0.1 percent.
Compact Beneficiaries
MCC commented that our report suggests that (1) nonpoor households
could feasibly be excluded from the count of beneficiaries, (2)
persons living on off-shore islets and in the hinterland should
not be counted as beneficiaries, and (3) MCC's definition of the
local beneficiary population includes urban dwellers. Regarding
nonpoor households, we do not suggest that they could be excluded
or that this would be desirable. Our report indicates that MCC's
analysis shows there will be poor, rural beneficiaries as well as
other beneficiaries and that MCC has not quantified what portion
of the compact's benefits poor, rural beneficiaries will receive.
Regarding off-shore and hinterland beneficiaries, we do question
whether inhabitants of off-shore islets and of villages more than
10 kilometers from MCC projects on Santo and Efate would benefit
fully from MCC's projects. However, these persons are less than
half of the total number we question. In Santo and Efate, 51
percent of the catchment area population are not inhabitants of
the hinterland or off-shore islets, but rather residents of Efate,
near the tourism center and capital of Port Vila and along a
portion of the Ring Road that is already paved. It is not clear
how construction of a road elsewhere on the island will benefit
these residents when they already have access to Port Vila via the
existing paved road. Regarding the definition of "local
population"--which MCC's underlying documentation says will
receive 43 percent of compact benefits--MCC's written comments
asserted that its definition of this population does not include
urban dwellers. Although we asked MCC to support this assertion,
MCC did not provide any additional documents. In separate
comments, MCC added that although its documentation does not
contain a very detailed reference specifically saying exactly how
many business people, transport providers and tourism operators
are foreigners and urban, MCC believes it reasonable to conclude
that urban dwellers are included in these categories. However, we
note that not every urban dweller is a business person, transport
provider, or tourism operator and that some urban beneficiaries
would therefore fall into the "local population" that receives an
estimated 43 percent of benefits. Further, MCC's definition of the
term in its internal documents includes persons who are not poor
and not rural.
Program Risks
MCC commented that it undertakes sensitivity analysis for all of
its ERR calculations, although it may focus on risks other than
those we identified. In addition, MCC states that it differs
significantly from us regarding the precise nature and severity of
program risks in general. As our report notes, we offer several
alternative scenarios to illustrate the maximum impact of certain
areas of risk on projected compact benefits. The scenarios look at
the effect of individual risk areas, rather than the possibility
of projects' failing to achieve certain benefits in multiple areas
simultaneously. Regarding induced benefits and efficiency gains,
we note that MCC's documentation of its economic analysis
questions the valuing of efficiency gains as induced benefits to
the Vanuatu economy. Regarding maintenance, MCC states that its
compact provides a significant amount of money to assist the
Vanuatu PWD in providing timely and adequate maintenance. However,
as our report notes, Vanuatu's record of maintenance is poor, and
achieving a change in maintenance performance through donor
assistance would show a significant break from past experience.
Finally, regarding contingencies, MCC questions our use of the
Flyvbjerg, Holm, and Buhl study as a basis for evaluating its use
of contingencies and states that it provided us with price and
contingency information to support its use of contingencies. The
Flyvbjerg, Holm, and Buhl study and our report use the same basis
of comparison for evaluating the adequacy of contingencies, and we
have updated the language of our report to make this clear. In
addition, as our report notes, we did not receive full
documentation of the cost estimates and contingencies that MCC
used in developing its economic analyses. We received no
supporting documentation for the cost and contingency estimates
for the Santo and Efate projects, which account for 56 percent of
the total compact budget. A key document that MCC's contractor
said it used as a basis for developing the Santo and Efate costs
was unavailable to us, because the government of Vanuatu would not
provide it to MCC. Cost overruns remain a risk to project
benefits, and we have presented them as such.
As agreed with your offices, unless you publicly announce the
contents of this report earlier, we plan no further distribution
until 30 days from the report date. At that time, we will send
copies of this report to interested congressional committees as
well as to the CEO of MCC and the Secretary of the Treasury. We
will make copies available to others on request. In addition, this
report will be available at no charge on the GAO Web site at
http://www.gao.gov.
If you or your staff have any questions about this report, please
contact David Gootnick at (202) 512-3149 or [email protected].
Contact points for our Offices of Congressional Relations and
Public Affairs may be found on the last page of this report. GAO
staff who made major contributions to this report are listed in
appendix VII.
Sincerely yours,
David Gootnick
Director
International Affairs and Trade
Appendix I: Objectives, Scope, and Methodology
At the request of the Chairman of the House Committee on Foreign
Affairs, we examined the Millennium Challenge Corporation's (MCC)
economic analyses of the Vanuatu compact. Specifically, we
examined
o MCC's methods of projecting the compact's economic benefits and
methods of calculating the benefits,
o MCC's portrayal and analysis of the projected benefits, and
o risks that could affect the compact's impact on poverty
reduction and economic growth.
To accomplish our objectives, we reviewed MCC's record of due
diligence, including its opportunity memo, investment memo, due
diligence book, economic analysis spreadsheets, and contractor
reports. These documents are restricted from public dissemination
based on MCC policy, but MCC made them available to us for
analysis. We also reviewed the supporting documents used in MCC's
due diligence process, such as previous donor reports and studies.
We supplemented this review with interviews with MCC officials in
Washington, D.C., and MCA-Vanuatu officials in Vanuatu. We then
used this information to (1) summarize the process that MCC used
to develop its economic model and (2) identify the logic, data,
methods, and assumptions used to determine the compact's projected
costs and benefits, number of beneficiaries, and effects on per
capita income, gross domestic product (GDP), and poverty.
We evaluated MCC's statements of compact impact in its public
documents such as the congressional notification and the compact
itself--as well as MCC's internal investment memo--by comparing
these statements with the underlying analyses and data used to
support them. We analyzed the November 2005 investment memo
spreadsheet containing the source data and calculations used for
MCC's statements about per capita income and GDP and reviewed its
contents and formulas to determine the methodology and source data
for MCC's statements and to assess the accuracy of the
calculations. We then used MCC's own data and formulas from the
November 2005 investment memo to determine MCC's impact in
individual years and compare the results with the statements made
in the Vanuatu compact and congressional notification. We could
not validate most of MCC's underlying data and assumptions,
because some data, such as construction and maintenance cost data,
were not available or could not be checked within the time frames
of our engagement. Further, we used the calculations and formulas
from MCC's November 2005 investment memo, corrected population
figures, and the data from MCC's April 2006 final economic
analysis to recalculate MCC's statements about per capita income
impacts. We reviewed MCC's economic analysis spreadsheet to find
the amount of benefits that MCC's analysis assumed would flow to
poor, rural beneficiaries and reviewed spreadsheets provided by
MCA-Vanuatu to determine the assumptions, methodology and data
used to calculate the beneficiary population in Santo and Efate.
We compared the spreadsheet data with Vanuatu village maps and
population figures from the 1999 National Population and Housing
Census provided by MCA-Vanuatu to evaluate MCC's calculations and
assumptions about the villages included as beneficiaries.
We identified risks to MCC's compact results based on our review
of MCC's internal documentation and donor and academic literature.
We also met with Vanuatu, MCC, and contractor officials and
interested parties such as tourism and agriculture business owners
in Vanuatu. We focused our field work on the three projects
representing more than half of the compact budget and on Vanuatu's
two most populous islands, Santo and Efate. We examined MCC's due
diligence documentation and contacted MCC's contractor to
determine the methodology used to estimate construction costs. We
then compared MCC's procedures to findings and GAO's best practice
criteria for cost estimation. We compared MCC's assumptions about
maintenance continuing for 20 years to previous donor experiences
gleaned from donor reporting or interviews with the Asian
Development Bank, Australian Agency for International Development,
European Union, International Monetary Fund, New Zealand Agency
for International Development, and World Bank. We determined the
need for the phasing of benefits based on discussions with
knowledgeable agriculture and tourism representatives in Vanuatu
and a review of MCC's source documentation. We further compared
MCC's treatment of induced benefits to the approach taken by the
World Bank and MCC's treatment of direct benefits to economic
principles about the monetization of efficiency gains. We modeled
risk scenarios that provide alternative methods of looking at
compact benefits using MCC's April 2006 economic model
spreadsheet. In modeling these risks, we used the data from MCC's
economic analyses; we did not validate these data. In these
scenarios, we determined the effect of the scenario on the
economic rate of return by assuming the phasing in of costs over 3
years and of benefits over 5 years, first determining the effect
of phasing alone, then examining the phasing of costs and benefits
coupled with the following:
o Lack of maintenance--eliminating all future large maintenance
expenditures and taking away a fraction of the growing expected
benefits each year, so that at the end of the 20-year period,
2027, benefits basically return to their 2012 level.
o Lack of induced benefits--deleting those benefits from the cost
model.
o Lack of monetized efficiency gains--deleting those benefits from
the cost model.
Appendix II: Prior Development Assistance to Vanuatu
Of Pacific island countries, Vanuatu receives among the largest
amounts of aid; however, Vanuatu's economic performance and social
development since 1990 have lagged behind other countries in the
region. Vanuatu depends on donor assistance for its development
budget; from its gaining independence, in 1980, to 2002, aid
receipts averaged around 17 percent of GDP. In 1997, Vanuatu
initiated structural adjustment efforts with ADB assistance
designed to stimulate the private sector and help realize the
country's growth potential in agriculture and tourism. However,
these reforms have had a limited impact on growth in
private-sector investment and GDP. In 2003, the government of
Vanuatu prepared the Priorities and Action Agenda to focus
available resources, including donor assistance, on Vanuatu's
development priorities. The current strategies of Vanuatu's major
donors--which include Australia, France, New Zealand, the European
Union, Japan, and MCC--are aimed at strengthening Vanuatu's
productive sectors and supporting private sector-led development
and are based on the Vanuatu government's stated goals. MCC
anticipates that its average annual assistance level of $13.1
million over the next 5 years will position the United States as
one of Vanuatu's top two official donors (see table 1).
Table 1: Average Annual Contributions of Top Five Donors to
Vanuatu in 2004 and 2005
(Dollars in millions)
Donor Average annual contribution
Australia $19.1
France $5.3
New Zealand $4.8
European Union $4.6
Japan $3.5
Source: Organization for Economic Cooperation and Development.
According to the ADB, Vanuatu's development budget has
concentrated on building new capital infrastructure; however,
according to donor reporting, these assets have not been
adequately maintained. From 1996 to 2004, Vanuatu received about
$35 million in assistance for transportation infrastructure
projects including the construction and rehabilitation of roads,
bridges, and wharfs, and the extension and upgrading of airport
terminals and runways.
According to studies by the World Bank and ADB, Vanuatu faces
significant development challenges. Some of Vanuatu's challenges
are shared by other small states and island economies,^1 while
others are either the result of, or exacerbated by, political
instability and, according to ADB, inappropriate policies or
regulations. Like other small island economies, Vanuatu's
agriculture and tourism sectors are sensitive to extreme weather
conditions and natural disasters. For example, frequent cyclones
cause production shocks and restrict the range of viable crops and
tree species. Vanuatu's narrow production base also leaves the
economy open to economic shocks from changes in commodity prices,
especially for copra. Despite these constraints, Vanuatu has a
wealth of natural resources and growth opportunities in
agriculture, fisheries, and tourism. However, growth has been
hindered by substantial barriers to private-sector development,
including political uncertainty,^2 high costs of doing business,
poor and costly infrastructure, lack of a secured transactions
framework,^3 and issues with land tenure.
^1According to the World Bank, some of the characteristics shared by small
states such as Vanuatu include remoteness and insularity, susceptibility
to natural disasters, limited institutional capacity, limited
diversification, reliance on external trade, and limited access to
external capital. Commonwealth Secretariat/World Bank, Small States:
Meeting Challenges in the Global Economy (Washington, D.C.: 2000).
^2Beginning in the early 1990s, Vanuatu has witnessed a series of
short-lived coalition governments, comprising numerous parties and
involving frequent changes of ministers. There have been eleven
governments in power since 1990, with most lasting only about 1 year. This
constant turnover has delayed reforms as coordination of policies and
obtaining consensus for needed reforms has proved difficult. International
Monetary Fund, Vanuatu: Selected Issues, IMF Country Report No. 07/93
(Washington, D.C.: 2007).
^3The legal framework for secured transactions is incomplete. According to
the ADB, the legal system does not allow for property to be used as
collateral for loans, particularly outside of Port Vila and Luganville.
Thus borrowers can neither purchase property on credit nor obtain loans
against their own assets. This accounts for high borrowing costs, higher
interest rates, smaller loans offered, and loans of shorter maturity,
which limit the private sector's access to credit and entrepreneurship.
IMF, Vanuatu: Selected Issues.
Appendix III: MCC Estimates of Per Capita Income Impact
and Poor, Rural Beneficiaries
Per Capita Income Impact
In its economic analysis for Vanuatu, MCC did not correctly
account for population growth in projecting per capita income and
incorrectly deflated the value of the benefit stream that was
already in real terms. When the benefit stream is no longer
deflated, its value increases. However, when population growth is
accounted for, the per capita income benefit is reduced. MCC
assumes that the population is growing by approximately 2.6
percent each year, but MCC's analysis of cumulative income per
capita benefits for 2006 to 2015 is based on 2005 population
estimates, rather than on an estimate of population that increases
each year. Figure 11 contrasts the estimated increase in income
per capita over the 2005 baseline in the given year with and
without our corrections to deflating of benefits and to population
growth. The errors largely cancel out, reducing the per capita
income benefit projected by MCC from $51 (3.9 percent) to $49 (3.7
percent) in 2010 and from $61 (4.6 percent) to $57 (4.3 percent)
in 2015.
Figure 11: MCC Compact's Projected Impact on Vanuatu Per Capita
Income with and without Adjusted Population Estimates
Note: In addition to adjusting for population growth and
eliminating the deflating of benefits already in real terms, our
revised projections of per capita income benefits use April 2006
data from MCC, including a benefit start date of 2007 for
construction spending, a benefit start date of 2008 for some
projects, and updated numeric values for costs and benefits.
The impact of the MCC compact on per capita income is a relatively
small addition to Vanuatu's expected per capita income without
MCC. Figure 12 shows, in dollar terms, MCC's projected impact on
Vanuatu's per capita income compared with the expected levels of
per capita income, using MCC's assumptions of 3 percent annual
growth in GDP and 2.6 percent in population.
Figure 12: MCC Compact's Projected Impact on Vanuatu Per Capita
Income in Real Terms Relative to Per Capita Income without MCC
Note: Assumes 0.4 percent growth in per capita income, based on
MCC's assumed GDP growth of 3 percent without the compact and
MCC's assumed Vanuatu population growth rate of 2.6 percent.
Calculation of Beneficiaries
MCC calculated approximately 65,000 beneficiaries on the eight
islands receiving MCC projects (see table 2).^1 However, MCC's
determination of beneficiary numbers in Santo and Efate contained
calculation errors and relied on problematic assumptions that
villages near portions of the road not improved by MCC, as well as
those on off-shore islets, would fully benefit from the compact.
Correcting the calculation and data errors and fully discounting
MCC's assumptions would reduce the beneficiary count on Efate and
Santo by 32 percent, from MCC's stated 26,553 to 18,070.
^1Before signing the compact, MCC determined the number of potential
beneficiaries but did not quantify targets for the compact's effect on
poverty. MCC has funded an additional survey that will provide updated
baselines and the basis for establishing these targets.
Table 2: Number of Poor, Rural Beneficiaries of Vanuatu Compact,
as Calculated by MCC
Island Project Number of beneficiaries
Efate Round Island Road 13,819
Santo Port Olry Road 7,404
Santo South Coast Road Bridges 5,330
Subtotal, Santo and 26,553
Efate
Other projects
Tanna Whitesands Road 6,190
Malekula Lits Lits Road 5,676
Malekula Southwest Bay Airstrip 4,570
Pentecost Loltong Wharf and North-South 7,858
Road
Epi Lamen Bay Wharf 1,560
Malo Malo Roads Upgrading 3,480
Ambae Ambae Roads Reconstruction 9,340
Total 65,227
Source: GAO analysis of MCC data.
Our examination of the calculation of MCC's beneficiary estimate
for the Efate Ring Road, Santo East Coast Road, and Santo South
Coast Road Bridges shows that MCC's estimate of rural
beneficiaries in Santo and Efate would be increased by
approximately 5,100 persons, or 19 percent, if adjustments were
made to correct data errors^2 (see table 3).
^2We identified the following errors: (1) MCC's beneficiary estimates use
data from the 1999 census and have not been updated to 2005. This results
in an underestimate of the beneficiary population. (2) In the data
provided to us, the total of the populations of villages included in the
MCC analysis does not match the beneficiary count in the monitoring and
evaluation plan. (3) For the Efate Ring Road project, MCC's data
double-counted some villages. (4) For all three projects, the 1999
population figures used for some villages do not match the populations
shown in the 1999 Vanuatu census. For the Santo East Coast Road, the
population figures do not match for 53 of 56 villages in the catchment
area.
Table 3: Effect of Calculation and Data Errors on Beneficiary Population
in Efate and Santo
Santo South
Efate Ring Santo East Coast Road
Road Coast Road Bridges Total
MCC Statement: beneficiary count
in monitoring and evaluation plan 13,819 7,404 5,330 26,553
GAO recalculation of MCC data^a 15,439 7,518 8,732 31,689
Difference 1,620 114 3,402 5,136
Source: GAO analysis of MCC data.
aGAO's recalculation uses 1999 Vanuatu census data, removes double-counted
villages, and updates the population to 2005 using MCC's assumed
population growth factor in rural areas of 2 percent per year.
MCC also made problematic assumptions about the geographic reach of
project benefits that increased the number of beneficiaries by including
more rural areas as benefiting from the project. Fully discounting these
assumptions would reduce the beneficiary count from our adjusted total of
31,689 to 18,070 (see table 4).
Table 4: Effect of MCC Assumptions on Beneficiary Population in Efate and
Santo
Santo South
Efate Ring Santo East Coast Road
Road Coast Road Bridges Total
GAO calculation using MCC data 15,439 7,518 8,732 31,689
Discounting population of areas (6,889) 0 (3,222) (10,111)
not adjacent to MCC road
project
Discounting population of (2,779) (196) (533) (3,508)
off-shore islets
Total^a 5,770 7,322 4,978 18,070
Source: GAO analysis of MCC data.
aBecause of rounding, numbers in columns may not sum to totals.
In estimating beneficiaries of both the Efate Ring Road and the Santo
South Coast Road Bridges projects, MCC included the populations of
villages not located near MCC's road projects and populations that face
geographic barriers to using the projects. These population counts were
not prorated or adjusted to account for the barriers that some villages
face in realizing benefits from MCC's projects.
o In Efate, MCC assumed that the populations of villages located
along an already improved portion of the road adjacent to Port
Vila in the Mele, Eratap, and Erakor areas--which will not be
addressed by the MCC compact--would benefit from construction of
the road elsewhere on the island (see fig. 13). MCA-Vanuatu stated
that the populations of these villages were included because they
own land where the road will be constructed and most of the
villagers' gardens are located in these areas. (The source for
this assumption is not documented.) The inhabitants of these
villages represent 45 percent of the total number (15,439) of
poor, rural Efate beneficiaries. MCC also did not account for
"most" villagers and instead included the entire village
population in these areas in its totals.
Figure 13: MCC Efate Ring Road Catchment Area
o For the Santo South Coast Road Bridges, MCC included villages in
West Santo more than 10 kilometers from the bridge projects and
villages across a river that vehicles can cross only by fording;
after heavy rains, the river becomes impassable. MCC's compact
program does not include a bridge across this river. Using a 2005
population estimate, we determined that MCC included 3,222 people
in the count of beneficiaries of South Coast Road Bridges based on
this assumption--37 percent of the total number (8,732) of South
Coast Road Bridges beneficiaries.
For all three projects, MCC assumed that inhabitants of off-shore
islets would fully benefit, although they are not connected to the
improved roads and must travel by boat to reach them. MCC
officials told us that they expect that residents of off-shore
islets will benefit from increased tourism spending after
construction of roads and would benefit from decreased
transportation costs after coming to the mainland. Using 2005
population estimates, we determined that 2,779 persons (18 percent
of the Efate Ring Road beneficiary population) live on off-shore
islets. For the Santo East Coast Road and South Coast Road
Bridges, respectively, 196 (3 percent of the beneficiary
population) and 533 (6 percent of the beneficiary population) live
on off-shore islets.
Appendix IV: MCC Cost Estimates for the Vanuatu Compact
The reliability of MCC's cost estimates for the Vanuatu compact
construction cannot be evaluated, because MCC was unable to fully
provide the data that it used to produce them. MCC's contractor
based its construction cost estimates for MCC's transportation
infrastructure activities primarily on two sources. To build cost
estimates for seven of the compact projects, representing 15
percent of total compact cost, the contractor used a 2003 analysis
it had performed for the ADB. To prepare cost estimates for the
three projects on Santo and Efate, representing 56 percent of
compact cost, the contractor used estimates prepared by a second
contractor for the government of Vanuatu.^1
o For the seven ADB projects, MCC's file of documents used in its
due diligence included the original cost estimate but did not
include an analysis that showed how these costs were updated to
account for inflation and changes in the scope of the work. MCC's
documentation states that these previous costs were multiplied by
1.17 to account for inflation and also adjusted for changes in the
scope.^2
o For the three projects on Santo and Efate, MCC's file did not
include any source information or analysis. MCC's contractor later
told us that it derived the costs of the projects on Santo and
Efate from an existing report by a second contractor to the
government of Vanuatu, dated approximately 2004. MCC's contractor
told us that because MCC's scope of work was more realistic than
that proposed in 2004, which included realigning portions of the
roadways, the MCC cost estimate was prepared anew using source
information from the previous report. We requested, but did not
receive, a copy of this report to the government of Vanuatu and a
copy of the analysis showing how MCC developed its cost estimates
using this source data. According to MCC officials, the government
of Vanuatu was not willing to provide the report to MCC for our
review.
According to previous GAO work, high-quality, reliable cost
estimates should be accurate, comprehensive, well-documented, and
validated.^3 MCC cost estimates used in the economic analyses have
shortcomings in each of these areas.
^1The remaining 29 percent of costs are for warehouses, and for oversight
and administrative functions.
^2MCC's contractor provided us with spreadsheet calculations of the costs
for five of the compact projects for MCC. The sheets included estimated
quantities and unit costs but did not indicate how these quantities and
costs were derived.
^3We developed these criteria as part of a forthcoming publication on best
practices in cost estimation and implemented the criteria in a recent
report. See GAO, Telecommunications: GSA Has Accumulated Adequate Funding
for Transition to New Contracts but Needs Cost Estimation Policy,
[38]GAO-07-268 (Washington, D.C.: February 23, 2007).
o Accurate. The source data for the estimates is from 2003 and
approximately 2004, and changes in the cost environment beyond
general inflation may not be accounted for. MCC also incorrectly
calculated the length of the Santo East Coast Road as 70
kilometers in its economic analyses, instead of the 55 kilometers
found by a PWD survey, according to a PWD official. Other elements
of this criterion cannot be independently assessed without the
source data for the cost estimates used in the model.
o Comprehensive. We could not assess the cost estimates against
this criterion without the source data for the estimates used in
the model.
o Well-documented. MCC did not fully document any cost estimation
procedures and checks that it performed as part of its due
diligence. The due diligence documents we received from MCC
contain a narrative describing the cost estimate procedures, but
no documentation showing the analysis that MCC's contractor
performed to update previous cost estimates.
o Validated. MCC's due diligence documentation contains no
evidence that it obtained an independent cost estimate to verify
the work of its contractor in updating its own estimate for the
ADB projects and reworking the estimates of another contractor for
the Efate and Santo projects. An independent cost estimate
provides the estimator with an unbiased test of the reasonableness
of the estimate and reduces the cost risk associated with the
project by demonstrating that alternate methods generate similar
results.
Appendix V: Illustrative Alternative Calculations of Vanuatu
Compact Impact
This appendix describes the results from several alternative
scenarios, which disaggregate the effects of the various types of
benefits on the overall ERR, GDP and per capita income in order to
illustrate the significance of each of the components.
MCC reported different values of summary statistics in the
investment memo dated November 2005 as compared with the final
analysis that reported more recent data (see table 5).
Table 5: Summary of Compact Impacts as Presented by MCC, with GAO
Recalculations
Increase in level
of GDP with MCC in Increase in level
year 5 vs. GDP of real income per
without MCC in year capita in 2010 vs.
Compact ERR 5 2005 baseline
MCC's anticipated 24.7 percent 3.2 percent 3.9 percent^a
effect as presented
in investment memo
MCC's anticipated 24.2 percent^b 3.2 percent 3.7 percent^c
effect using
corrected or updated
data
Source: GAO analysis of MCC data.
aDerived from MCC's own data and calculations.
bIn its updated and final April 2006 economic analysis, MCC adjusted the
24.7 percent reported in its congressional notification downward slightly
to 24.2 percent.
cGAO calculation after correcting MCC calculation errors.
To illustrate the maximum impact of certain areas of risk on projected
compact benefits, we modified the model using MCC data for different
scenarios. We phased costs and benefits to mirror more accurately the rate
of construction and accrual of potential benefits. Phasing costs has the
effect of increasing ERR, while phasing benefits decreases it; the net
effect is to decrease ERR. Within this modified framework that includes
phasing, we eliminated each main type of benefit--direct and induced--one
at a time to understand their relative importance.
o For "no induced benefits," we eliminated all induced benefits
from expanded tourism, agriculture, or land development as a
result of the compact but did not change the direct benefits.
o For "efficiency gains are not monetized," we eliminated all
direct benefits such as cost savings for existing users of a road
or benefits from newly generated traffic, which are attributable
to the project, but did not change the induced benefits.
Finally, we examined the impact of poor maintenance performance
after compact expiration in 2011 on total benefits from the phased
project. We modeled this by eliminating all future large
maintenance expenditures and taking away a fraction of the growing
expected benefits each year, so that at the end of the 20-year
period, 2027, benefits basically return to their 2012 level.
These scenarios present illustrative cases and should be
considered as extremes of a possible range of cases that capture
uncertainty effects on summary statistics, such as the ERR, GDP,
and income per capita. MCC has observed that overall benefits may
understate the true impact of the compact because spillover
effects, such as easier access to schools and medical facilities
due to the improved infrastructure, are not captured by these
measures. These effects represent long-term benefits to society as
whole, which are not accounted for by the analysis performed by
either MCC or GAO.
Table 6: Summary of Compact ERR, GDP, and Per Capita Income impacts under
Alternative Scenarios of Accounting for Risks to Benefits
Increase in GDP Increase in real
with MCC in year 5 income per capita
relative to GDP level in year 5
without MCC in relative to 2005
GAO analysis^a Compact ERR year 5 baseline
(1) Costs are phased 16.5 percent 1.1 percent 1.4 percent
over 3 years and
benefits are phased
over 5 years
Costs are phased over 3 years and benefits are phased
over 5 years, and
(2) induced benefits 5.5 percent 0.7 percent 0.8 percent
are not realized^b
(3) efficiency gains 11.8 percent 0.8 percent 1.0 percent
are not monetized^c
(4) large-scale 13.8 percent 1.1 percent 1.3 percent
maintenance is not
undertaken^d
Source: GAO analysis of MCC data.
aIn our analysis, benefits start in 2010 and are phased in equal
increments over 5 years, from 2010 to 2014, with phasing completed by year
5. Costs are phased over 3 years to reflect projected timing of
construction.
bIn addition to phasing benefits and costs, we eliminated induced effects
of the project on agriculture, tourism, fisheries, and the development of
subdivided beachfront land.
cIn addition to phasing benefits and costs, we eliminated road user cost
savings and savings from wasted surface trips, lost trips, longer
diversions, and enforced longer trips from road closures.
dIn addition to phasing benefits and costs, we assumed that total benefits
will increase, peak, and decrease such that their value in 2027 will equal
their original value in 2012. The large capital outlays for road
rehabilitation in 2017 and 2026 in Santo and Efate have been eliminated.
Table 7 reports the ERR for individual projects as modified in our
scenarios.
Table 7: Summary of Selected Compact Project Impacts under Alternative
Scenarios Accounting for Risks to Benefits
Costs are phased over 3 years,
benefits are phased over 5 years,
and
Costs are
phased over 3
years and Induced Efficiency Large-scale
MCC's benefits are benefits gains are maintenance
anticipated phased over 5 are not not is not
Projects ERR years realized monetized undertaken
Overall 24.2 percent 16.5 percent 5.5 11.8 13.8 percent
compact percent percent
Efate: Ring 20.6 percent 13.6 percent negative 10.8 11.2 percent
Road percent
Santo: Port 33.8 percent 22.6 percent 10.3 15.5 19.4 percent
Olry Road percent percent
Santo: South 24.3 percent 14.3 percent -4.0 12.4 10.8 percent
Coast percent percent
Bridges
Tanna: 17.7 percent 12.1percent 8.6 -0.3 8.4 percent
Whitesands percent percent
Road
Malekula: 22.5 percent 15.1 percent 15.1 negative 14.3 percent
Lits Lits percent
Road
Malekula: 11.4 percent 6.3 percent 1.1 6.1 percent 2.4 percent
South West percent
Bay Airstrip
Pentecost: 15.6 percent 8.9 percent 5.6 4.4 percent 4.7 percent
Loltong percent
Wharf/N-S
Road
Source: GAO analysis of MCC data.
Appendix VI: Comments from the Millennium Challenge Corporation
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
See comment 6.
See comment 5.
See comment 4.
See comment 3.
See comment 2.
See comment 1.
See comment 11.
See comment 10.
See comment 9.
See comment 8.
See comment 7.
See comment 16.
See comment 15.
See comment 14.
See comment 13.
See comment 12.
See comment 18.
See comment 17.
The following are GAO's comments on the Millennium Challenge Corporation
letter dated June 8, 2007.
GAO Comments
1. MCC quotes from its statement in the compact and our statement
in the text of the draft report. However, the figure we cite for
per capita income in 2015--37 percent--originates in MCC's
investment memo, not in our analysis.
2. MCC notes that it emphasized repeatedly that its key statements
in question were factually correct and that it did not intend to
overstate benefits. At issue in this finding are both the facts
represented by MCC's data and calculations and MCC's
representation of these data and calculations in its public
statements. As our report notes, MCC's portrayal of the compact's
projected impact on income and economic growth can be understood
as cumulative only by reviewing supporting source documents and
spreadsheets, which are not publicly available. Our report does
not assert or imply that MCC's portrayal was intentionally
misleading; we did not determine through our audit, or address in
the report, the reasons for the gap between MCC's portrayal and
its underlying analysis.
3. MCC comments that our assertion about its portrayal of income
benefits is itself likely to mislead. In response to this comment,
and as part of our final review process, we clarified the terms
and presentation of our analysis of MCC's statements and data and
made modifications as appropriate.
4. MCC asserts that it "stated that per capita income would
increase by a cumulative amount of 15.4% by 2010 relative to the
2005 baseline" and that this is how it understood the underlying
data in 2005. However, MCC documents in 2005 and later do not
state that the portrayed effect on per capita income is
cumulative. These documents include MCC's November 2005 Investment
Memo; the March 2006 Vanuatu compact; the January 4 and March 7,
2006, congressional notifications; MCC press release; and the
April 2007 MCC Annual Report for 2006. We asked MCC to provide any
publicly available document that discloses that the per capita
income effect is cumulative; however, MCC did not provide any such
document.
5. MCC comments that its "underlying data do not support the
conclusion that cumulative growth over 5 years would be only
3.9%." Our report does not make any assertions about the
cumulative growth rate in income per capita. As stated in figure
7, we graph the increase in the level of income per capita in each
year relative to the 2005 baseline. In 2010, this increase is 3.9
percent.
6. We agree with MCC that "the use of five-year cumulative figures
is misleading." Since MCC's public documents do not state that the
increase in per capita income is cumulative, we maintain that
MCC's statements regarding the increase in average per capita
income are inconsistent with its data.
7. MCC comments that "the data also indicate that the effect on
the per capita GDP growth rate will be moderate but rising, albeit
much smaller than three percentage points." (We understand this
statement to mean that the effect on aggregate GDP growth will be
smaller than 3 percentage points, given that the sentences that
follow the statement discuss aggregate, not per capita, GDP.) As
the note to figure 9 of our report indicates, we agree with MCC
that its data indicate that the compact will have a positive,
albeit small, effect on GDP growth rate. MCC provided us with its
computation showing that the compact's effect on GDP growth rate
will increase over a 20-year period and will reach 0.54 percentage
points in year 20--not the 3 percent growth rate increase implied
by its portrayal of the compact's projected results. Based on
MCC's calculation, the compact will enable the Vanuatu economy to
double in size in 20 years. Without the compact, the Vanuatu
economy will double in size in 24 years.
8. MCC comments that "GAO's presentation suggests that [near and
non-poor] households (local business people, government personnel,
transport providers, and tourism service providers) could feasibly
be excluded and that this would be desirable." We are not
suggesting that such people could be feasibly excluded or that
this would be desirable. Our report notes, "Given that many of
MCC's benefits flow to the expatriate-dominated tourism sector,
MCC makes an appropriate assumption that the local population will
not receive all the benefits." Our report's analysis indicates
that there will be poor, rural beneficiaries in addition to other
beneficiaries but observes that MCC has not quantified what
portion of the compact's benefits the poor, rural beneficiaries
will receive.
9. MCC states that we argue that the total number of rural
beneficiaries should be reduced by one-third. In fact, our report
observes that the total number of beneficiaries may be overstated.
MCC specifically disputes our observation that households living
in the hinterland or on off-shore islets may not be beneficiaries.
However, of the 13,619 persons in the catchment areas that we
question, 6,889 (51 percent) reside not in the hinterland or
off-shore islets but near the Efate tourism center and capital of
Port Vila and along a portion of the Ring Road that is already
paved. Given that MCC's benefits in Efate are primarily based on
increased tourist activity and road user cost savings from the
improved road, it is not clear how construction of a road
elsewhere on the island will lead to benefits when the existing
paved road already provides access to Port Vila. As documented in
appendix III, we do question whether inhabitants of off-shore
islets and of villages more than 10 kilometers from MCC projects
would benefit fully from MCC's projects. MCC includes these people
in its count of beneficiaries without accounting for, and
quantifying, the likely decrease in benefits that will accrue to a
person who lives both across an occasionally impassable river and
more than 10 kilometers from an MCC project bridge. As MCC's
internal analysis states regarding the Santo South Coast Road,
"the project does not entail a general improvement of the road,
only of river crossings. Neither would it give all-weather access
all the way to Tasiriki, and so to the west coast. It must be
said, therefore, that the scale of these benefits are more than
usually doubtful." Elsewhere in the report, it notes that the
"benefits to the west coast [Santo] population will be slight."
10. MCC asserts that its definition of "local population" does not
include urban dwellers. Although we asked MCC to support this
assertion, we received no additional documents. In separate
comments, MCC added that although its documentation does not
contain a very detailed reference specifically saying exactly how
many business people, transport providers and tourism operators
are foreigners and urban, MCC believes it is reasonable to
conclude that urban dwellers are included in these categories.
However, we note that not every urban dweller is a business
person, transport provider, or tourism operator and that some
urban beneficiaries would therefore fall into the "local
population" that receives an estimated 43 percent of benefits. On
the basis of MCC's definitions of "local population" and other
terms, we determined that "local population" includes residents of
nonrural areas. MCC's definition of the term local population
states, "We have combined three categories: local producers, local
consumers and inhabitants of remote communities. In reality these
are the same people and we have labeled them simply `Local
population.'" MCC defines local producers as including
"landowners; existing and potential lessees of land; processors of
primary produce, chiefly for export; and the owners and operators
of tourist facilities." This definition does not distinguish
between rural and urban dwellers. Further, given that expatriates
dominate in Vanuatu's formal economy sectors and that the urban
center of Port Vila is the center of Vanuatu's tourist industry,
it is likely that "processors of primary produce, chiefly for
export; and the owners and operators of tourist facilities"
include urban dwellers and those that are not poor. Finally, MCC's
definition of "inhabitants of remote communities" includes the
poor inhabitants of the "informal urban settlements" on the
outskirts of Port Vila. Although these people come from rural
areas, they are now in an urban environment and included within
MCC's definition of the local population.
11. MCC states that it has "a reasonable expectation that the
majority of the 65,000 [beneficiaries] are poor." However, MCC's
public documents use the phrase "65,000 poor, rural inhabitants,"
without acknowledging that not all of the 65,000 beneficiaries
live in poverty. We agree that MCC would be correct to suggest
that the majority of the projected beneficiaries are poor, as our
report states: "The poverty level, defined as having an income of
one U.S. dollar per day, is 51 percent in rural areas."
12. About program risks in general, MCC states that it differs
from us significantly regarding the precise nature and severity of
risks. As we note in our report, we offer several alternative
scenarios to illustrate the maximum impact of individual areas of
risk on projected compact benefits. These scenarios examine the
effect of individual risk areas and do not consider the
possibility of projects' failing to achieve certain benefits in
multiple areas simultaneously.
13. We agree with MCC that attempting to quantify induced benefits
is important and that our fieldwork generally affirmed MCC's
assumptions. We maintain that quantifying induced benefits
constitutes a risk to projected impact, because realization of
such benefits depends on beneficiaries' response to the
opportunities that the compact provides.
14. We asked MCC to support its assertion that "it is highly
plausible in a poor country that the vast majority of [time and
cost] savings would be used to generate income." We did not
receive additional documentation supporting this assertion.
However, MCC stated that its assertion is based on extensive field
consultations with beneficiary groups. MCC's documentation of its
methodology for assessing efficiency gains states that its
valuation of the average value of passengers' time is "extremely
crude." MCC's documentation adds: "The very concept of `working
hours' in a largely subsistence economy is open to question. And
what is the value of tourists' time to the national economy? It is
arguable that faster journey times allow tourists more time to
spend on things other than travel. But there is no research to
support a more sophisticated approach to valuing time."
15. MCC notes other factors that may benefit the economy. As our
report indicates (see footnote 15), we agree that "other benefits
may accrue to Vanuatu from the MCC compact. Increased economic
activity in tourism may have spillover benefits for other sectors
of the economy, and the welfare of Vanuatu's citizens may improve
due to increased access to health care and educational
opportunities."
16. MCC believes that we have overstated the risk from poor
maintenance of the infrastructure investment and have not
accounted for Vanuatu's increasing its maintenance spending by
$4.6 million or the assistance and maintenance equipment MCC is
providing to Vanuatu. We have added additional information to our
report regarding MCC's program to improve maintenance capability.
Nevertheless, maintenance remains a significant risk. As we noted
in our report, Vanuatu's record on transport infrastructure
maintenance is poor. Other donors, including officials from the
ADB and the World Bank, identified maintenance as a significant
challenge.
17. MCC states that "the projects that were under consideration in
the compact were at an advanced stage in preparation, either at
preliminary design or final design stage," not at the alternatives
analysis stage. Our previous report, which cited a study by
Flyvbjerg, Holm, and Buhl as part of a larger report on improving
cost and benefit analysis for transportation projects, used the
term "alternatives analysis stage"; however, the study uses the
term "time of decision to build" to define the point of
comparison. MCC's "time of decision to build" is when it has
completed its due diligence process and decided to sign a compact.
As such, the basis for comparison in the study by Flyvbjerg et al.
and in our report is identical. To help clarify the basis of
comparison, we have replaced "alternatives analysis stage" with
"time of decision to build." MCC states that it provided us the
final cost estimates, including price and physical contingencies,
for each subproject within the compact. However, as our report
notes, we did not receive full documentation of the cost estimates
and contingencies that MCC used in developing its economic
analyses. We received no supporting documentation for the cost and
contingency estimates for the Santo and Efate projects, which
account for 56 percent of the total compact budget. A key document
that MCC's contractor told us it used as a basis for developing
the Santo and Efate costs was not available to us because the
government of Vanuatu would not provide it to MCC. The
documentation of MCC's cost estimates and contingencies for
specific projects that we received shows a design contingency of
20 percent. (One project--MCC's smallest project, the $400,000
South West Bay airstrip--adds an additional 10 percent
construction contingency.) This 20 percent contingency is slightly
less than the average cost overrun found by Flyvbjerg et al. MCC
may complete the projects at, under, or above a 20 percent
contingency. However, cost overruns remain a risk to project
benefits, and we have presented them as such.
18. See comment 2.
Appendix VII: GAO Contact and Staff Acknowledgments
GAO Contact
David B. Gootnick, Director, 202-512-3149
Staff Acknowledgments
In addition to the person named above, Emil Friberg, Jr. (Assistant
Director), Gergana Danailova-Trainor, Reid Lowe, Angie Nichols-Friedman,
Michael Simon, and Seyda Wentworth made key contributions to this report.
Also, David Dornisch, Etana Finkler, Ernie Jackson, and Tom McCool
provided technical assistance.
(320451)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( [39]www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
[40]www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: [41]www.gao.gov/fraudnet/fraudnet.htm
E-mail: [42][email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [43][email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [44][email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
[45]www.gao.gov/cgi-bin/getrpt?GAO-07-909 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact David Gootnick, (202) 512-3149,
[email protected].
Highlights of [46]GAO-07-909 , a report to the Chairman, Committee on
Foreign Affairs, House of Representatives
July 2007
MILLENNIUM CHALLENGE CORPORATION
Vanuatu Compact Overstates Projected Program Impact
In January 2004, Congress established the Millennium Challenge Corporation
(MCC) for foreign assistance. Congress has appropriated almost $6 billion
to MCC. As of March 2007, MCC had signed almost $3 billion in compacts
with 11 countries, including a 5-year, $65.7 million compact with Vanuatu.
MCC states that the Vanuatu compact will have a transformational effect on
the country's economy, increasing per capita income and GDP and benefiting
65,000 poor, rural people. GAO examined (1) MCC's methods of projecting
economic benefits, (2) MCC's portrayal and analysis of the projected
benefits, and (3) risks that may affect the compact's impact. GAO reviewed
MCC's analyses and met with officials and business owners in Vanuatu as
well as with other donors.
[47]What GAO Recommends
GAO recommends that the Chief Executive Officer of MCC (1) revise the
public reporting of the Vanuatu compact's projected impact, (2) assess
whether similar reporting in other compacts accurately reflects underlying
analyses, and (3) improve its economic analyses by more fully accounting
for risks to project benefits. MCC did not directly address our
recommendations but commented that it had not intended to make misleading
statements and that its portrayal of projected results was factual and
consistent with underlying data.
MCC projects that the Vanuatu compact's transportation infrastructure
projects will provide direct benefits such as reduced transportation costs
and induced benefits from growth in tourism and agriculture. MCC estimated
the costs and benefits over 20 years, with benefits beginning in full in
2008 or 2009 and growing each year, and it counted poor, rural
beneficiaries by defining the area where benefits were likely to accrue.
Using projected benefits and costs, MCC calculated the compact's economic
rate of return (ERR) and its effects on Vanuatu's gross domestic product
(GDP) and per capita income.
MCC's portrayal of the projected impact does not reflect its underlying
data. MCC states that per capita income will increase by approximately
$200, or 15 percent, by 2010 and by $488, or 37 percent, by 2015. However,
MCC's underlying data show that these figures represent the sum of
individual years' gains in per capita income relative to 2005 and that
actual gains will be $51, or 3.9 percent, in 2010 and $61, or 4.6 percent,
in 2015. MCC also states that GDP will increase by an additional 3 percent
a year, but its data show that after GDP growth of 6 percent in 2007, the
economy's growth will continue at about 3 percent, as it would without the
compact. MCC states that the compact will benefit approximately 65,000
poor, rural inhabitants, but this statement does not identify the
financial benefits that accrue to the rural poor or reflect its own
analysis that 57 percent of benefits go to others.
We identified five key risks that could affect the compact's projected
impacts. (1) Cost estimate contingencies may not be sufficient to cover
project overruns. (2) Compact benefits will likely accrue more slowly than
MCC projected. (3) Benefit estimates assume continued maintenance, but
MCC's ability to ensure maintenance will end in 2011, and Vanuatu's
maintenance record is poor. (4) Induced benefits depend on businesses' and
residents' response to new opportunities. (5) Efficiency gains, such as
time saved in transit, may not increase per capita income. Our analysis of
these areas of risk illustrates the extent that MCC's projections are
dependent on assumptions of immediate realization of benefits, long-term
maintenance, realization of induced benefits, and benefits from efficiency
gains.
Vanuatu Compact's Impact on Per Capita Income According to MCC Statement
vs. MCC Data
References
Visible links
35. http://www.gao.gov/cgi-bin/getrpt?GAO-06-805
36. http://www.gao.gov/cgi-bin/getrpt?GAO-07-357
37. http://www.gao.gov/cgi-bin/getrpt?GAO-05-172
38. http://www.gao.gov/cgi-bin/getrpt?GAO-07-268
39. http://www.gao.gov/
40. http://www.gao.gov/
41. http://www.gao.gov/fraudnet/fraudnet.htm
42. mailto:[email protected]
43. mailto:[email protected]
44. mailto:[email protected]
45. http://www.gao.gov/cgi-bin/getrpt?GAO-07-909
46. http://www.gao.gov/cgi-bin/getrpt?GAO-07-909
*** End of document. ***