Millennium Challenge Corporation: Independent Reviews and
Consistent Approaches Will Strengthen Projections of Program
Impact (17-JUN-08, GAO-08-730).
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) for foreign assistance. Eligible countries
submit compact proposals for MCC funding for projects aimed at
reducing poverty through economic growth. To assess the proposed
compacts' likely impact, MCC performs economic analyses
estimating the compacts' economic rate of return (ERR) and
effects on income and poverty as well as the number of compact
beneficiaries. MCC uses these analyses to inform its decisions to
fund proposed compacts and to inform Congress and the public
about its progress in achieving its mission of poverty reduction
through economic growth. GAO was asked to examine MCC's
projections of (1) ERR and (2) compacts' impact on income and
poverty as well as numbers of beneficiaries. GAO reviewed MCC's
stated impacts and analyses for four MCC compacts that
represented 41 percent of MCC's compact assistance and met with
MCC officials.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-08-730
ACCNO: A82389
TITLE: Millennium Challenge Corporation: Independent Reviews and
Consistent Approaches Will Strengthen Projections of Program
Impact
DATE: 06/17/2008
SUBJECT: Agency missions
Beneficiaries
Cost analysis
Disadvantaged persons
Economic analysis
Economic growth
Economic policies
Economically depressed areas
Eligibility determinations
Federal aid to foreign countries
Foreign corporations
Foreign economic assistance
Impacted areas
Internal controls
International agreements
International economic relations
International organizations
International relations
Program evaluation
Records management
Reporting requirements
Standards
Foreign countries
Armenia
El Salvador
Lesotho
Mozambique
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-08-730
This text file was formatted by the U.S. Government Accountability
Office (GAO) to be accessible to users with visual impairments, as part
of a longer term project to improve GAO products' accessibility. Every
attempt has been made to maintain the structural and data integrity of
the original printed product. Accessibility features, such as text
descriptions of tables, consecutively numbered footnotes placed at the
end of the file, and the text of agency comment letters, are provided
but may not exactly duplicate the presentation or format of the printed
version. The portable document format (PDF) file is an exact electronic
replica of the printed version. We welcome your feedback. Please E-mail
your comments regarding the contents or accessibility features of this
document to [email protected].
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed
in its entirety without further permission from GAO. Because this work
may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this
material separately.
Report to the Chairman, Committee on Foreign Affairs, House of
Representatives:
United States Government Accountability Office:
GAO:
June 2008:
Millennium Challenge Corporation:
Independent Reviews and Consistent Approaches Will Strengthen
Projections of Program Impact:
GAO-08-730:
GAO Highlights:
Highlights of GAO-08-730, a report to the Chairman, Committee on
Foreign Affairs, House of Representatives.
Why GAO Did This Study:
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) for foreign assistance. Eligible countries submit
compact proposals for MCC funding for projects aimed at reducing
poverty through economic growth. To assess the proposed compacts�
likely impact, MCC performs economic analyses estimating the compacts�
economic rate of return (ERR) and effects on income and poverty as well
as the number of compact beneficiaries. MCC uses these analyses to
inform its decisions to fund proposed compacts and to inform Congress
and the public about its progress in achieving its mission of poverty
reduction through economic growth. GAO was asked to examine MCC�s
projections of (1) ERR and (2) compacts� impact on income and poverty
as well as numbers of beneficiaries. GAO reviewed MCC�s stated impacts
and analyses for four MCC compacts that represented 41 percent of MCC�s
compact assistance and met with MCC officials.
What GAO Found:
MCC used different time frames and methods to calculate ERRs for its
compacts with Armenia, El Salvador, Lesotho, and Mozambique. In
calculating ERR for 20 projects within the compacts, MCC used a 20-year
time frame for 9 projects and used different time frames for the other
11. In 2 of the 11 projects, using a 20-year time frame, as MCC used
for similar projects, would reduce the ERR below the level MCC set as
the minimum acceptable ERR. At the compact level, MCC�s use of varying
time frames did not affect the ERRs significantly. MCC used varying
methods to account for the costs of same-sector projects, although its
approaches to determining project benefits were generally similar. MCC
also used two different methods to calculate compact-level ERR;
however, the choice of method did not reduce it below the minimum ERR.
In three of the four compacts that we reviewed, MCC did not retain
documentation of the economic analyses used to support the investment
decision, but continued to modify the analyses. MCC has recently begun
to standardize elements of its economic analyses and centralize its
records management.
MCC identified and corrected analytic errors in its projections of
impact on income and poverty. MCC also used varying methods to project
these impacts.
* In responding to GAO�s questions about its published projections of
impact on income and poverty, MCC identified analytic errors for three
of the four compacts and, in correcting these errors, generally lowered
the projected impacts. Correcting these errors raised one projection by
5 percent but reduced others by 3 percent to 96 percent. According to
MCC officials, the revised projections would not have affected MCC�s
decision to recommend signing the compacts. The officials noted that,
in the future, compact impact projections will undergo peer review.
However, MCC has not documented procedures for these reviews.
* MCC used varying methods for its projections of impact on income and
poverty, limiting comparability and replicability across compacts. To
project impact on income for Armenia, El Salvador, and Mozambique, MCC
estimated the compacts� impact by summing the total benefits of
individual compact projects and adding them to the income that would
have prevailed without MCC. However, for Lesotho, MCC estimated the
impact on income based on the published results of a World Bank model
based on elements different from those of the MCC compact. In response
to our questions, MCC revised its initial estimate of the effect of
income growth on poverty for Mozambique by presenting two estimates,
based on Mozambique-specific and cross-country data, respectively.
Although a number of methods for projecting poverty impact are valid,
the method chosen can affect the results, and MCC�s guidelines do not
identify preferred methods for these calculations. MCC also used
varying methods to estimate numbers of beneficiaries for the compacts
and has not provided specific criteria for defining beneficiaries;
however, MCC officials reported they are taking steps to provide more
detailed guidance.
What GAO Recommends:
GAO recommends that the Chief Executive Officer of MCC (1) adopt and
implement written procedures for a secondary independent review of its
economic analyses and (2) improve MCC�s guidelines by identifying a
consistent approach with preferred methods for projecting compacts�
impact on income and poverty. MCC concurred with GAO�s recommendations.
To view the full product, including the scope and methodology, click on
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-730]. For more
information, contact David Gootnick at (202) 512-3149 or
[email protected].
[End of section]
Contents:
Letter:
Results in Brief:
Background:
MCC Used Different Time Frames and Methods to Calculate ERRs but Is
Taking Steps to Increase Consistency:
MCC Made Analytic Errors in Compact Impact Projections and Used Varying
Methods that Affected the Projections' Results:
Conclusions:
Recommendations for Executive Action:
Agency Comments and Our Evaluation:
Appendix I: Objectives, Scope, and Methodology:
Appendix II: MCC Minimum Acceptable ERRs:
Appendix III: Compact ERRs:
Appendix IV: Impact of Alternative Beneficiary Counts on El Salvador
Compact Impact Projections:
Appendix V: Comments from the Millennium Challenge Corporation:
GAO Comments:
Appendix VI: GAO Contact and Staff Acknowledgments:
Tables:
Table 1: Compact Impact Indicators Included in MCC's Public Reporting:
Table 2: MCC Compact Hurdle Rates and Hurdle Rate Definition:
Table 3: Comparison of MCC Compact ERRs Stated in Investment Memo with
ERRs Calculated Using a 20-Year Time Frame:
Table 4: Compact ERR Using Alternative Methods:
Table 5: MCC's Compact-Level Impact Estimates for El Salvador, with
Alternative Assumptions for Estimating Beneficiaries of Education
Projects:
Figures:
Figure 1: MCC Compact Development and Implementation Process:
Figure 2: Illustrative Examples of MCC Economic Analysis at Compact,
Project, and Activity Levels:
Figure 3: Illustration of Net benefits and ERR Calculations:
Figure 4: Summary of Compacts for Armenia, El Salvador, Lesotho, and
Mozambique:
Figure 5: MCC Revisions to Impact Projections for Armenia, Based on Its
Corrections of Analytic Errors:
Figure 6: MCC Revisions to Impact Projections for El Salvador, Based on
Its Corrections of Analytic Errors:
Figure 7: MCC Revisions to Impact Projections for Mozambique, Based on
Its Corrections of Analytic Errors:
Figure 8: MCC's Alternative Methods for Calculating Compact ERR:
Abbreviations:
CEO: chief executive officer:
ERR: economic rate of return:
GDP: gross domestic product:
MCA: Millennium Challenge Account:
MCC: Millennium Challenge Corporation:
OMB: Office of Management and Budget:
[End of section]
United States Government Accountability Office:
Washington, DC 20548:
June 17, 2008:
The Honorable Howard L. Berman:
Chairman:
Committee on Foreign Affairs:
House of Representatives:
Dear Chairman Berman:
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) to administer the Millennium Challenge Account (MCA)
for foreign assistance. MCC's mission is to reduce poverty through
sustainable economic growth in some of the world's poorest countries
that create and maintain sound policy environments. MCC has received
appropriations for fiscal years 2004 to 2008 totaling more than $7.5
billion and, as of March 2008, has signed $5.5 billion in compacts
[Footnote 1] with 16 countries. The President has requested $2.225
billion for fiscal year 2009.
MCC uses income and country performance criteria to annually select a
list of countries eligible for MCA assistance. Eligible countries may
then submit proposals for compacts containing multiple projects for
MCC's review and approval. On receiving proposals for MCA assistance,
MCC undertakes a comprehensive review, or due diligence,[Footnote 2]
which seeks to ensure that the proposed compacts will advance MCC's
mission. As part of due diligence, MCC assesses the potential economic
impact of each compact. During due diligence, MCC identifies a minimum
acceptable economic rate of return[Footnote 3] (ERR) for the compact
and its projects; compares estimated costs and benefits to determine
the compact's ERR; and projects the compact's impact, including its
number of beneficiaries and its impact on income, economic growth, and
poverty.
MCC uses its projections, as well as other information gathered during
due diligence, to inform its internal decisions to fund proposed
projects and compacts. The results of the due diligence assessment are
reported in an investment memo--an internal document prepared by MCC's
transaction team[Footnote 4] that analyzes the compact--submitted to
MCC's investment committee.[Footnote 5] MCC also publishes its compact
impact projections during ongoing consultations with Congress. This
information--found in MCC documents such as compacts, compact
summaries, annual reports, and congressional notifications and budget
justifications--sets expectations for the compact and provides
information to Congress and the public about MCC's progress in
achieving its mission.
In July 2007, we reported that MCC's portrayal of the projected impact
of its Vanuatu compact did not reflect MCC's underlying analyses.
[Footnote 6] Our recommendations included that MCC revise its public
reporting of the Vanuatu compact's projected impact and assess whether
similar reporting for other compacts accurately reflects underlying
economic analyses. The committee subsequently requested that we examine
MCC's economic analyses for its compacts with other countries.
As agreed with your office, for this report, we assessed (1) MCC's
projections of ERR and (2) MCC's projections of compacts' impact on
income and poverty as well as numbers of beneficiaries.
To carry out this review, we reviewed MCC compacts with four countries:
Armenia, El Salvador, Lesotho, and Mozambique. When we began our work
in August 2007, these four countries represented about 41 percent of
the $3.85 billion MCC had set aside for 13 compacts and included the
most recent compacts signed in Eurasia, Latin America, and Africa. To
assess MCC's compact projections, we reviewed MCC public and internal
documents as well as relevant World Bank and International Monetary
Fund documents. We also reviewed MCC's original and revised spreadsheet
calculations of its economic projections and interviewed MCC economists
and other officials in Washington, D.C., regarding these analyses.
Finally, we consulted Office of Management and Budget (OMB) and GAO
guidance on establishing and implementing internal controls.[Footnote
7] We determined that the data we used were sufficiently reliable for
the purposes of our analysis; however, we did not independently assess
the reliability of all data and assumptions that affect the projections
and that MCC used in its underlying economic analyses of compact
projects and activities. We also did not assess MCC's progress in
implementing these compacts or toward its projected results. We
conducted this performance audit from August 2007 to June 2008, in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives. (See app. I for additional
details of our scope and methodology.)
Results in Brief:
MCC used different time frames and methods to calculate ERRs for the
four compacts we reviewed. In calculating ERR for the 20 projects
within the compacts that we reviewed, MCC used a 20-year time frame for
9 projects and used different time frames for the other 11 projects. In
2 of the 11 projects, applying a 20-year time frame, as MCC had done in
similar projects, would reduce the ERR below the level MCC set as the
minimum acceptable ERR. At the compact level, MCC's use of varying time
frames did not significantly affect the results of the ERR
calculations. MCC used varying methods to account for the costs of same-
sector projects, although its approaches to determining project
benefits were generally similar. MCC also used two different methods to
calculate compact-level ERR; however, the choice of method did not
reduce it below the minimum ERR. In three of the four compacts that we
reviewed, MCC did not retain documentation of the economic analyses
used to support the investment decision, but continued to modify the
analyses. MCC has recently begun to standardize elements of its
economic analyses and centralize its records management.
MCC identified and corrected analytic errors in its projections of
impact on income and poverty. In addition, MCC used varying methods for
projecting compact impact on income and poverty, which affected the
projections' results.
* Analytic errors. In responding to our questions about its published
impact projections, MCC identified analytic errors for three of the
four compacts[Footnote 8] and, in correcting these errors, generally
lowered its projected impacts on poverty and income. Correcting these
errors raised one projection by 5 percent but reduced others by 3
percent to 96 percent. For example, for Armenia, MCC corrected an
erroneous baseline, reducing the projected decrease in rural poverty
from 6 percentage points to 3 percentage points. For Mozambique, MCC
used a corrected formula and data, reducing its projection of the
number of persons to be lifted out of poverty in Mozambique in 2015
from 270,000 to either 27,000 or 56,000, depending on the approach
selected. According to MCC officials, the revised projections would not
have affected MCC's decision to recommend signing the compacts. The
officials noted that MCC's impact projections had not undergone a final
check for accuracy and validity but that MCC has begun to implement a
peer review process for compacts currently under development. However,
MCC has not documented procedures for these reviews.
* Varying methods. MCC used varying methods to project the four
compacts' impact on income and poverty, limiting the projections'
comparability and replicability across compacts. For example, for
Armenia, El Salvador, and Mozambique, MCC estimated the impact of
compact projects on income by summing the total benefits of individual
compact projects and adding them to the income that would have
prevailed without MCC. For Lesotho, MCC extrapolated its compact
results from the published results of a World Bank model based on
elements different from those of the MCC compact. In response to our
questions, MCC revised its poverty impact projection for Mozambique by
presenting two estimates of the relationship between income growth and
poverty--estimates based on either Mozambique country-specific data or
its initial ad hoc estimate, which MCC stated was consistent with cross-
country experience. Although a number of methods for projecting impact
are valid, the method chosen can affect the results, and MCC's
guidelines do not identify preferred methods for these calculations.
MCC also used varying methods to estimate numbers of beneficiaries for
the compacts and has not provided specific criteria for defining
beneficiaries; however, MCC officials reported they are taking steps to
provide more detailed guidance for estimating beneficiaries.
To improve the reliability and comparability of its projected ERR and
economic impacts, we recommend that the CEO of MCC take the following
actions:
* adopt and implement written procedures for a secondary independent
review of the methods and results of its economic analyses and:
* improve MCC's guidelines by identifying a consistent approach with
preferred methods for projecting compacts' impact on income and
poverty.
In commenting on a draft of this report, MCC concurred with our
recommendations and outlined steps it is taking, including developing
standard practices and templates and initiating a peer review process.
MCC stated that in many cases the inconsistencies we identified were
technically appropriate. MCC also stated that it disagreed with what it
saw as our report's implication that ERRs are disconnected from income
and poverty effects. However, we assert that ERRs do not provide
information about MCC's impact on the poor and are not an absolute
measure of income benefits, but a relative measure of benefits in
relation to costs. We have reprinted MCC's comments, with our response,
in appendix V. We have incorporated technical clarifications from MCC
where appropriate.
Background:
MCC conducts its economic analyses during the compact development
process, the first of three compact phases. These analyses include
establishing the minimum ERR that the compact should achieve,
projecting the compact's ERR, and estimating the compact's impact on
income and growth. MCC did not publish ERR information for three of the
four compacts we reviewed, but its public documents include statements
of compact impact on income and poverty. Of the four compacts that we
reviewed--Armenia, El Salvador, Lesotho, and Mozambique--three include
funding for road projects, and three include funding for water
projects, among other projects.
Compact Development Process:
If MCC determines that a country is eligible for assistance, the
country may submit a compact proposal, which generally comprises
several projects. According to MCC guidelines,[Footnote 9] the proposal
should include economic analyses of proposed projects to demonstrate
their likely impact on growth and poverty in the country. MCC also may
provide assistance and feedback in developing the proposal, including
making grants to facilitate the development of the compact (see fig.
1).
Figure 1: MCC Compact Development and Implementation Process:
[See PDF for image]
This figure is an illustration of the MCC Compact Development and
Implementation Process, as follows:
Development compact:
* Eligibility determination:
- Country proposal development;
* Opportunity memo:
- MCC's due diligence review;
* Investment memo:
- Compact negotiation and MCC Board approval.
Finalize supplemental agreements:
* Compact signing:
- MCC and country complete entry into force requirements.
Implement compact:
* Entry into force:
- MCC authorizes fund disbursement and oversees country implementation
of compact.
Source: GAO analysis of MCC data.
[End of figure]
After a country submits its proposal, MCC's transaction team for the
compact conducts a preliminary assessment of the proposal and reports
its findings in an internal opportunity memo to the MCC investment
committee. MCC assembles a different transaction team for each compact.
If the opportunity memo is approved, the team launches a detailed due
diligence review that includes economic analyses of the proposed
projects. As members of MCC country transaction teams, MCC's lead
economists undertake these analyses while working with other members of
the team and country officials. According to MCC, as part of this
effort, MCC economists review the preliminary analyses performed by
country counterparts and consultants. These due diligence reviews,
including conducting economic analyses, have lasted, on average,
slightly more than 10 months for the 16 countries with signed compacts
as of March 31, 2008.
At the conclusion of due diligence, the transaction team sends an
investment memo to the MCC investment committee, with recommendations
based on its assessment of the proposal. MCC notifies Congress 15 days
prior to beginning negotiations with the country, sending a formal
congressional notification.[Footnote 10] For the four compacts we
examined, the congressional notifications included some of the results
of MCC's economic analyses. If compact negotiations with the eligible
country are successful, the investment committee submits the proposed
compact to the MCC Board for approval.[Footnote 11] With the Board's
approval, MCC and the country sign the compact before completing
additional agreements--such as a disbursement agreement and procurement
agreement--and ultimately implementing the projects funded in the
compact.[Footnote 12]
Economic Analyses:
For each compact proposal, MCC and the eligible country conduct
economic analyses, which MCC uses to inform its decisions to fund
compacts and report to Congress and the public.[Footnote 13] MCC
generally conducts these analyses at the compact and project levels
(see fig. 2).
Figure 2: Illustrative Examples of MCC Economic Analysis at Compact,
Project, and Activity Levels:
[See PDF for image]
This illustration offers examples of MCC economic analysis at compact,
project, and activity levels, as follows:
Compact-level analysis:
* Aggregate economic rate of return (ERR);
* Aggregate income and poverty effects;
* Aggregate number of beneficiaries.
Project-level analysis:
* ERR, number of beneficiaries;
Illustrative examples:
- Water and sanitation;
- Road construction.
Activity level analysis:
* ERR;
Illustrative examples:
- Individual city systems;
- Individual road segments.
Sources: GAO analysis of MCC data; Nova Development (clip art).
[End of figure]
The economic analyses include projections of compacts' impact,
including their ERR, the impact on income and poverty, and the
projected number of beneficiaries.
* ERR analysis. To provide a basis for assessing the compact and
project ERRs, MCC sets the minimum acceptable ERR that compacts and
projects should achieve to be eligible for funding. (See app. II for a
discussion of MCC's minimum ERRs for the four countries.) If the
compact or project does not meet the minimum rate, MCC retains the
discretion to fund it but requires justification based on the specific
circumstances.
The ERR analysis compares costs and benefits, where the costs are the
MCA grants and the country's future recurrent costs--such as
maintenance expenses--and the benefits are increases in incomes in
recipient countries. (See fig. 3.) MCC guidelines state that normal
practice is to calculate ERRs using 10-, 20-, and 30-year time horizons
to determine project and compact ERRs' sensitivity to varying time
frames. If the ERR is sensitive to this time horizon, the guidelines
require that this be noted explicitly.
Figure 3: Illustration of Net benefits and ERR Calculations:
[See PDF for image]
This illustration depicts the following Net benefits and ERR
calculations:
Benefits, minus Costs, equals Net benefits, generates ERR.
Year: 1;
Annual benefit to the country: Benefits1;
Expenditures to implement project(s): Costs1;
Annual average expected rate of return on each dollar of MCC
assistance: Net benefit1.
Year: 2-9;
Annual benefit to the country: [Empty];
Expenditures to implement project(s): [Empty];
Annual average expected rate of return on each dollar of MCC
assistance: [Empty].
Year: 10;
Annual benefit to the country: Benefits10;
Expenditures to implement project(s): Costs10;
Annual average expected rate of return on each dollar of MCC
assistance: Net benefit10;
Year 1-10 yields 10-year ERR.
Year: 11-19;
Annual benefit to the country: [Empty];
Expenditures to implement project(s): [Empty];
Annual average expected rate of return on each dollar of MCC
assistance: [Empty].
Year: 20;
Annual benefit to the country: Benefits20;
Expenditures to implement project(s): Costs20;
Annual average expected rate of return on each dollar of MCC
assistance: Net benefit20;
Year 1-20 yields 20-year ERR.
Year: 21-29;
Annual benefit to the country: [Empty];
Expenditures to implement project(s): [Empty];
Annual average expected rate of return on each dollar of MCC
assistance: [Empty].
Year: 30;
Annual benefit to the country: Benefits30;
Expenditures to implement project(s): Costs30;
Annual average expected rate of return on each dollar of MCC
assistance: Net benefit30;
Year 1-30 yields 30-year ERR.
Source: GAO analysis of MCC data.
Note: The ERR of a project is the discount rate (interest rate) at
which the present value of the project's cost stream is equal to the
present value of its benefits stream.
[End of figure]
* Impact analysis. MCC's guidelines state that economic analyses should
quantify the proposed projects' expected beneficiaries and expected
impact on incomes and on poverty.[Footnote 14] Based on our analysis,
MCC does not establish minimum thresholds for compact impact.
Public Reporting:
While MCC's transaction team reports both ERR and impact projections to
the MCC investment committee in its investment memo, MCC's public
reporting for the four compacts in our review generally included
information about only its projections of compact impact; for these
four countries MCC published ERR projections for only Armenia. MCC's
public reporting included projections of compact impact on income and
poverty for three countries and projections of gross domestic product
(GDP) growth for one country. For each compact, MCC also estimated the
number of beneficiaries. (See table 1.)
Table 1: Compact Impact Indicators Included in MCC's Public Reporting:
Increased income in compact areas:
Armenia: [Check];
El Salvador: [Check];
Lesotho: [Empty];
Mozambique: [Check].
Increased GDP growth rate:
Armenia: [Empty];
El Salvador: [Empty];
Lesotho: [Check];
Mozambique: [Empty].
Decreased poverty:
Armenia: [Check];
El Salvador: [Check];
Lesotho: [Empty];
Mozambique: [Check].
Number of beneficiaries:
Armenia: [Check];
El Salvador: [Check];
Lesotho: [Check];
Mozambique: [Check].
Source: GAO analysis of MCC compacts.
[End of table]
Compact Projects and Funding:
The types of projects included in the four compacts vary, although El
Salvador, Lesotho, and Mozambique include water projects[Footnote 15]
and Armenia, El Salvador, and Mozambique include road projects. The
compacts provide a total of approximately $1.56 billion in MCA
assistance.[Footnote 16] (See fig. 4.)
Figure 4: Summary of Compacts for Armenia, El Salvador, Lesotho, and
Mozambique:
[See PDF for image]
This figure is a map of the world, depicting the locations of Armenia,
El Salvador, Lesotho, and Mozambique, and providing the following
information:
Country: Armenia;
Compact size: $235.65 million;
Projects:
* Rural road rehabilitation;
* Irrigation;
Area of intervention: Rural areas.
Country: El Salvador;
Compact size: $460.94 million;
Projects:
* Education and community infrastructure;
* Agricultural productivity;
* Road construction and rehabilitation;
Area of intervention: Northern zone.
Country: Lesotho;
Compact size: $362.55 million;
Projects:
* Water and sanitation;
* Health;
* Private sector development;
Area of intervention: Entire country.
Country: Mozambique;
Compact size: $506.92 million;
Projects:
* Water and sanitation;
* Land tenure;
* Farmer income;
Area of intervention: Four northern provinces.
Sources: GAO analysis of MCC data; Map Resources (map).
[End of figure]
MCC Used Different Time Frames and Methods to Calculate ERRs but Is
Taking Steps to Increase Consistency:
MCC used different time frames to calculate project-level ERRs; these
differences did not affect the ERR significantly in 18 of the 20
compact projects[Footnote 17] we examined, but two ERRs would fall
below the minimum ERR if MCC applied the 20-year time frame it used for
other compacts and similar projects. At the compact level, the
different time frames did not change the ERR significantly. In
addition, we found that MCC used varying methods to account for the
costs of same-sector projects, although its approaches to determining
benefits were generally similar. MCC also used two different methods to
calculate compact-level ERR; however, the choice of method did not
change the ERRs significantly. In some cases, MCC did not fully retain
its documentation of its economic analyses. MCC has recently taken
steps to standardize elements of its economic analyses and improve its
records management and plans to implement additional measures.
MCC Used Different Time Frames for Project and Compact ERRs:
In its ERR calculations for 20 projects included in the four compacts
we reviewed, MCC used a 20-year time frame for 9 projects and used
different time frames for 11 projects. Nearly all project ERRs that MCC
initially calculated met the minimum ERR set for each compact.[Footnote
18] Our analysis shows that for 9 of the 11 ERRs calculated with other
time frames, recalculating the ERR with a 20-year time frame does not
lower the ERR below the minimum ERR. However, for two projects,
recalculating the ERR with a 20-year time frame produces a result below
the minimum ERR.
* The ERR for the El Salvador Community Infrastructure project, which
MCC calculated as slightly below the minimum ERR of 10.8 percent over
25 years, drops to 9.0 percent when calculated over 20 years.
Correcting a calculation error further reduces the El Salvador
project's ERR to 6.8 percent over 20 years--four percentage points
below the El Salvador minimum ERR.
* The ERR for the Mozambique roads project, which MCC calculated at
10.3 percent over 24 years, drops to 8.1 percent over 20 years, below
the minimum ERR of 8.76 percent. Three of the four individual
Mozambique roads MCC analyzed as part of this project also fall below
the minimum ERR at 20 years.[Footnote 19]
If MCC had applied a 20-year time frame in calculating the ERR for
these two projects, the projects might have been restructured to
increase their ERR or MCC would have had to specifically justify the
exception. MCC does not currently have a policy addressing what steps
to take in cases where subsequent analysis results in a project ERR
below the minimum. MCC stated that it would address changes such as
this on a case-by-case basis depending on the timing of the change
within the compact development process, and the magnitude of the
change. MCC officials also noted that the ERR time frame is only one
aspect of MCC's analysis of the ERR's sensitivity to various factors.
In addition, MCC used different time frames in calculating ERRs for
comparable projects in different compacts. MCC calculated water project
ERRs for El Salvador, Lesotho, and Mozambique over 25, 20, and 21
years, respectively, and calculated road project ERRs for Armenia, El
Salvador, and Mozambique over 20, 25, and 24 years, respectively. MCC
officials told us they explored the ERRs' sensitivity to varying time
frames, as MCC's guidelines require, but they did not regard as
prescriptive the guidelines' statement that normal practice is to
examine 10-, 20-, and 30-year time horizons.[Footnote 20] MCC officials
also noted that MCC economists have recently committed to the use of a
default 20-year time horizon for their analyses. MCC also will alter
this time frame for specific circumstances or projects whose benefits
have a longer duration--such as education projects and large-scale
construction projects--or have a shorter or longer physical life
expectancy.
At the compact level, our analysis shows that applying a 20-year time
frame in place of varying time frames does not significantly affect the
results of the ERR calculations. (See app. III for a summary of our
comparison of the compact minimum ERR with the compact ERRs that MCC
reported and our calculations of the 20-year ERRs.)
MCC Used Varying Methods in Calculating Project and Compact ERRs:
In calculating project ERRs, we found that MCC used varying methods to
account for the investment and recurrent cost components of water and
road sector projects for the four compacts we reviewed.
* Investment costs. For water projects, MCC counted the cost of
construction in Lesotho as one lump sum in the first year of the
analysis but phased in the cost in Mozambique and El Salvador over
multiple years. Phasing the costs evenly over the 5-year compact in
Lesotho would have increased the ERR for both the urban and rural water
projects.[Footnote 21] For road projects, MCC phased the investment
cost over time for Armenia, El Salvador, and Mozambique. However, in El
Salvador, MCC accounted for the salvage value[Footnote 22] of the
project in the last year of the analysis but did not include a similar
projection in the analysis for Armenia and Mozambique.
* Recurrent costs. For water projects, MCC counted Lesotho and
Mozambique's recurrent costs in the years after the original compact
investment. For El Salvador, MCC counted recurrent costs only during
the initial 5-year compact period. Including future recurrent costs
would have reduced the ERR. For road projects, MCC spread recurrent
costs over time for Armenia, El Salvador, and Mozambique.
According to our review, MCC's approaches to determining the benefits
of water and road projects were generally similar.
* Water projects. MCC generally counted as water project benefits
increased time available for work, resulting from less time spent
fetching water or being ill, and lower spending on health care and
water.
* Road projects. MCC generally estimated traffic volumes on the roads
and determined road project benefits based on the savings from reduced
travel time and vehicle operating costs for existing and generated
traffic.[Footnote 23]
In calculating the compact-level ERRs for the four compacts' investment
memos, MCC used two different methods.[Footnote 24]
* For Armenia and Mozambique, MCC first determined the net benefits in
each year for each project and then calculated the compact ERR based on
the total net benefits.
* For El Salvador and Lesotho, MCC first determined each project's net
benefits and ERR and then determined the compact ERR by averaging the
project ERRs, weighting each ERR according to the project's budgeted
size relative to the overall compact.
According to MCC officials, the choice of method depended on the
preference of the transaction team's lead economist. Our analysis shows
that using the second method to recalculate each compact ERR reduces it
by less than 2 percentage points and does not reduce it below the
minimum ERR. However, these results demonstrate that the choice of
method influences the compact-level ERR and may affect the
comparability of ERRs across compacts. (See app. III for details of our
analysis of MCC compact ERRs.)
Insufficient Records Management Led to Discrepancies between Investment
Memos and Underlying Analyses:
In three of the four cases we reviewed, the projections of ERR reported
to the MCC investment committee differ from those in MCC's underlying
spreadsheets.
* For Armenia, MCC changed its estimate of project benefits after the
investment memo.
* For Lesotho, the MCC spreadsheets contain figures calculated using
different methods or reflecting additional analysis after the
investment memo.
* For Mozambique, the calculation of the figures presented in the
investment memo relied on spreadsheet formula links that were not
properly updated at the time. According to MCC, when they provided the
spreadsheets to us, they updated the formulas and overwrote the
original calculations.
For these three compacts, MCC's records management did not fully
preserve the information and analysis used to support the investment
decision. OMB guidance and GAO guidelines for internal controls both
note the importance of controls over the information that U.S. agencies
use to make decisions.[Footnote 25]
MCC Has Taken Several Steps to Improve the Consistency of ERR Analysis:
After completing due diligence for the four compacts we studied, and
during the course of our audit, MCC told us they took or began taking
steps to increase the consistency of its analyses and improve its
records management. For example, MCC:
* committed to using 20 years as the default time frame for calculating
ERRs and identified projects where this time frame may be altered;
* began revising its guidance to clarify that ERR sensitivity should be
tested by varying the time frame of the analysis, rather than
specifying 10-, 20-, and 30-year analyses;
* began developing standards for consistently analyzing certain types
of projects across compacts;
* established that summing net benefits and costs across all projects
is the preferred method for calculating a compact-level ERR; and:
* planned to implement a data management system to centralize its
records management.
MCC Made Analytic Errors in Compact Impact Projections and Used Varying
Methods that Affected the Projections' Results:
MCC identified and corrected analytic errors in its projections of
compact impact for Armenia, El Salvador, and Mozambique, generally
reducing each compact's estimated impact on income and poverty.
[Footnote 26] According to MCC, the revised calculations would not have
affected its approval of the compacts, but it will perform peer reviews
of future impact projections. In addition, MCC used different methods
for projecting compact impact on income and poverty, limiting the
estimates' comparability and replicability; its current guidance does
not address the choice of method for these projections. MCC also used
different methods to estimate numbers of compact beneficiaries, but
stated that it is taking steps to provide more detailed guidance for
estimating number of beneficiaries.
MCC Identified and Corrected Analytic Errors for Three Compacts:
In responding to our questions about its impact analyses for Armenia,
El Salvador, and Mozambique,[Footnote 27] MCC identified a number of
analytic errors in its projections of impact on income and poverty. MCC
subsequently corrected these errors, generally reducing the projected
impacts on income and poverty for each compact.
* Armenia. MCC determined that it had included road project benefits in
estimating the income increase from agriculture for Armenia. MCC also
used the wrong baseline in projecting poverty effects. Correcting these
errors affected projections of the compact's effect, reducing the
estimated increase in rural areas' real income from agriculture after 5
years from 5 percent to 3 percent and lowering the estimated decline in
Armenia's poverty rate from 6 percentage points to 3 percentage points.
Figure 5 summarizes MCC's original impact projections for Armenia and
its revisions after correcting the errors it identified.
Figure 5: MCC Revisions to Impact Projections for Armenia, Based on Its
Corrections of Analytic Errors:
[See PDF for image]
This figure is a table depicting the following data:
Armenia:
Income/growth: Annual income in rural areas in 2010;
Original calculation: $36 million increase;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Income/growth: Annual income in rural areas in 2015;
Original calculation: $133 million increase;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Income/growth: Real income from agriculture in rural areas at end of
compact;
Original calculation: 5 percent increase;
MCC's revised calculation: 3 percent increase;
Absolute change: -2 percentage points;
Percent change: -40%.
Income/growth: Real income from agriculture in rural areas in 2013[A];
Original calculation: 23 percent increase;
MCC's revised calculation: 9 percent increase;
Absolute change: -14 percentage points;
Percent change: -61%.
Poverty: Rural poverty rate[B];
Original calculation: 6 percentage point decrease;
MCC's revised calculation: 3 percentage point decrease;
Absolute change: -3 percentage points;
Percent change: -50%.
Beneficiaries:
Original calculation: 750,000;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Source: GAO analysis of MCC data.
[A] MCC stated projections of real income increases in Armenia using an
index, with the baseline set at 100 in 2005. Thus, MCC's initial
projection of an increase from 2005 baseline index of 100 to 123 in
2013 corresponds to a 23 percent increase in real income, which was
later revised to 109, or a 9 percent increase, for the same year.
[B] MCC stated projections of poverty rate reductions in Armenia using
a baseline of 32 percent. Thus, MCC's initial projections of a
reduction in the poverty rate from 2004 baseline of 32 percent to a
target of 26 percent in year 2013 corresponds to a decline of 6
percentage points, which was later revised to a reduction to 29
percent, or a decline of 3 percentage points, for the same year.
[End of figure]
* El Salvador. MCC determined that it had made an error in its formula
for projecting per capita income increases and overestimated income
increases in its projections of poverty rate reduction for El Salvador.
In correcting these errors, MCC lowered its projections of income and
poverty impact. Figure 6 summarizes MCC's original impact projections
for El Salvador and its revisions after correcting the errors it
identified. MCC also presented alternative methods for identifying
beneficiaries, which further reduces the results of its compact-level
projections. (See app. IV.)
Figure 6: MCC Revisions to Impact Projections for El Salvador, Based on
Its Corrections of Analytic Errors:
[See PDF for image]
This figure is a table depicting the following data:
El Salvador:
Income/growth: Incomes within the region within 5 years;
Original calculation: 18 percent increase;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Income/growth: Incomes within the region within 10 years;
Original calculation: 26 percent increase;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Income/growth: Annual per capita income of beneficiaries within 5
years;
Original calculation: $148 increase;
MCC's revised calculation: $123 increase;
Absolute change: -$25;
Percent change: -17%.
Income/growth: Annual per capita income of beneficiaries within 10
years;
Original calculation: $230 increase;
MCC's revised calculation: $189 increase;
Absolute change: -$41;
Percent change: -18%.
Poverty: Number of persons for whom poverty is alleviated[A];
Original calculation: 150,000 persons;
MCC's revised calculation: 145,000 persons;
Absolute change: -5,000 persons;
Percent change: -3%.
Poverty: Poverty rate in the Northern Zone within 5 years;
Original calculation: 11 percentage point decrease;
MCC's revised calculation: 10 percentage point decrease;
Absolute change: -1 percentage point;
Percent change: -9%.
Poverty: Poverty rate in the Northern Zone within 10 years;
Original calculation: 17 percentage point decrease;
MCC's revised calculation: 15 percentage point decrease;
Absolute change: -2 percentage point2;
Percent change: -12%.
Beneficiaries:
Original calculation: 850,000;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Source: GAO analysis of MCC data.
Notes: For El Salvador, MCC calculated with-and without-project
scenarios and cited these figures in its public documents. To estimate
compact impact attributable to MCC, we calculated the differences
between with-and without-project scenarios.
MCC also presented alternative methods for identifying beneficiaries of
two education projects, which affect the results of MCC's compact
impact analysis. (See app. IV.)
[A] MCC's spreadsheet calculations show a figure of more than 160,000
Salvadorans lifted out of poverty in year 10 of the project. However,
MCC's compact summary stated that "the program is projected to directly
alleviate the poverty of over 150,000 Salvadorans." The compact
summary's projection of 150,000 is used here.
[End of figure]
* Mozambique. MCC determined that it had not updated formula links in
its analytic spreadsheets for Mozambique. Correcting this error led MCC
to revise the projected income increase upward by $4 million for 2015
and downward by $13 million for 2025. In addition, MCC corrected both
the formula and the data[Footnote 28] used to calculate the effect of
this income increase on poverty in Mozambique, presenting two
alternative approaches for estimating poverty elasticity.[Footnote 29]
In making these changes, MCC lowered the projected decline in the
poverty rate as well as the number of persons likely to be lifted out
of poverty because of the compact. For example, MCC originally
projected a 7 percent reduction in Mozambique's poverty rate in 2015;
using the alternative approaches for estimating poverty elasticity, MCC
projected a poverty rate reduction of either 0.6 percent or 2 percent
in 2015. Likewise, MCC originally projected that 270,000 people would
be lifted out of poverty in Mozambique in 2015; using the alternative
approaches, MCC projected that either 27,000 persons or 56,000 persons
would be lifted out of poverty in 2015. Figure 7 summarizes MCC's
original impact projections for Mozambique and its revisions after
correcting errors and presenting alternative approaches for estimating
poverty elasticity.
Figure 7: MCC Revisions to Impact Projections for Mozambique, Based on
Its Corrections of Analytic Errors:
[See PDF for image]
This figure is a table depicting the following data:
Mozambique:
Income/growth: Income in 2015;
Original calculation: $75 million increase;
MCC's revised calculation: $79 million increase;
Absolute change: +$4 million;
Percent change: +5%.
Income/growth: Income in 2025;
Original calculation: $180 million increase;
MCC's revised calculation: $167 million increase;
Absolute change: -$13 million;
Percent change: -7%.
Poverty: Poverty rate in 2015;
Original calculation: 7 percent decrease;
MCC's revised calculation: 0.6 decrease (lower estimate); 2 percent
decrease (higher estimate);
Absolute change: -6.4 percentage points (lower estimate); -5 percentage
points (higher estimate);
Percent change: -91% (lower estimate); -66% (higher estimate).
Poverty: Poverty rate in 2025;
Original calculation: 16 percent decrease;
MCC's revised calculation: 0.7 decrease (lower estimate); 3 percent
decrease (higher estimate);
Absolute change: -15.3 percentage points (lower estimate); -13
percentage points (higher estimate);
Percent change: -96% (lower estimate); -83% (higher estimate).
Poverty: Number of persons lifted out of poverty in 2015;
Original calculation: 270,000 persons;
MCC's revised calculation: 27,000 persons (lower estimate); 56,000
persons (higher estimate);
Absolute change: -243,000 persons (lower estimate); -214,000 persons
(higher estimate);
Percent change: -90% (lower estimate); -79% (higher estimate).
Poverty: Number of persons lifted out of poverty in 2025;
Original calculation: 440,000 persons;
MCC's revised calculation: 32,000 persons (lower estimate); 43,000
persons (higher estimate);
Absolute change: -408,000 persons (lower estimate); -397,000 persons
(higher estimate);
Percent change: -93% (lower estimate); -90% (higher estimate).
Beneficiaries:
Original calculation: 5 million by 2015;
MCC's revised calculation: No change;
Absolute change: 0;
Percent change: 0.
Source: GAO analysis of MCC data.
Notes: For Mozambique, MCC's public documents stated projections of
poverty (1) with the MCC compact and (2) without the MCC compact. To
estimate compact impact attributable to MCC, we calculated the
differences between with-and without-project figures.
In addition to correcting its formula used to calculate the effect of
increased income on poverty in Mozambique, MCC presented alternative
approaches for estimating poverty elasticity. MCC's poverty impact
projections using both alternatives are shown here.
[End of figure]
Revised Projections Would Not Have Changed Funding Decisions, but
Future Compact Impact Projections Will Undergo Review:
According to MCC officials, the revisions to its initial calculations
of compact impact on income and poverty for Armenia, El Salvador, and
Mozambique would not have changed MCC's decision to recommend each
compact to the MCC Board. The officials emphasized that MCC's economic
impact projections are one of many aspects of the due diligence process
that inform its compact investment decisions. A senior official stated
that MCC's compact-level analysis is separate from the decision about
whether to invest in specific projects; the two levels of analysis are
connected but play different roles in MCC's decision making. MCC
officials also told us they used conservative data and assumptions to
project compacts' impact on income, growth, and poverty. However,
future public statements would reflect MCC's corrections and revised
analyses.[Footnote 30]
In addition, MCC officials said that although lead economists
independently review the economic projections performed by others on
the transaction team for each country, calculations and assumptions
performed by the lead economists for the compacts we reviewed did not
undergo a final check for accuracy and validity. The officials stated
that such a review might have caught the errors that MCC later
identified and corrected.[Footnote 31] They further stated that, with
fewer compacts undergoing due diligence in the future, MCC will have
the staff capacity to ensure that such reviews are performed. MCC
officials told us they have begun to implement this peer review for
compacts currently under development. However, MCC has not documented
its procedures for conducting these reviews, so it is unclear what
criteria and level of detail the peer review includes.
MCC Used Varying Methods to Project Compact Impact on Income and
Poverty:
MCC used different methods to project the four compacts' impact on
income and poverty, limiting the estimates' comparability and
replicability. Although the method chosen can affect the results, MCC
has not provided preferred methods, or guidelines for selecting a
method, for these calculations.
MCC Used Varying Methods for Economic Growth Projections:
MCC used a different method to estimate impact on economic growth for
Armenia, El Salvador, and Mozambique than it used for Lesotho. As a
result, the estimates are not comparable.
* For Armenia, El Salvador, and Mozambique, MCC estimated the impact of
compact projects on aggregate income or income growth.[Footnote 32]
This method entailed summing the total benefits of individual compact
projects and adding them to the income that would have prevailed
without MCC.
* For Lesotho, MCC estimated the impact of the compact on the country's
GDP growth rate using the published results from a World Bank
simulation model. According to MCC, the World Bank's model showed that
a public and private investment of about $100 million would lead to an
increase in GDP growth rate of 3.75 percentage points. Reasoning that
its projected investment in the Lesotho compact is analogous to the
investment in the World Bank model, MCC extrapolated from the World
Bank's estimate to project that the compact, if successfully
implemented, would nearly double Lesotho's GDP growth rate.[Footnote
33]
According to MCC, its use of GDP growth impacts based on the published
World Bank model's results was appropriate, in that the model was
consistent with the scale and type of public infrastructure investments
proposed under the compact. MCC also noted that the World Bank model
provided more information than did MCC's analyses for the other
compacts.[Footnote 34] However, our analysis of the World Bank report
[Footnote 35] shows some differences between MCC's and the World Bank's
interventions. For example, whereas MCC's compact comprises only
increased public investment in infrastructure, the World Bank report's
model projects the combined effect of three elements: increase in
garment exports, growth in commercial agriculture, and increase in
public investment in infrastructure.[Footnote 36] Because the model's
three elements have different and interlinked effects, MCC's
extrapolation of the effect of one of these elements, public investment
in infrastructure, to its compact requires a number of assumptions that
cannot be validated.
Although MCC guidelines call for projecting compacts' impact on
economic growth starting from the project level, MCC has not provided
preferred methods, or guidelines for selecting a method, for estimating
compacts' impact on GDP growth rate.
MCC Used Varying Methods to Project Poverty Reduction:
MCC estimated compacts' impact on the poverty rate for Armenia, El
Salvador, and Mozambique based on the responsiveness, or elasticity, of
the poverty rate to changes in income.[Footnote 37] MCC uses several
income measures in this analysis. For Armenia, MCC estimated poverty
reduction based on its elasticity with respect to agricultural value
added to the local economy; for El Salvador, with respect to increased
annual national GDP; and for Mozambique, with respect to increased
annual regional GDP. [Footnote 38]
Each of these measures can be a valid measure of income and
corresponding elasticity of the poverty rate to income changes.
However, the types of data used to estimate poverty elasticity can
significantly affect poverty impact projections. For example, in
projecting the Mozambique compact's impact on poverty, MCC initially
used an ad hoc estimate of elasticity rather than an elasticity based
on historical income and poverty data specific to Mozambique; according
to MCC officials, the ad hoc elasticity was consistent with
conservative cross-country estimates. In a response to our questions
about its original analysis, MCC projected revised poverty impacts for
Mozambique in two ways, in both cases correcting for analytic errors
discussed above. For one revised projection, MCC used its initial
elasticity based on cross-country data. For the second projection, MCC
used an alternative elasticity calculated from Mozambique data, which
resulted in lower poverty reduction estimates.[Footnote 39] The
resulting estimates for the number of persons to be lifted out of
poverty were 56,000 versus 27,000 persons by 2015 and 43,000 versus
32,000 persons by 2025, respectively.
MCC's guidelines call for estimating compacts' impact on reducing the
poverty rate but do not discuss or prioritize various possible methods
for estimating poverty reduction.[Footnote 40] Because the methods
chosen for each compact can vary, the resulting estimates of poverty
reduction may not be comparable or replicable across countries.
MCC Used Different Methods to Estimate Beneficiaries and Is Taking
Steps to Strengthen Guidance:
MCC used two different methods to estimate the number of beneficiaries
it reported for the compacts we reviewed: (1) summing beneficiaries
from the individual projects, with adjustments for double counting and
timing of benefits, and (2) counting the intervention area's entire
population as beneficiaries.
* For Armenia and Mozambique, MCC counted the population of the areas
where the compact is being implemented as project-specific
beneficiaries and summed these project totals to produce the reported
number of beneficiaries. MCC corrected for double counting of
beneficiaries in areas where more than one project was implemented. For
Mozambique, MCC projected a gradual increase in the number of
beneficiaries as benefits accrue over time, whereas for Armenia it did
not.
* For El Salvador, MCC reported that the entire population of the
intervention area would benefit,[Footnote 41] although the project-
specific beneficiary counts did not sum to the area's population.
* For Lesotho, assuming that the compact would have economywide impact,
MCC reported the country's entire population as beneficiaries although
estimates of project beneficiaries were available.
MCC guidelines define program beneficiaries as individuals or groups
that derive economic gains from MCC investments. However, the
guidelines do not explicitly define economic gain or provide criteria
for counting beneficiaries based on the amount of accrued benefits or
the degree of exposure to compact-provided services. As a result, it is
unclear whether MCC's transaction teams would count as equal
beneficiaries a person using a compact-related benefit once and a
person using the same benefit regularly or whether MCC would apply an
equivalence scale to adjust for varying use of the benefit. For
example, for El Salvador education programs, MCC first counted as
beneficiaries all individuals entering the programs--assuming they
would experience some income increases, and later revised the method to
include only individuals completing the programs, assuming they would
experience measurable income increases. The transaction team's choice
of method for defining beneficiaries, in turn, affects the compact-
level projections of impact on income and poverty. For example, MCC's
estimate of the number of persons to be lifted out of poverty by the
compact as a whole ranges from 145,000 in the first case to 83,000 in
the second, depending on the assumptions used. (See app. IV for more
details.)
According to MCC officials, MCC is in the process of making its
guidance regarding program beneficiaries more operational by
establishing criteria for defining beneficiaries. The officials also
noted that increases in income are considered a starting point for
measuring economic gains and therefore determining the number of
compact beneficiaries.
Conclusions:
As MCC works in some of the world's poorest countries, it has taken on
the difficult challenge of projecting its compacts' likely ERR, and
economic growth and poverty effects. However, our analysis shows that
analytic errors affected the results, and the choice of analytical
method can change the results of economic projections. Without a
consistent approach and preferred methods for its due diligence
economic analyses, MCC's transaction teams and individual team members
have used different methods to identify compact results. This
heterogeneity means that different teams could reach different
conclusions about the worthiness of individual projects or compacts
based on the method chosen. MCC has refined its guidance over time, and
continues to do so. Further refinements that include formal procedures
for reviewing the results of due diligence economic analyses and
establishing a consistent approach with preferred methods--which could
be modified if required by specific country conditions--would help MCC
reduce the likelihood of errors and provide a common MCC lens for
projecting compacts' ERRs and impacts. This in turn would enhance the
reliability and comparability of the information MCC uses internally
for decision making as well as the information it provides to Congress
and the public for oversight of MCC's activities.
Recommendations for Executive Action:
To improve the reliability and comparability of its projected ERR and
economic impacts, we recommend that the CEO of MCC take the following
actions:
* Adopt and implement written procedures for a secondary independent
review of the methods and results of its economic analyses.
* Improve MCC's guidelines by identifying a consistent approach with
preferred methods for projecting compacts' impact on income and
poverty.
Agency Comments and Our Evaluation:
MCC provided comments regarding a draft of this report, which we have
reprinted, with our response, in appendix V. MCC also provided
technical clarifications, which we have incorporated as appropriate.
In commenting on a draft of this report, MCC concurred with our
recommendations and outlined related steps that it is taking--including
developing standard practices and templates, initiating an independent
peer review process, and posting its economic analyses on the Internet.
MCC commented that there is not a single cost-benefit standard practice
and that in many cases the inconsistencies we identified were
technically appropriate. We acknowledge that there is not a single
standard practice; however, we maintain that consistent analytic
approaches are needed to ensure the reliability of MCC's analyses.
According to MCC, the choice of analytic time frames was in some cases
based on the judgment of the individual lead economist rather than on
established criteria. The steps that MCC has stated it is taking--using
a default time frame for analyses and subjecting the time frame
decision to peer review--will help to enhance the comparability across
compacts of the time frame of ERR calculations. MCC also stated that it
disagreed with what it saw as our report's implication that ERRs are
disconnected from income and poverty effects. However, we assert that
ERRs do not provide information about MCC's impact on any specific
population group, including the poor. Also, ERRs are a relative measure
of benefits in relation to costs that can fluctuate owing to changes in
costs alone--and therefore cannot be considered absolute measures of
impact on income.
We are sending copies of this report to interested congressional
committees as well as the Chief Executive Officer of MCC. We will also
make copies available to others upon request. In addition, this report
will be available at no charge on the GAO Web site at [hyperlink
http://www.gao.gov].
If you or your staff have any questions about this report, please
contact David Gootnick at (202) 512-3149 or [email protected]. Contact
points for our Offices of Congressional Relations and Public Affairs
may be found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix VI.
Sincerely yours,
Signed by:
David Gootnick:
Director:
International Affairs and Trade:
[End of section]
Appendix I: Objectives, Scope, and Methodology:
At the request of the chairman of the House Committee on Foreign
Affairs, we assessed the Millennium Challenge Corporation's (MCC)
economic analyses, including its projections of the compact's economic
rate of return (ERR), changes in income and poverty, and the number of
beneficiaries. To carry out this review, we reviewed MCC compacts with
four countries: Armenia, El Salvador, Lesotho, and Mozambique. We
selected these countries based on their percentage of total compact
funding, the recentness of the compact, and geographical
representation. These four compacts make up a total of about $1.56
billion in compact assistance. When we began our work in August 2007,
this represented about 41 percent of the $3.85 billion MCC had set
aside for 13 compacts[Footnote 42] and included the most recent
compacts signed in Eurasia, Latin America, and Africa. We chose to
include two African countries in acknowledgement of MCC's focus on
Africa, which accounted for 7 of MCC's 13 compact countries and more
than half of the total compact value.
MCC's compact-level projections are influenced by the specific data
points and assumptions used in project-and activity-level analysis. We
determined that the data we used were sufficiently reliable for the
purposes of our analysis; however, within our scope of work, we did not
independently evaluate the thousands of data points and assumptions at
the project and activity levels or the sensitivity analyses used in
MCC's economic analyses and therefore are not assessing all aspects of
the validity of MCC's ERR or impact projections. We also did not assess
MCC's progress in implementing these compacts, and therefore its
progress toward achieving projected compact results.
To assess MCC's compact ERR projections, we reviewed MCC's public and
internal documents for statements of compact and project ERR. When we
began our review, MCC had published the compact and project ERRs for
only Armenia.[Footnote 43] However, for all four countries, MCC stated
ERRs in the internal investment memo. Next, we consulted MCC's guidance
for ERR analyses as well as the spreadsheets provided by MCC to support
its calculations of ERR and document its work. We reviewed MCC guidance
on calculating ERR and minimum ERRs and compared MCC's internal
documentation and spreadsheets to elements of this guidance. We
examined the spreadsheets to determine how MCC aggregated project-level
ERRs into one compact-level ERR for the four countries in our review
and identified the two approaches---summing net benefits and using
weighted averages--that MCC used. To determine the effects of MCC's
alternative approach on the compact ERR, we used MCC's cost and benefit
data to calculate the alternative ERRs. We also used these data to
calculate compact-and project-level ERRs over different time horizons
to explore the sensitivity of the ERRs to differing time horizons. To
determine how MCC approached same-sector projects in different
compacts, we studied road and water projects because these were each in
three of the four compacts. We then compared the broad approaches MCC
used in assessing the costs and benefits associated with these
projects. We interviewed MCC economists and other officials regarding
MCC's ERR analysis to further discuss MCC's approaches and clarify
aspects of MCC's analysis. We also consulted Office of Management and
Budget (OMB) guidance and GAO guidance[Footnote 44] on establishing and
implementing internal controls to inform our assessment of MCC's
processes for developing and maintaining information for the purpose of
management decision making.
To assess MCC's projections of the number of compact beneficiaries, and
changes in income and poverty, we first compiled and analyzed MCC's
public documents for statements of compact impact. These documents
included MCC's compacts, compact summaries, congressional notifications
and budget justifications, and MCC's annual reports. MCC makes all of
these documents available online.[Footnote 45] We identified statements
that fell into four categories of economic projections: impact on
national or regional income, impact on national or regional gross
domestic product (GDP) growth rate, impact on national or regional
poverty, and number of compact beneficiaries. MCC reviewed and
concurred with our compilation and summary of these statements of
compact impact. We also reviewed MCC's guidance on projecting these
economic impacts. We reviewed the spreadsheets for each compact that
MCC used to conduct and document its economic analyses. After an
initial examination of these spreadsheets, we met with MCC economists
and other officials to discuss MCC's methods and calculations for
projecting compact impact. MCC officials also provided responses to our
questions in written form. In these responses, MCC revised its initial
calculations of its impacts in Armenia, El Salvador, and Mozambique and
also presented alternative methods. We then reviewed the updated
information that MCC submitted to us and calculated the magnitude of
the difference in MCC's original statements and those supported by its
revised analyses. In the case of Lesotho, we reviewed the World Bank
Country Economic Memo regarding the economic growth model that MCC used
to estimate compact impact in Lesotho. We also reviewed World Bank
country assistance strategies and International Monetary Fund Article
IV consultation reports and Poverty Reduction Strategy Papers to
improve our contextual understanding of each country's compact program.
Finally, we consulted OMB guidance and GAO guidance on establishing and
implementing internal controls to inform our assessment of MCC's
internal processes.
We conducted this performance audit from August 2007 to June 2008, in
accordance with generally accepted government auditing standards. Those
standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.
[End of section]
Appendix II: MCC Minimum Acceptable ERRs:
MCC has issued two definitions of the minimum acceptable ERR, which MCC
refers to as the hurdle rate, for compacts and projects. In its initial
guidelines, issued in April 2005, MCC defined the hurdle rate as the
average of the country's real growth rates for the previous 3 years.
[Footnote 46] The January 2006 guidelines did not set a hurdle rate.
The November 2006 guidelines defined the hurdle rate as the greater of
(1) two times the average real growth rate of GDP for the country for
the most recent 3 years for which data are available or (2) two times
the average real growth rate of GDP for all of the MCC eligible
countries for each country for the most recent 3 years for which data
are available.[Footnote 47] The November 2006 guidelines also state
that the hurdle rate may not be higher than 15 percent.
In setting the hurdle rate for each compact that we reviewed, MCC
applied the definition of the rate from the guidelines current at the
time. For Armenia and Mozambique, MCC used its April 2005 guidance. For
El Salvador, according to MCC officials, and for Lesotho, MCC used the
definition in the November 2006 guidelines. (See table 2.)
Table 2: MCC Compact Hurdle Rates and Hurdle Rate Definition:
Compact: Armenia;
MCC hurdle rate[A]: 12.5 percent;
Hurdle rate definition: April 2005 guidance: The country's average real
growth rate for the past 3 years.
Compact: Mozambique;
MCC hurdle rate[A]: 8.76 percent;
Hurdle rate definition: April 2005 guidance: The country's average real
growth rate for the past 3 years.
Compact: El Salvador;
MCC hurdle rate[A]: 10.8 percent;
Hurdle rate definition: November 2006 guidance: Based on two times the
average real growth rate of GDP for all of the MCC eligible countries
for each country for the most recent 3 years for which data are
available.
Compact: Lesotho;
MCC hurdle rate[A]: 10.8 percent;
Hurdle rate definition: November 2006 guidance: Based on two times the
average real growth rate of GDP for all of the MCC eligible countries
for each country for the most recent 3 years for which data are
available.
Source: MCC documents.
[A] For Armenia, Mozambique, and Lesotho, the hurdle rate was stated in
the investment memo. The investment memo for El Salvador did not state
the hurdle rate, but MCC officials reported that a hurdle rate of 10.8
percent, based on the November 2006 definition, was applied.
[End of table]
In two cases, the ERRs that MCC initially calculated did not meet the
compact hurdle rates. MCC originally calculated the El Salvador
Community Infrastructure Project ERR as slightly below the hurdle rate
at 10.4 percent over 25 years. MCC's investment memo for Lesotho stated
that the ERR for the Rural Water Project was under 6 percent.
Subsequent revisions to the Lesotho analysis reduced the projected ERR
to less than 1 percent. The Lesotho investment memo discusses the 6
percent ERR but notes a different view within the transaction team that
other benefits are not captured by the economic analyses.
[End of section]
Appendix III: Compact ERRs:
We assessed the sensitivity of MCC's compact-level ERRs to the use of
varying time frames and varying methods for calculating the compact-
level ERR. We found in each case that the variance does affect the
results, but does not change the ERRs to below the applicable hurdle
rate.
Time Frames:
MCC used different time frames to calculate and report compact ERRs.
* For Armenia, MCC calculated and reported a 20-year time frame for the
compact and both of its projects.
* For El Salvador, MCC calculated a 25-year ERR for most projects and
for some projects reported the ERR as a 25-year ERR in its investment
memo. MCC officials explained that their El Salvador country
counterparts originally performed the analysis over 25 years and that
MCC's lead economist determined this to be a reasonable approach.
* For Lesotho, MCC used a 20-year ERR for all but one ERR calculation
but did not state the time frame of the calculations in the investment
memo.
* For Mozambique, MCC used a 24-year ERR for most compact projects.
According to MCC officials, MCC intended to present a 25-year ERR, but
a delay in the beginning of compact implementation pushed the compact's
time horizon 1 year into the future and reduced the time frame for the
analysis to 24 years. The Mozambique investment memo does not state the
number of years used for the time frame of the ERR calculation.
Table 3 summarizes our comparison of the compact hurdle rates with the
ERRs that MCC reported and our calculations of the 20-year ERRs. Our
analysis shows that applying a 20-year time frame in place of varying
time frames does not significantly affect the results of the ERR
calculations by lowering it below the hurdle rate.
Table 3: Comparison of MCC Compact ERRs Stated in Investment Memo with
ERRs Calculated Using a 20-Year Time Frame:
Armenia:
Hurdle rate: 12.5 percent;
ERR reported in investment memo[A]: 25 percent over 20 years;
GAO calculation: 20-year ERR[B]: 27.4 percent.
El Salvador[C]:
Hurdle rate: 10.8 percent;
ERR reported in investment memo[A]: 21 percent over 25 years;
GAO calculation: 20-year ERR[B]: 17.0 percent[D].
Lesotho:
Hurdle rate: 10.8 percent;
ERR reported in investment memo[A]: 16.3 percent, period of years not
specified;
GAO calculation: 20-year ERR[B]: 15.9 percent[E].
Mozambique:
Hurdle rate: 8.76 percent;
ERR reported in investment memo[A]: 19.6 percent, period of years not
specified;
GAO calculation: 20-year ERR[B]: 17.2 percent.
Source: GAO analysis of MCC economic analyses.
[A] In some cases, MCC's underlying spreadsheets stated different ERRs
than those reported in the investment memo. However; for Armenia, the
revision to a 27.4 percent ERR occurred after the investment memo. We
based our calculations on the spreadsheet calculations of the ERRs: for
Armenia, 27.4 percent over 20 years; for El Salvador, 20.6 percent over
25 years; for Lesotho, 16.4 percent over 20 years; and for Mozambique,
18.7 percent over 24 years.
[B] These alternative ERRs use the method used by MCC for the original
ERR--that is, weighted averages for El Salvador and Lesotho and sums of
net benefits for Armenia and Mozambique.
[C] MCC provided updated spreadsheets for its education and water and
sanitation projects that reduced some expected benefits and incorrectly
entered some compact administration costs for another project. Using
the weighted average method originally used by MCC, the ERR with these
revisions would be 16.3 percent at 20 years.
[D] The formula used to calculate the ERR for the Productive
Development project in El Salvador returns an error when used to
calculate the ERR for 20 years. However, since the total net benefits
for the project at 20 years are negative, the ERR also would be
negative. We used a zero ERR over 20 years in place of a negative ERR
for this project in calculating the compact ERR. The total net benefits
and ERR for the Productive Development project become positive at 21
years.
[E] MCC originally calculated the Lesotho compact-level ERR using a
weighted average of ERRs, some of which did not match the ERRs in its
underlying spreadsheet calculations and the investment memo. We have
used the ERRs in these underlying spreadsheets to calculate the 20-year
ERR. MCC also originally used a 30-year ERR for the Metolong Dam
project to calculate the 20-year compact-level ERR. We have used the 20-
year ERR for the Metolong Dam for this 20-year compact ERR calculation.
[End of table]
Methods:
Figure 8 illustrates the two different methods--summing net benefits
and calculating a weighted average--that MCC used to determine a
compact-level ERR.
Figure 8: MCC's Alternative Methods for Calculating Compact ERR:
[See PDF for image]
This figure is an illustration of MCC's alternative methods for
calculating Compact ERR, as follows:
Alternative 1: Sum of net benefits across all projects, for each year:
Net benefits project A, plus Net benefits project B, plus Net benefits
project C, equal Compact net benefits; Calculate compact ERR, yields
Compact ERR.
Alternative 2: Weighted average of project ERRs:
ERR Project A, times Cost share A, equals Weighted project ERR A;
ERR Project B, times Cost share B, equals Weighted project ERR B;
ERR Project C, times Cost share C, equals Weighted project ERR C;
Combined to calculate overall weighted average, yields Compact ERR.
Source: GAO synthesis of MCC information.
[End of figure]
We compared the results of the compact ERR calculation using different
methods and determined that the choice of method affects the results,
but does not reduce the ERR below the hurdle rate. See table 4 for
details of our analysis.
Table 4: Compact ERR Using Alternative Methods:
Country: Armenia;
ERR reported in investment memo: 25 percent over 20 years;
ERR from underlying spreadsheets[A]: 27.4 percent over 20 years;
ERR using alternative method: 25.6 percent over 20 years;
Difference: - 1.8.
Country: El Salvador;
ERR reported in investment memo: 21 percent over 25 years;
ERR from underlying spreadsheets[A]: 20.6 percent over 25 years;
ERR using alternative method: 19.6 percent over 25 years;
Difference: -1.
Country: Lesotho;
ERR reported in investment memo: 16.3 percent, period of years not
specified;
ERR from underlying spreadsheets[A]: 16.4 percent over 20 years;
ERR using alternative method: 15.4 percent over 20 years;
Difference: -1.
Country: Mozambique;
ERR reported in investment memo: 19.6 percent, period of years not
specified;
ERR from underlying spreadsheets[A]: 18.7 percent over 24 years;
ERR using alternative method: 17.3 percent over 24 years;
Difference: -1.4.
Source: GAO analysis of MCC economic analyses:
[A] The ERR in MCC's underlying spreadsheets for Armenia, Lesotho, and
Mozambique was different from that stated in its investment memos.
However; for Armenia, the revision to a 27.4 percent ERR occurred after
the investment memo. In order to determine the ERRs from an alternative
method, we needed to use these spreadsheets; therefore they are the
appropriate comparison for determining the magnitude of the change in
results from using the alternative method. El Salvador's reported ERR
was rounded in the investment memo.
[End of table]
[End of section]
Appendix IV: Impact of Alternative Beneficiary Counts on El Salvador
Compact Impact Projections:
During the course of our engagement, MCC changed its method for
determining the beneficiaries and benefits of education projects in El
Salvador. In its initial calculations for both projects, MCC assumed
that all students entering the programs would experience an increase in
income. In its revised calculations for the formal education project,
MCC assumed that all nongraduates would have zero income increases--an
underestimate of the impact--because of lack of data. For the informal
education project, MCC estimated an income increase of more than 200
percent for those who obtain employment, based on a study conducted by
the training institute in El Salvador. However, MCC assumed that many
entering students would not complete the program or obtain employment
and their income would therefore not increase. (See table 5.)
Table 5: MCC's Compact-Level Impact Estimates for El Salvador, with
Alternative Assumptions for Estimating Beneficiaries of Education
Projects:
Results of compact-level analysis:
Incomes in the region: 5 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income: 18
percent increase;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: 10 percent increase.
Incomes in the region: 10 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income: 26
percent increase;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: 13 percent increase.
Annual per capita income of beneficiaries: 5 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income:
$123 increase;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: $73 increase.
Annual per capita income of beneficiaries: 10 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income:
$189 increase;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: $97 increase.
Number of persons for whom poverty is alleviated: 10 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income:
145,000 persons;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: 83,000 persons.
Poverty rate in the Northern Zone: 5 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income: 10
percentage point decrease;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: 6 percentage point decrease.
Poverty rate in the Northern Zone: 10 years;
Assumption for estimating beneficiaries of education projects[A]:
Individuals entering the program will experience increase in income: 15
percentage point decrease;
Assumption for estimating beneficiaries of education projects[A]:
Individuals completing the program[B] will experience increase in
income[C]: 8 percentage point decrease.
Source: GAO analysis of MCC data.
Note: These figures also reflect MCC's corrections of formula errors.
[A] MCC's compact with El Salvador comprises a formal education project
and an informal education project. We consider these two projects
jointly for the purposes of this summary table.
[B] For the informal education project, MCC estimated the number of
individuals who complete the program and obtain employment.
[C] The revised projections also reflect MCC changes to (1) the number
of beneficiaries of the productive development project, and (2)
projected income increases resulting from the water and sanitation
project.
[End of table]
[End of section]
Appendix V: Comments from the Millennium Challenge Corporation:
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Millennium Challenge Corporation:
Reducing Poverty Through Growth:
875 Fifteenth Street NW:
Washington, DC 20005-2221:
(202)521-3600:
(202521-3700 (Fax):
[hyperlink, http://www.mcc.gov]:
May 23, 2008
Mr. David B. Gootnick:
Director, International Affairs and Trade:
U.S. Government Accountability Office:
441 G Street, NW:
Washington, DC 20001:
Dear Mr. Gootnick:
Thank you for the opportunity to respond to GAO's draft report, MCC:
Independent Reviews and Consistent Approaches Will Strengthen
Projections of Program Impact.
MCC appreciates GAO's recognition of the ground-breaking nature of
MCC's projections of program impact, reflected in the report's
conclusion: "As it works in some of the world's poorest countries, MCC
has taken on the difficult challenge of projecting its compacts' likely
ERR, and economic growth and poverty effects." MCC concurs with GAO's
overall recommendations for enhanced guidance and a secondary
independent review of economic analyses. Indeed, as GAO notes, MCC has
already "taken steps to standardize elements of its economic analyses
and centralize its records management" in addition to establishing an
independent peer review process.
Independent Reviews:
Economic analysis, conducted for the purpose of informing MCC
investment decisions, already undergoes a series of reviews as part of
our standard practice. Much of the initial analysis is performed by
professional counterparts in our partner countries and by private
consultants hired by them or by MCC. In every case, MCC economists
represent an independent review of that preliminary analysis.
MCC supports an additional layer of review and, as the report notes,
has already established an independent peer review process. This
process, which is managed by the Chief Economist, provides a review of
the models, formulae and parameters used to estimate the expected
impact of programs. MCC has already used the new process to review
forecasts for one compact program currently under consideration, and
has begun the process for a second country. [See comment 1]
MCC has taken the unprecedented step of making our economic analysis
accessible on our public website, opening our work to the broader
scrutiny of interested NGOs, academics, and private analysts, which we
believe will generate useful feedback further strengthening our
analysis. MCC has posted the economic models and resulting impact
estimates for nine countries, along with explanatory descriptions and
documentation. We will post the analysis for all other compacts before
the end of this fiscal year. Independent observers, including senior
officials at Center for Global Development and Bread for the World
Institute, have lauded this initiative as setting a new standard for
transparency in government. [See comment 1]
Consistency in Technical Approaches:
The report recognizes that "MCC has refined its guidance over time, and
continues to do so." MCC is still a new and evolving agency, and we are
continually refining our standard procedures, for economic analysis and
in other areas. MCC's rigorous, transparent use of economic analysis to
estimate the cost-effectiveness of potential interventions is
exceptional within the international development community and U.S.
foreign assistance agencies.
We endorse GAO's conclusion that "establishing a consistent approach
with preferred methods - which could be modified if required by
specific conditions - would help MCC to reduce the likelihood of
errors." MCC has an effort underway to develop standard practices and
baseline templates for a number of core sectors prevalent in past
proposals, including roads, water services & sanitation, and
agriculture & irrigation. We are also considering such templates for
education, land reform, and micromicrofinance. [See comment 1]
Placing Consistency in Context:
GAO analyzed more than 20 projects' economic models, most of which
comprise numerous spreadsheets incorporating data from reams of
academic research, country sources, and professional judgment. The
report cited a number of "inconsistencies" in MCC practices, and GAO
identified some as minor, with trivial effect on our estimates. In many
cases, these "inconsistencies" are technically appropriate variations
of standard practices that can be explained by the specific country
context. [Footnote 48]
There is simply no single cost-benefit "standard practice," because
many of the technical details are by necessity context specific. In
some cases, the data available allow reasonable estimation using
different models, and MCC needs the flexibility, as the GAO report
notes, to apply professional judgment as to when the attainment of
uniformity in practice is either not cost-effective or will yield
perverse results. For example, when existing data allow the use of a
different model and the collection of new data needed for the standard
model would require both significant time and cost, MCC might find the
alternative model acceptable. Similarly, when application of a standard
time horizon does not appropriately reflect a project whose useful life
is either shorter or longer, MCC will prefer to "deviate" [See comment
2] from consistent practice rather than inaccurately estimate the
economic impact of the proposed investment. In all such cases, however,
MCC will fully subject these decisions to peer review and document them
for public viewing and any subsequent external assessment. MCC will
also use sensitivity analysis to explore the implications of
alternative assumptions or parameters. [See comment 1]
MCC's ERRs Estimate the Impact on Local Incomes:
GAO differentiates between ERRS and what the report refers to as MCC's
"projections of compacts' impact on income and poverty," which implies
that our ERRS are an aggregate measure of impact disconnected from
welfare levels experienced by the low-income residents in our partner
countries. This characterization is based on a common misunderstanding
about how MCC calculates ERRs and what those numbers represent.
MCC follows standard cost-benefit practices in most ways, but includes
only incremental increases in incomes earned by households and domestic
firms. By excluding other possible benefit streams that do not affect
domestic incomes, MCC's ERRs represent a direct estimation of the
magnitude by which local incomes will rise as a result of the MCC
program. When MCC reports high returns for our projects, these
estimates do not reflect a broad impersonal measure of economic
activity, but rather a much more tangible estimate of the project's
effect on people's lives. [See comment 3]
Conclusion:
MCC uses economic analysis and other technical work to direct taxpayer
funds to investments that will generate significant benefits in well-
governed developing countries. MCC's performance of cost-benefit
assessments for virtually all proposed investments, posted on our
public website, is unprecedented. This assessment represents a critical
tool for accountability. For every proposed project, MCC assesses
whether it is a wise investment of American taxpayer funds that will
generate ample returns for our intended beneficiaries.
It is instructive to compare MCC practices to those found in other
foreign assistance agencies around the world. Most other aid agencies
rarely, if ever, subject their projects to such technical scrutiny. MCC
does it as a matter of practice, and our practices have generally been
both reliable and transparent.
Both our use of economic analysis as a tool for decision-making and our
openness to public review are critical parts of the MCC model. MCC has
established a new standard for project efficacy and transparency, and
we will continue to review and enhance our practices to meet the high
standards we have set for ourselves.
Sincerely,
Signed by:
Rodney G. Bent:
Deputy CEO:
Millennium Challenge Corporation:
[End of letter]
The following are GAO's comments from the Millennium Challenge
Corporation letter, dated May 23, 2008.
GAO Comments:
1. We recognize MCC's constructive responses to the issues identified
in the course of our audit--including posting economic analyses on the
Internet, developing standard practices and templates, and initiating
an independent peer review process.
2. We acknowledge that there is not a single standard practice;
however, we maintain that consistent analytic approaches are needed to
ensure the reliability and comparability of MCC's analyses. According
to MCC, the choice of analytic time frames was in some cases based on
the judgment of the individual lead economist rather than on
established criteria. The steps that MCC has stated it is taking--using
a default time frame for analyses and subjecting the time frame
decision to peer review--will help to enhance the comparability of its
of ERR calculations across compacts.
3. We disagree that MCC's ERRs estimate the project's effect on
people's lives. ERRs do not provide information about MCC's impact on
any specific population group, including the poor. ERRs also are a
relative measure of benefits in relation to costs and can fluctuate
owing to changes in costs alone. For example, if costs increase and the
income benefits remain the same, the ERR would decrease. Therefore,
ERRs cannot be considered an absolute measure of income benefits.
[End of section]
Appendix VI: GAO Contact and Staff Acknowledgments:
GAO Contact:
David B. Gootnick, Director, 202-512-3149 or [email protected]:
Staff Acknowledgments:
In addition to the person named above, Emil Friberg, Jr. (Assistant
Director), Todd M. Anderson, Gergana Danailova-Trainor, Reid Lowe,
Michael Simon, and Seyda Wentworth made key contributions to this
report. Also, C. Etana Finkler, Ernie Jackson, and Tom McCool provided
technical assistance.
[End of section]
Footnotes:
[1] An MCC compact is an agreement between the U.S. government, acting
through MCC, and the government of a country eligible for MCC
assistance. MCC's authorizing legislation, Public Law 108-199, limits
compact duration to no more than 5 years.
[2] The due diligence review also includes an evaluation of countries'
consultative processes used to develop the proposal, donor
coordination, and environmental and social impact, among other things.
[3] Economic rate of return is the expected annual average return to
the countries' firms, individuals, or sectors for each dollar that MCC
spends on the project. For example, if MCC spent $100,000 on a project
in year 1 and expected that the project would yield net benefits of
$120,000 in year 2, the project's ERR for year 1 would be (120,000-
100,000)/100,000=0.2, or 20 percent.
[4] Transaction teams comprise MCC staff, personnel from other U.S.
agencies, and consultants.
[5] The investment committee consists of MCC's Chief Executive Officer
(CEO), vice presidents, and other senior officials. The committee
reviews the memo and decides whether to recommend proceeding to compact
negotiations.
[6] GAO, Millennium Challenge Corporation: Vanuatu Compact Overstates
Projected Program Impact, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-07-909] (Washington, D.C.: July, 11, 2007).
[7] An internal control is an integral component of an organization's
management that provides reasonable assurance that the agency is
achieving: effectiveness and efficiency of operations, reliability of
financial reporting, and compliance with applicable laws and
regulations. For OMB guidance, see OMB circular A-123. For GAO
guidance, see Standards for Internal Control in the Federal Government,
[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1]
(Washington, D.C.: November 1999).
[8] We could not review MCC's calculations for the Lesotho compact,
because MCC based its projections on a previous World Bank economic
growth model rather than its own calculations.
[9] MCC first issued guidelines for the compact development process in
April 2005. MCC issued revised guidelines in January 2006 and, most
recently, in November 2006.
[10] MCC also must report to Congress prior to obligating funds.
[11] The Secretary of State serves as MCC Board chair, and the
Secretary of the Treasury serves as vice-chair. Other board members are
the U.S. Trade Representative, the Administrator of the U.S. Agency for
International Development (USAID), the CEO of MCC, and up to four
Senate-confirmed public members who are appointed by the President from
lists of individuals submitted by congressional leadership.
[12] We recently reported on MCC's progress in developing and
implementing compacts. See GAO, Millennium Challenge Corporation:
Analysis of Compact Development and Future Obligations and Current
Disbursements of Compact Assistance, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO-08-577R], (Washington, D.C.: Apr. 11, 2008).
[13] MCC stated that these analyses do not capture all aspects of MCC's
potential impact, such as the effect of government reforms or policy
changes.
[14] MCC's guidelines discuss this type of economic analysis under the
heading "Beneficiary Analysis." For the purposes of our report, we
refer to this type of analysis as projections of impact on income and
poverty or simply impact projections.
[15] In El Salvador, the water project is a component of the Community
Infrastructure project.
[16] This amount includes both obligations and commitments. As of March
2008, MCC had provided a total of $5.5 billion for all 16 countries
with signed compacts. When we began our work in August 2007, MCC had
signed 13 compacts totaling $3.85 billion.
[17] In some cases, the ERRs we refer to were calculated by MCC at the
activity level.
[18] The four compacts' minimum ERRs are 12.5 percent for Armenia, 10.8
percent for El Salvador, 10.8 percent for Lesotho, and 8.76 percent for
Mozambique. MCC set the minimum ERR for each compact applying
definitions contained in the guidelines current at the time. The
initial ERRs for all but two projects--the El Salvador Community
Infrastructure project and the Lesotho Rural Water project--met the
respective minimums for each country. See app. II for further
discussion of MCC's minimum ERR.
[19] Specifically, the Rio Lurio-Metoro Road segment ERR is 7.2
percent, the Namialo-Rio Lurio ERR is 5.9 percent, and the Nampula-Rio
Ligonha ERR is 6.3 percent.
[20] MCC's April 2005 guidance stated that project ERRs should be
defined "over the natural life of that component."
[21] According to MCC, the urban water analysis was revised after the
investment memo to phase in costs.
[22] Salvage value is the estimated value of an asset at the end of its
useful life.
[23] MCC used an alternative method for calculating road project
benefits in El Salvador. Because the existing road was nearly
impassable, MCC thought its preferred method based on existing traffic
estimates was a poor predictor of the project's impact. MCC instead
substituted a measure of increases in land values as a proxy for income
growth. MCC's transaction team reported both scenarios to the
investment committee--the ERR would be 24 percent using a land value-
based measure and between 13.8 percent and 14.7 percent using the
traffic count method. The ERR in either method exceeds MCC's minimum
ERR of 10.8 percent.
[24] MCC's April 2005 guidance noted that overall compact ERRs may be
calculated by using cost-weighted averages of project components,
combining cost and benefit flows, or another approach--depending on the
facts and circumstances of the compact. The January 2006 guidance and
the November 2006 guidance do not define a procedure for calculating
compact-level ERR.
[25] See OMB Circular A-123, "Management's Responsibility for Internal
Control," revised Dec. 21, 2004; and GAO, Standards for Internal
Control in the Federal Government, [hyperlink, http://www.gao.gov/cgi-
bin/getrpt?GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999).
[26] MCC's November 2006 Guidelines for Economic and Beneficiary
Analysis states that poverty may be defined according to country-
specific definitions, such as the official poverty line, or according
to international standards such as the World Bank's extreme poverty
definition of $1.08 per capita per day in purchasing power parity, or
$2 per day. In making its public statements of poverty impact in the
countries we examined, MCC used country-specific definitions.
[27] We reviewed MCC's impact analysis spreadsheets for these three
compacts. For Lesotho, although MCC economists conducted ERR analyses
for activities and projects under the compact, MCC ultimately based its
compact-level projections on a previous World Bank economic growth
model. As such, we were not able to assess MCC's compact-level impact
projections for Lesotho.
[28] In its initial projections of the impact of increased incomes on
poverty in Mozambique, MCC used two types of income estimates - GDP and
GDP per capita - to compare with-and without-project scenarios.
[29] Poverty elasticity measures the extent to which economic growth
reduces poverty by estimating the percentage change in poverty caused
by a 1 percent change in income.
[30] As of May 2008, MCC has posted spreadsheets on its Web site
showing calculations of ERR for El Salvador projects. These
spreadsheets do not include MCC's compact ERR calculation or
calculations of compact impact on income and poverty.
[31] According to OMB and GAO guidelines, an effective control
environment for data processing may include edit checks. OMB Circular A-
123 calls on federal agencies to establish management controls to
ensure that reliable and timely information is maintained for decision
making. See OMB Circular A-123, "Management's Responsibility for
Internal Control," revised Dec. 21, 2004. GAO's Standards for Internal
Control in the Federal Government cites edit checks as an example of a
control activity used in information processing [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1] (Washington,
D.C.: November 1999).
[32] Income growth is defined as the percentage change of income from
year to year.
[33] Specifically, MCC projected that the compact could nearly double
GDP growth by the end of the 5-year compact implementation period
(using the International Monetary Fund's baseline of 2.6 percent). MCC
also stated that the acceleration of GDP growth was expected to
continue, propelling growth toward 7 percent per annum within 5 years
after compact completion.
[34] The World Bank model tracks flows of all transactions, from sector
to sector, within the economy. In most countries, the economists could
not use such analysis either because no well-calibrated model exists or
because the MCC package is not easily incorporated into existing
models.
[35] The World Bank, Lesotho Country Economic Memorandum: Growth and
Employment Options Study, Report No. 35359-LS, Apr. 21, 2005.
[36] An alternative scenario developed by the World Bank involves an
approximate $50 million investment in infrastructure by the government
of Lesotho. MCC did not choose to extrapolate the results of this
scenario.
[37] MCC did not publicly report an estimate of poverty impact for
Lesotho.
[38] Poverty analysis generally involves measuring economic welfare of
individuals and constructing poverty lines to determine the number of
people deemed poor--the poverty rate--and the depth of poverty--the
poverty gap. Viable measures of economic welfare include, for example,
income per capita, consumption per some "standardized" adult, food
share of total expenditures, and nutritional indicators. For each
compact, MCC measured poverty by the country's headcount poverty rate-
-that is, the number of people with incomes below the poverty line.
[39] Cross-country data are data drawn from multiple countries. MCC's
memorandum additionally noted the possibility of using a mixed approach
that assumes a nonlinear relationship between poverty and income, using
different values for specific ranges of poverty.
[40] MCC's November 2006 guidelines state that impact on poverty should
be measured in terms of both the poverty rate and the poverty gap.
However, MCC did not estimate changes in the poverty gap for any of the
four compacts we reviewed. According to the guidelines, "the poverty
gap is calculated as the sum of money required to bring all poor
households up to the poverty line, and the effect of an MCC investment
on the poverty gap would reflect incremental income to poor households
in aggregate. The poverty rate, in contrast, would not reflect, for
example, significant improvements in income levels for households
remaining below the poverty line." As a result of estimating the impact
on the poverty rate but not on the poverty gap, MCC estimates its
compacts' impacts on the number of beneficiaries lifted out of poverty
but does not evaluate potential impact on the severity of
beneficiaries' poverty.
[41] In MCC's public documents, MCC cited the entire population of the
Northern Zone of El Salvador (850,000 people) as the total number of
compact beneficiaries. In addition, in its 2006 annual report, MCC
characterizes these beneficiaries as poor. However, according to MCC's
investment memo, 450,000 (53 percent) of these people are poor. The
investment memo further defines poverty in the context of its
Productive Development Project in two ways: (1) relative poverty as
defined by El Salvador's General Directorate for Statistics and Census
and (2) more than half the population of the Northern Zone living on
less than $2 dollar per day and more than 25 percent living on less
than $1 per day. However, the investment memo does not specify the
poverty definition used to generate the estimate of 450,000 poor
people.
[42] As of March 2008, MCC had 16 signed compacts.
[43] These internal documents are restricted from public dissemination
based on MCC policy, but MCC made them available to us for analysis. As
of April 2008, MCC had posted its spreadsheets showing calculations of
project-level ERR for El Salvador projects on its Web site, see
[hyperlink, http://www.mcc.gov/programs/err/]. As of April 2008, MCC
has not released the supporting ERR spreadsheets for Armenia, Lesotho,
and Mozambique, but MCC officials have told us they plan to release all
such spreadsheets.
[44] See OMB circular A-123. For GAO guidance, see Standards for
Internal Control in the Federal Government, [hyperlink,
http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1] (Washington,
D.C.: November 1999).
[45] See [hyperlink, http://www.mcc.gov/about/reports/] and [hyperlink,
http://www.mcc.gov/countries/index.php].
[46] MCC's April 2005 guidelines noted that the hurdle rate definition
will be revised based on MCC's subsequent experience and data on the
experiences of developing countries in general. Although other
development agencies, such as the World Bank and the Asian Development
Bank, link their hurdle rate to the notion of opportunity cost of
capital rather than GDP growth rate, in practice the results are
similar, in that the resulting hurdle rate generally remains between 10
percent and 12 percent. According to MCC officials, they considered
other measures but decided on a method that was tied to economic growth
and did not require extensive and debatable data analysis--MCC's
current hurdle rates are generally between 10 percent and 15 percent.
[47] MCC's November 2006 guidelines note that the hurdle rates will be
set once per year, using data available in the September edition of the
International Monetary Fund's World Economic Outlook Database for the 3
previous years.
[48] In one example, local counterparts developed initial calculations
for a road project using a 25-year time horizon instead of the 20-year
horizon used by MCC in most other countries. MCC's economist reviewed
the analysis and found the difference unimportant to the outcome. The
economist decided it was better to accept the high-quality work done
locally than to insist on revisions solely for the sake of attaining
uniformity of practice.
[End of section]
GAO's Mission:
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people.
GAO examines the use of public funds; evaluates federal programs and
policies; and provides analyses, recommendations, and other assistance
to help Congress make informed oversight, policy, and funding
decisions. GAO's commitment to good government is reflected in its core
values of accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony:
The fastest and easiest way to obtain copies of GAO documents at no
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
weekday, GAO posts newly released reports, testimony, and
correspondence on its Web site. To have GAO e-mail you a list of newly
posted products every afternoon, go to [hyperlink, http://www.gao.gov]
and select "E-mail Updates."
Order by Mail or Phone:
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent.
Orders should be sent to:
U.S. Government Accountability Office:
441 G Street NW, Room LM:
Washington, D.C. 20548:
To order by Phone:
Voice: (202) 512-6000:
TDD: (202) 512-2537:
Fax: (202) 512-6061:
To Report Fraud, Waste, and Abuse in Federal Programs:
Contact:
Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
E-mail: [email protected]:
Automated answering system: (800) 424-5454 or (202) 512-7470:
Congressional Relations:
Ralph Dawn, Managing Director, [email protected]:
(202) 512-4400:
U.S. Government Accountability Office:
441 G Street NW, Room 7125:
Washington, D.C. 20548:
Public Affairs:
Chuck Young, Managing Director, [email protected]:
(202) 512-4800:
U.S. Government Accountability Office:
441 G Street NW, Room 7149:
Washington, D.C. 20548:
*** End of document. ***