    Millennium Challenge Corporation: Independent Reviews and
    Consistent Approaches Will Strengthen Projections of Program
    Impact (17-JUN-08, GAO-08-730).

    In January 2004, Congress established the Millennium Challenge
    Corporation (MCC) for foreign assistance. Eligible countries
    submit compact proposals for MCC funding for projects aimed at
    reducing poverty through economic growth. To assess the proposed
    compacts' likely impact, MCC performs economic analyses
    estimating the compacts' economic rate of return (ERR) and
    effects on income and poverty as well as the number of compact
    beneficiaries. MCC uses these analyses to inform its decisions to
    fund proposed compacts and to inform Congress and the public
    about its progress in achieving its mission of poverty reduction
    through economic growth. GAO was asked to examine MCC's
    projections of (1) ERR and (2) compacts' impact on income and
    poverty as well as numbers of beneficiaries. GAO reviewed MCC's
    stated impacts and analyses for four MCC compacts that
    represented 41 percent of MCC's compact assistance and met with
    MCC officials.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-08-730
        ACCNO:   A82389
      TITLE:     Millennium Challenge Corporation: Independent Reviews and
    Consistent Approaches Will Strengthen Projections of Program
    Impact
         DATE:   06/17/2008
      SUBJECT:   Agency missions
    	     Beneficiaries
    	     Cost analysis
    	     Disadvantaged persons
    	     Economic analysis
    	     Economic growth
    	     Economic policies
    	     Economically depressed areas
    	     Eligibility determinations
    	     Federal aid to foreign countries
    	     Foreign corporations
    	     Foreign economic assistance
    	     Impacted areas
    	     Internal controls
    	     International agreements
    	     International economic relations
    	     International organizations
    	     International relations
    	     Program evaluation
    	     Records management
    	     Reporting requirements
    	     Standards
    	     Foreign countries
    	     Armenia
    	     El Salvador
    	     Lesotho
    	     Mozambique

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-08-730

    This text file was formatted by the U.S. Government Accountability
    Office (GAO) to be accessible to users with visual impairments, as part
    of a longer term project to improve GAO products' accessibility. Every
    attempt has been made to maintain the structural and data integrity of
    the original printed product. Accessibility features, such as text
    descriptions of tables, consecutively numbered footnotes placed at the
    end of the file, and the text of agency comment letters, are provided
    but may not exactly duplicate the presentation or format of the printed
    version. The portable document format (PDF) file is an exact electronic
    replica of the printed version. We welcome your feedback. Please E-mail
    your comments regarding the contents or accessibility features of this
    document to Webmaster@gao.gov.

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed
    in its entirety without further permission from GAO. Because this work
    may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to reproduce this
    material separately.

    Report to the Chairman, Committee on Foreign Affairs, House of
    Representatives:

    United States Government Accountability Office:
    GAO:

    June 2008:

    Millennium Challenge Corporation:

    Independent Reviews and Consistent Approaches Will Strengthen
    Projections of Program Impact:

    GAO-08-730:

    GAO Highlights:

    Highlights of GAO-08-730, a report to the Chairman, Committee on
    Foreign Affairs, House of Representatives.

    Why GAO Did This Study:

    In January 2004, Congress established the Millennium Challenge
    Corporation (MCC) for foreign assistance. Eligible countries submit
    compact proposals for MCC funding for projects aimed at reducing
    poverty through economic growth. To assess the proposed compacts�
    likely impact, MCC performs economic analyses estimating the compacts�
    economic rate of return (ERR) and effects on income and poverty as well
    as the number of compact beneficiaries. MCC uses these analyses to
    inform its decisions to fund proposed compacts and to inform Congress
    and the public about its progress in achieving its mission of poverty
    reduction through economic growth. GAO was asked to examine MCC�s
    projections of (1) ERR and (2) compacts� impact on income and poverty
    as well as numbers of beneficiaries. GAO reviewed MCC�s stated impacts
    and analyses for four MCC compacts that represented 41 percent of MCC�s
    compact assistance and met with MCC officials.

    What GAO Found:

    MCC used different time frames and methods to calculate ERRs for its
    compacts with Armenia, El Salvador, Lesotho, and Mozambique. In
    calculating ERR for 20 projects within the compacts, MCC used a 20-year
    time frame for 9 projects and used different time frames for the other
    11. In 2 of the 11 projects, using a 20-year time frame, as MCC used
    for similar projects, would reduce the ERR below the level MCC set as
    the minimum acceptable ERR. At the compact level, MCC�s use of varying
    time frames did not affect the ERRs significantly. MCC used varying
    methods to account for the costs of same-sector projects, although its
    approaches to determining project benefits were generally similar. MCC
    also used two different methods to calculate compact-level ERR;
    however, the choice of method did not reduce it below the minimum ERR.
    In three of the four compacts that we reviewed, MCC did not retain
    documentation of the economic analyses used to support the investment
    decision, but continued to modify the analyses. MCC has recently begun
    to standardize elements of its economic analyses and centralize its
    records management.

    MCC identified and corrected analytic errors in its projections of
    impact on income and poverty. MCC also used varying methods to project
    these impacts.

    * In responding to GAO�s questions about its published projections of
    impact on income and poverty, MCC identified analytic errors for three
    of the four compacts and, in correcting these errors, generally lowered
    the projected impacts. Correcting these errors raised one projection by
    5 percent but reduced others by 3 percent to 96 percent. According to
    MCC officials, the revised projections would not have affected MCC�s
    decision to recommend signing the compacts. The officials noted that,
    in the future, compact impact projections will undergo peer review.
    However, MCC has not documented procedures for these reviews.

    * MCC used varying methods for its projections of impact on income and
    poverty, limiting comparability and replicability across compacts. To
    project impact on income for Armenia, El Salvador, and Mozambique, MCC
    estimated the compacts� impact by summing the total benefits of
    individual compact projects and adding them to the income that would
    have prevailed without MCC. However, for Lesotho, MCC estimated the
    impact on income based on the published results of a World Bank model
    based on elements different from those of the MCC compact. In response
    to our questions, MCC revised its initial estimate of the effect of
    income growth on poverty for Mozambique by presenting two estimates,
    based on Mozambique-specific and cross-country data, respectively.
    Although a number of methods for projecting poverty impact are valid,
    the method chosen can affect the results, and MCC�s guidelines do not
    identify preferred methods for these calculations. MCC also used
    varying methods to estimate numbers of beneficiaries for the compacts
    and has not provided specific criteria for defining beneficiaries;
    however, MCC officials reported they are taking steps to provide more
    detailed guidance.

    What GAO Recommends:

    GAO recommends that the Chief Executive Officer of MCC (1) adopt and
    implement written procedures for a secondary independent review of its
    economic analyses and (2) improve MCC�s guidelines by identifying a
    consistent approach with preferred methods for projecting compacts�
    impact on income and poverty. MCC concurred with GAO�s recommendations.

    To view the full product, including the scope and methodology, click on
    [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-08-730]. For more
    information, contact David Gootnick at (202) 512-3149 or
    gootnickd@gao.gov.

    [End of section]

    Contents:

    Letter:

    Results in Brief:

    Background:

    MCC Used Different Time Frames and Methods to Calculate ERRs but Is
    Taking Steps to Increase Consistency:

    MCC Made Analytic Errors in Compact Impact Projections and Used Varying
    Methods that Affected the Projections' Results:

    Conclusions:

    Recommendations for Executive Action:

    Agency Comments and Our Evaluation:

    Appendix I: Objectives, Scope, and Methodology:

    Appendix II: MCC Minimum Acceptable ERRs:

    Appendix III: Compact ERRs:

    Appendix IV: Impact of Alternative Beneficiary Counts on El Salvador
    Compact Impact Projections:

    Appendix V: Comments from the Millennium Challenge Corporation:

    GAO Comments:

    Appendix VI: GAO Contact and Staff Acknowledgments:

    Tables:

    Table 1: Compact Impact Indicators Included in MCC's Public Reporting:

    Table 2: MCC Compact Hurdle Rates and Hurdle Rate Definition:

    Table 3: Comparison of MCC Compact ERRs Stated in Investment Memo with
    ERRs Calculated Using a 20-Year Time Frame:

    Table 4: Compact ERR Using Alternative Methods:

    Table 5: MCC's Compact-Level Impact Estimates for El Salvador, with
    Alternative Assumptions for Estimating Beneficiaries of Education
    Projects:

    Figures:

    Figure 1: MCC Compact Development and Implementation Process:

    Figure 2: Illustrative Examples of MCC Economic Analysis at Compact,
    Project, and Activity Levels:

    Figure 3: Illustration of Net benefits and ERR Calculations:

    Figure 4: Summary of Compacts for Armenia, El Salvador, Lesotho, and
    Mozambique:

    Figure 5: MCC Revisions to Impact Projections for Armenia, Based on Its
    Corrections of Analytic Errors:

    Figure 6: MCC Revisions to Impact Projections for El Salvador, Based on
    Its Corrections of Analytic Errors:

    Figure 7: MCC Revisions to Impact Projections for Mozambique, Based on
    Its Corrections of Analytic Errors:

    Figure 8: MCC's Alternative Methods for Calculating Compact ERR:

    Abbreviations:

    CEO: chief executive officer:

    ERR: economic rate of return:

    GDP: gross domestic product:

    MCA: Millennium Challenge Account:

    MCC: Millennium Challenge Corporation:

    OMB: Office of Management and Budget:

    [End of section]

    United States Government Accountability Office:
    Washington, DC 20548:

    June 17, 2008:

    The Honorable Howard L. Berman:
    Chairman:
    Committee on Foreign Affairs:
    House of Representatives:

    Dear Chairman Berman:

    In January 2004, Congress established the Millennium Challenge
    Corporation (MCC) to administer the Millennium Challenge Account (MCA)
    for foreign assistance. MCC's mission is to reduce poverty through
    sustainable economic growth in some of the world's poorest countries
    that create and maintain sound policy environments. MCC has received
    appropriations for fiscal years 2004 to 2008 totaling more than $7.5
    billion and, as of March 2008, has signed $5.5 billion in compacts
    [Footnote 1] with 16 countries. The President has requested $2.225
    billion for fiscal year 2009.

    MCC uses income and country performance criteria to annually select a
    list of countries eligible for MCA assistance. Eligible countries may
    then submit proposals for compacts containing multiple projects for
    MCC's review and approval. On receiving proposals for MCA assistance,
    MCC undertakes a comprehensive review, or due diligence,[Footnote 2]
    which seeks to ensure that the proposed compacts will advance MCC's
    mission. As part of due diligence, MCC assesses the potential economic
    impact of each compact. During due diligence, MCC identifies a minimum
    acceptable economic rate of return[Footnote 3] (ERR) for the compact
    and its projects; compares estimated costs and benefits to determine
    the compact's ERR; and projects the compact's impact, including its
    number of beneficiaries and its impact on income, economic growth, and
    poverty.

    MCC uses its projections, as well as other information gathered during
    due diligence, to inform its internal decisions to fund proposed
    projects and compacts. The results of the due diligence assessment are
    reported in an investment memo--an internal document prepared by MCC's
    transaction team[Footnote 4] that analyzes the compact--submitted to
    MCC's investment committee.[Footnote 5] MCC also publishes its compact
    impact projections during ongoing consultations with Congress. This
    information--found in MCC documents such as compacts, compact
    summaries, annual reports, and congressional notifications and budget
    justifications--sets expectations for the compact and provides
    information to Congress and the public about MCC's progress in
    achieving its mission.

    In July 2007, we reported that MCC's portrayal of the projected impact
    of its Vanuatu compact did not reflect MCC's underlying analyses.
    [Footnote 6] Our recommendations included that MCC revise its public
    reporting of the Vanuatu compact's projected impact and assess whether
    similar reporting for other compacts accurately reflects underlying
    economic analyses. The committee subsequently requested that we examine
    MCC's economic analyses for its compacts with other countries.

    As agreed with your office, for this report, we assessed (1) MCC's
    projections of ERR and (2) MCC's projections of compacts' impact on
    income and poverty as well as numbers of beneficiaries.

    To carry out this review, we reviewed MCC compacts with four countries:
    Armenia, El Salvador, Lesotho, and Mozambique. When we began our work
    in August 2007, these four countries represented about 41 percent of
    the $3.85 billion MCC had set aside for 13 compacts and included the
    most recent compacts signed in Eurasia, Latin America, and Africa. To
    assess MCC's compact projections, we reviewed MCC public and internal
    documents as well as relevant World Bank and International Monetary
    Fund documents. We also reviewed MCC's original and revised spreadsheet
    calculations of its economic projections and interviewed MCC economists
    and other officials in Washington, D.C., regarding these analyses.
    Finally, we consulted Office of Management and Budget (OMB) and GAO
    guidance on establishing and implementing internal controls.[Footnote
    7] We determined that the data we used were sufficiently reliable for
    the purposes of our analysis; however, we did not independently assess
    the reliability of all data and assumptions that affect the projections
    and that MCC used in its underlying economic analyses of compact
    projects and activities. We also did not assess MCC's progress in
    implementing these compacts or toward its projected results. We
    conducted this performance audit from August 2007 to June 2008, in
    accordance with generally accepted government auditing standards. Those
    standards require that we plan and perform the audit to obtain
    sufficient, appropriate evidence to provide a reasonable basis for our
    findings and conclusions based on our audit objectives. We believe that
    the evidence obtained provides a reasonable basis for our findings and
    conclusions based on our audit objectives. (See app. I for additional
    details of our scope and methodology.)

    Results in Brief:

    MCC used different time frames and methods to calculate ERRs for the
    four compacts we reviewed. In calculating ERR for the 20 projects
    within the compacts that we reviewed, MCC used a 20-year time frame for
    9 projects and used different time frames for the other 11 projects. In
    2 of the 11 projects, applying a 20-year time frame, as MCC had done in
    similar projects, would reduce the ERR below the level MCC set as the
    minimum acceptable ERR. At the compact level, MCC's use of varying time
    frames did not significantly affect the results of the ERR
    calculations. MCC used varying methods to account for the costs of same-
    sector projects, although its approaches to determining project
    benefits were generally similar. MCC also used two different methods to
    calculate compact-level ERR; however, the choice of method did not
    reduce it below the minimum ERR. In three of the four compacts that we
    reviewed, MCC did not retain documentation of the economic analyses
    used to support the investment decision, but continued to modify the
    analyses. MCC has recently begun to standardize elements of its
    economic analyses and centralize its records management.

    MCC identified and corrected analytic errors in its projections of
    impact on income and poverty. In addition, MCC used varying methods for
    projecting compact impact on income and poverty, which affected the
    projections' results.

    * Analytic errors. In responding to our questions about its published
    impact projections, MCC identified analytic errors for three of the
    four compacts[Footnote 8] and, in correcting these errors, generally
    lowered its projected impacts on poverty and income. Correcting these
    errors raised one projection by 5 percent but reduced others by 3
    percent to 96 percent. For example, for Armenia, MCC corrected an
    erroneous baseline, reducing the projected decrease in rural poverty
    from 6 percentage points to 3 percentage points. For Mozambique, MCC
    used a corrected formula and data, reducing its projection of the
    number of persons to be lifted out of poverty in Mozambique in 2015
    from 270,000 to either 27,000 or 56,000, depending on the approach
    selected. According to MCC officials, the revised projections would not
    have affected MCC's decision to recommend signing the compacts. The
    officials noted that MCC's impact projections had not undergone a final
    check for accuracy and validity but that MCC has begun to implement a
    peer review process for compacts currently under development. However,
    MCC has not documented procedures for these reviews.

    * Varying methods. MCC used varying methods to project the four
    compacts' impact on income and poverty, limiting the projections'
    comparability and replicability across compacts. For example, for
    Armenia, El Salvador, and Mozambique, MCC estimated the impact of
    compact projects on income by summing the total benefits of individual
    compact projects and adding them to the income that would have
    prevailed without MCC. For Lesotho, MCC extrapolated its compact
    results from the published results of a World Bank model based on
    elements different from those of the MCC compact. In response to our
    questions, MCC revised its poverty impact projection for Mozambique by
    presenting two estimates of the relationship between income growth and
    poverty--estimates based on either Mozambique country-specific data or
    its initial ad hoc estimate, which MCC stated was consistent with cross-
    country experience. Although a number of methods for projecting impact
    are valid, the method chosen can affect the results, and MCC's
    guidelines do not identify preferred methods for these calculations.
    MCC also used varying methods to estimate numbers of beneficiaries for
    the compacts and has not provided specific criteria for defining
    beneficiaries; however, MCC officials reported they are taking steps to
    provide more detailed guidance for estimating beneficiaries.

    To improve the reliability and comparability of its projected ERR and
    economic impacts, we recommend that the CEO of MCC take the following
    actions:

    * adopt and implement written procedures for a secondary independent
    review of the methods and results of its economic analyses and:

    * improve MCC's guidelines by identifying a consistent approach with
    preferred methods for projecting compacts' impact on income and
    poverty.

    In commenting on a draft of this report, MCC concurred with our
    recommendations and outlined steps it is taking, including developing
    standard practices and templates and initiating a peer review process.
    MCC stated that in many cases the inconsistencies we identified were
    technically appropriate. MCC also stated that it disagreed with what it
    saw as our report's implication that ERRs are disconnected from income
    and poverty effects. However, we assert that ERRs do not provide
    information about MCC's impact on the poor and are not an absolute
    measure of income benefits, but a relative measure of benefits in
    relation to costs. We have reprinted MCC's comments, with our response,
    in appendix V. We have incorporated technical clarifications from MCC
    where appropriate.

    Background:

    MCC conducts its economic analyses during the compact development
    process, the first of three compact phases. These analyses include
    establishing the minimum ERR that the compact should achieve,
    projecting the compact's ERR, and estimating the compact's impact on
    income and growth. MCC did not publish ERR information for three of the
    four compacts we reviewed, but its public documents include statements
    of compact impact on income and poverty. Of the four compacts that we
    reviewed--Armenia, El Salvador, Lesotho, and Mozambique--three include
    funding for road projects, and three include funding for water
    projects, among other projects.

    Compact Development Process:

    If MCC determines that a country is eligible for assistance, the
    country may submit a compact proposal, which generally comprises
    several projects. According to MCC guidelines,[Footnote 9] the proposal
    should include economic analyses of proposed projects to demonstrate
    their likely impact on growth and poverty in the country. MCC also may
    provide assistance and feedback in developing the proposal, including
    making grants to facilitate the development of the compact (see fig.
    1).

    Figure 1: MCC Compact Development and Implementation Process:

    [See PDF for image]

    This figure is an illustration of the MCC Compact Development and
    Implementation Process, as follows:

    Development compact:
    * Eligibility determination:
    - Country proposal development;
    * Opportunity memo:
    - MCC's due diligence review;
    * Investment memo:
    - Compact negotiation and MCC Board approval.

    Finalize supplemental agreements:
    * Compact signing:
    - MCC and country complete entry into force requirements.

    Implement compact:
    * Entry into force:
    - MCC authorizes fund disbursement and oversees country implementation
    of compact.

    Source: GAO analysis of MCC data.

    [End of figure]

    After a country submits its proposal, MCC's transaction team for the
    compact conducts a preliminary assessment of the proposal and reports
    its findings in an internal opportunity memo to the MCC investment
    committee. MCC assembles a different transaction team for each compact.
    If the opportunity memo is approved, the team launches a detailed due
    diligence review that includes economic analyses of the proposed
    projects. As members of MCC country transaction teams, MCC's lead
    economists undertake these analyses while working with other members of
    the team and country officials. According to MCC, as part of this
    effort, MCC economists review the preliminary analyses performed by
    country counterparts and consultants. These due diligence reviews,
    including conducting economic analyses, have lasted, on average,
    slightly more than 10 months for the 16 countries with signed compacts
    as of March 31, 2008.

    At the conclusion of due diligence, the transaction team sends an
    investment memo to the MCC investment committee, with recommendations
    based on its assessment of the proposal. MCC notifies Congress 15 days
    prior to beginning negotiations with the country, sending a formal
    congressional notification.[Footnote 10] For the four compacts we
    examined, the congressional notifications included some of the results
    of MCC's economic analyses. If compact negotiations with the eligible
    country are successful, the investment committee submits the proposed
    compact to the MCC Board for approval.[Footnote 11] With the Board's
    approval, MCC and the country sign the compact before completing
    additional agreements--such as a disbursement agreement and procurement
    agreement--and ultimately implementing the projects funded in the
    compact.[Footnote 12]

    Economic Analyses:

    For each compact proposal, MCC and the eligible country conduct
    economic analyses, which MCC uses to inform its decisions to fund
    compacts and report to Congress and the public.[Footnote 13] MCC
    generally conducts these analyses at the compact and project levels
    (see fig. 2).

    Figure 2: Illustrative Examples of MCC Economic Analysis at Compact,
    Project, and Activity Levels:

    [See PDF for image]

    This illustration offers examples of MCC economic analysis at compact,
    project, and activity levels, as follows:

    Compact-level analysis:
    * Aggregate economic rate of return (ERR);
    * Aggregate income and poverty effects;
    * Aggregate number of beneficiaries.

    Project-level analysis:
    * ERR, number of beneficiaries;
    Illustrative examples:
    - Water and sanitation;
    - Road construction.

    Activity level analysis:
    * ERR;
    Illustrative examples:
    - Individual city systems;
    - Individual road segments.

    Sources: GAO analysis of MCC data; Nova Development (clip art).

    [End of figure]

    The economic analyses include projections of compacts' impact,
    including their ERR, the impact on income and poverty, and the
    projected number of beneficiaries.

    * ERR analysis. To provide a basis for assessing the compact and
    project ERRs, MCC sets the minimum acceptable ERR that compacts and
    projects should achieve to be eligible for funding. (See app. II for a
    discussion of MCC's minimum ERRs for the four countries.) If the
    compact or project does not meet the minimum rate, MCC retains the
    discretion to fund it but requires justification based on the specific
    circumstances.

    The ERR analysis compares costs and benefits, where the costs are the
    MCA grants and the country's future recurrent costs--such as
    maintenance expenses--and the benefits are increases in incomes in
    recipient countries. (See fig. 3.) MCC guidelines state that normal
    practice is to calculate ERRs using 10-, 20-, and 30-year time horizons
    to determine project and compact ERRs' sensitivity to varying time
    frames. If the ERR is sensitive to this time horizon, the guidelines
    require that this be noted explicitly.

    Figure 3: Illustration of Net benefits and ERR Calculations:

    [See PDF for image]

    This illustration depicts the following Net benefits and ERR
    calculations:

    Benefits, minus Costs, equals Net benefits, generates ERR.

    Year: 1;
    Annual benefit to the country: Benefits1;
    Expenditures to implement project(s): Costs1;
    Annual average expected rate of return on each dollar of MCC
    assistance: Net benefit1.

    Year: 2-9;
    Annual benefit to the country: [Empty];
    Expenditures to implement project(s): [Empty];
    Annual average expected rate of return on each dollar of MCC
    assistance: [Empty].

    Year: 10;
    Annual benefit to the country: Benefits10;
    Expenditures to implement project(s): Costs10;
    Annual average expected rate of return on each dollar of MCC
    assistance: Net benefit10;

    Year 1-10 yields 10-year ERR.

    Year: 11-19;
    Annual benefit to the country: [Empty];
    Expenditures to implement project(s): [Empty];
    Annual average expected rate of return on each dollar of MCC
    assistance: [Empty].

    Year: 20;
    Annual benefit to the country: Benefits20;
    Expenditures to implement project(s): Costs20;
    Annual average expected rate of return on each dollar of MCC
    assistance: Net benefit20;

    Year 1-20 yields 20-year ERR.

    Year: 21-29;
    Annual benefit to the country: [Empty];
    Expenditures to implement project(s): [Empty];
    Annual average expected rate of return on each dollar of MCC
    assistance: [Empty].

    Year: 30;
    Annual benefit to the country: Benefits30;
    Expenditures to implement project(s): Costs30;
    Annual average expected rate of return on each dollar of MCC
    assistance: Net benefit30;

    Year 1-30 yields 30-year ERR.

    Source: GAO analysis of MCC data.

    Note: The ERR of a project is the discount rate (interest rate) at
    which the present value of the project's cost stream is equal to the
    present value of its benefits stream.

    [End of figure]

    * Impact analysis. MCC's guidelines state that economic analyses should
    quantify the proposed projects' expected beneficiaries and expected
    impact on incomes and on poverty.[Footnote 14] Based on our analysis,
    MCC does not establish minimum thresholds for compact impact.

    Public Reporting:

    While MCC's transaction team reports both ERR and impact projections to
    the MCC investment committee in its investment memo, MCC's public
    reporting for the four compacts in our review generally included
    information about only its projections of compact impact; for these
    four countries MCC published ERR projections for only Armenia. MCC's
    public reporting included projections of compact impact on income and
    poverty for three countries and projections of gross domestic product
    (GDP) growth for one country. For each compact, MCC also estimated the
    number of beneficiaries. (See table 1.)

    Table 1: Compact Impact Indicators Included in MCC's Public Reporting:

    Increased income in compact areas:
    Armenia: [Check];
    El Salvador: [Check];
    Lesotho: [Empty];
    Mozambique: [Check].

    Increased GDP growth rate:
    Armenia: [Empty];
    El Salvador: [Empty];
    Lesotho: [Check];
    Mozambique: [Empty].

    Decreased poverty:
    Armenia: [Check];
    El Salvador: [Check];
    Lesotho: [Empty];
    Mozambique: [Check].

    Number of beneficiaries:
    Armenia: [Check];
    El Salvador: [Check];
    Lesotho: [Check];
    Mozambique: [Check].

    Source: GAO analysis of MCC compacts.

    [End of table]

    Compact Projects and Funding:

    The types of projects included in the four compacts vary, although El
    Salvador, Lesotho, and Mozambique include water projects[Footnote 15]
    and Armenia, El Salvador, and Mozambique include road projects. The
    compacts provide a total of approximately $1.56 billion in MCA
    assistance.[Footnote 16] (See fig. 4.)

    Figure 4: Summary of Compacts for Armenia, El Salvador, Lesotho, and
    Mozambique:

    [See PDF for image]

    This figure is a map of the world, depicting the locations of Armenia,
    El Salvador, Lesotho, and Mozambique, and providing the following
    information:

    Country: Armenia;
    Compact size: $235.65 million;
    Projects:
    * Rural road rehabilitation;
    * Irrigation;
    Area of intervention: Rural areas.

    Country: El Salvador;
    Compact size: $460.94 million;
    Projects:
    * Education and community infrastructure;
    * Agricultural productivity;
    * Road construction and rehabilitation;
    Area of intervention: Northern zone.

    Country: Lesotho;
    Compact size: $362.55 million;
    Projects:
    * Water and sanitation;
    * Health;
    * Private sector development;
    Area of intervention: Entire country.

    Country: Mozambique;
    Compact size: $506.92 million;
    Projects:
    * Water and sanitation;
    * Land tenure;
    * Farmer income;
    Area of intervention: Four northern provinces.

    Sources: GAO analysis of MCC data; Map Resources (map).

    [End of figure]

    MCC Used Different Time Frames and Methods to Calculate ERRs but Is
    Taking Steps to Increase Consistency:

    MCC used different time frames to calculate project-level ERRs; these
    differences did not affect the ERR significantly in 18 of the 20
    compact projects[Footnote 17] we examined, but two ERRs would fall
    below the minimum ERR if MCC applied the 20-year time frame it used for
    other compacts and similar projects. At the compact level, the
    different time frames did not change the ERR significantly. In
    addition, we found that MCC used varying methods to account for the
    costs of same-sector projects, although its approaches to determining
    benefits were generally similar. MCC also used two different methods to
    calculate compact-level ERR; however, the choice of method did not
    change the ERRs significantly. In some cases, MCC did not fully retain
    its documentation of its economic analyses. MCC has recently taken
    steps to standardize elements of its economic analyses and improve its
    records management and plans to implement additional measures.

    MCC Used Different Time Frames for Project and Compact ERRs:

    In its ERR calculations for 20 projects included in the four compacts
    we reviewed, MCC used a 20-year time frame for 9 projects and used
    different time frames for 11 projects. Nearly all project ERRs that MCC
    initially calculated met the minimum ERR set for each compact.[Footnote
    18] Our analysis shows that for 9 of the 11 ERRs calculated with other
    time frames, recalculating the ERR with a 20-year time frame does not
    lower the ERR below the minimum ERR. However, for two projects,
    recalculating the ERR with a 20-year time frame produces a result below
    the minimum ERR.

    * The ERR for the El Salvador Community Infrastructure project, which
    MCC calculated as slightly below the minimum ERR of 10.8 percent over
    25 years, drops to 9.0 percent when calculated over 20 years.
    Correcting a calculation error further reduces the El Salvador
    project's ERR to 6.8 percent over 20 years--four percentage points
    below the El Salvador minimum ERR.

    * The ERR for the Mozambique roads project, which MCC calculated at
    10.3 percent over 24 years, drops to 8.1 percent over 20 years, below
    the minimum ERR of 8.76 percent. Three of the four individual
    Mozambique roads MCC analyzed as part of this project also fall below
    the minimum ERR at 20 years.[Footnote 19]

    If MCC had applied a 20-year time frame in calculating the ERR for
    these two projects, the projects might have been restructured to
    increase their ERR or MCC would have had to specifically justify the
    exception. MCC does not currently have a policy addressing what steps
    to take in cases where subsequent analysis results in a project ERR
    below the minimum. MCC stated that it would address changes such as
    this on a case-by-case basis depending on the timing of the change
    within the compact development process, and the magnitude of the
    change. MCC officials also noted that the ERR time frame is only one
    aspect of MCC's analysis of the ERR's sensitivity to various factors.

    In addition, MCC used different time frames in calculating ERRs for
    comparable projects in different compacts. MCC calculated water project
    ERRs for El Salvador, Lesotho, and Mozambique over 25, 20, and 21
    years, respectively, and calculated road project ERRs for Armenia, El
    Salvador, and Mozambique over 20, 25, and 24 years, respectively. MCC
    officials told us they explored the ERRs' sensitivity to varying time
    frames, as MCC's guidelines require, but they did not regard as
    prescriptive the guidelines' statement that normal practice is to
    examine 10-, 20-, and 30-year time horizons.[Footnote 20] MCC officials
    also noted that MCC economists have recently committed to the use of a
    default 20-year time horizon for their analyses. MCC also will alter
    this time frame for specific circumstances or projects whose benefits
    have a longer duration--such as education projects and large-scale
    construction projects--or have a shorter or longer physical life
    expectancy.

    At the compact level, our analysis shows that applying a 20-year time
    frame in place of varying time frames does not significantly affect the
    results of the ERR calculations. (See app. III for a summary of our
    comparison of the compact minimum ERR with the compact ERRs that MCC
    reported and our calculations of the 20-year ERRs.)

    MCC Used Varying Methods in Calculating Project and Compact ERRs:

    In calculating project ERRs, we found that MCC used varying methods to
    account for the investment and recurrent cost components of water and
    road sector projects for the four compacts we reviewed.

    * Investment costs. For water projects, MCC counted the cost of
    construction in Lesotho as one lump sum in the first year of the
    analysis but phased in the cost in Mozambique and El Salvador over
    multiple years. Phasing the costs evenly over the 5-year compact in
    Lesotho would have increased the ERR for both the urban and rural water
    projects.[Footnote 21] For road projects, MCC phased the investment
    cost over time for Armenia, El Salvador, and Mozambique. However, in El
    Salvador, MCC accounted for the salvage value[Footnote 22] of the
    project in the last year of the analysis but did not include a similar
    projection in the analysis for Armenia and Mozambique.

    * Recurrent costs. For water projects, MCC counted Lesotho and
    Mozambique's recurrent costs in the years after the original compact
    investment. For El Salvador, MCC counted recurrent costs only during
    the initial 5-year compact period. Including future recurrent costs
    would have reduced the ERR. For road projects, MCC spread recurrent
    costs over time for Armenia, El Salvador, and Mozambique.

    According to our review, MCC's approaches to determining the benefits
    of water and road projects were generally similar.

    * Water projects. MCC generally counted as water project benefits
    increased time available for work, resulting from less time spent
    fetching water or being ill, and lower spending on health care and
    water.

    * Road projects. MCC generally estimated traffic volumes on the roads
    and determined road project benefits based on the savings from reduced
    travel time and vehicle operating costs for existing and generated
    traffic.[Footnote 23]

    In calculating the compact-level ERRs for the four compacts' investment
    memos, MCC used two different methods.[Footnote 24]

    * For Armenia and Mozambique, MCC first determined the net benefits in
    each year for each project and then calculated the compact ERR based on
    the total net benefits.

    * For El Salvador and Lesotho, MCC first determined each project's net
    benefits and ERR and then determined the compact ERR by averaging the
    project ERRs, weighting each ERR according to the project's budgeted
    size relative to the overall compact.

    According to MCC officials, the choice of method depended on the
    preference of the transaction team's lead economist. Our analysis shows
    that using the second method to recalculate each compact ERR reduces it
    by less than 2 percentage points and does not reduce it below the
    minimum ERR. However, these results demonstrate that the choice of
    method influences the compact-level ERR and may affect the
    comparability of ERRs across compacts. (See app. III for details of our
    analysis of MCC compact ERRs.)

    Insufficient Records Management Led to Discrepancies between Investment
    Memos and Underlying Analyses:

    In three of the four cases we reviewed, the projections of ERR reported
    to the MCC investment committee differ from those in MCC's underlying
    spreadsheets.

    * For Armenia, MCC changed its estimate of project benefits after the
    investment memo.

    * For Lesotho, the MCC spreadsheets contain figures calculated using
    different methods or reflecting additional analysis after the
    investment memo.

    * For Mozambique, the calculation of the figures presented in the
    investment memo relied on spreadsheet formula links that were not
    properly updated at the time. According to MCC, when they provided the
    spreadsheets to us, they updated the formulas and overwrote the
    original calculations.

    For these three compacts, MCC's records management did not fully
    preserve the information and analysis used to support the investment
    decision. OMB guidance and GAO guidelines for internal controls both
    note the importance of controls over the information that U.S. agencies
    use to make decisions.[Footnote 25]

    MCC Has Taken Several Steps to Improve the Consistency of ERR Analysis:

    After completing due diligence for the four compacts we studied, and
    during the course of our audit, MCC told us they took or began taking
    steps to increase the consistency of its analyses and improve its
    records management. For example, MCC:

    * committed to using 20 years as the default time frame for calculating
    ERRs and identified projects where this time frame may be altered;

    * began revising its guidance to clarify that ERR sensitivity should be
    tested by varying the time frame of the analysis, rather than
    specifying 10-, 20-, and 30-year analyses;

    * began developing standards for consistently analyzing certain types
    of projects across compacts;

    * established that summing net benefits and costs across all projects
    is the preferred method for calculating a compact-level ERR; and:

    * planned to implement a data management system to centralize its
    records management.

    MCC Made Analytic Errors in Compact Impact Projections and Used Varying
    Methods that Affected the Projections' Results:

    MCC identified and corrected analytic errors in its projections of
    compact impact for Armenia, El Salvador, and Mozambique, generally
    reducing each compact's estimated impact on income and poverty.
    [Footnote 26] According to MCC, the revised calculations would not have
    affected its approval of the compacts, but it will perform peer reviews
    of future impact projections. In addition, MCC used different methods
    for projecting compact impact on income and poverty, limiting the
    estimates' comparability and replicability; its current guidance does
    not address the choice of method for these projections. MCC also used
    different methods to estimate numbers of compact beneficiaries, but
    stated that it is taking steps to provide more detailed guidance for
    estimating number of beneficiaries.

    MCC Identified and Corrected Analytic Errors for Three Compacts:

    In responding to our questions about its impact analyses for Armenia,
    El Salvador, and Mozambique,[Footnote 27] MCC identified a number of
    analytic errors in its projections of impact on income and poverty. MCC
    subsequently corrected these errors, generally reducing the projected
    impacts on income and poverty for each compact.

    * Armenia. MCC determined that it had included road project benefits in
    estimating the income increase from agriculture for Armenia. MCC also
    used the wrong baseline in projecting poverty effects. Correcting these
    errors affected projections of the compact's effect, reducing the
    estimated increase in rural areas' real income from agriculture after 5
    years from 5 percent to 3 percent and lowering the estimated decline in
    Armenia's poverty rate from 6 percentage points to 3 percentage points.
    Figure 5 summarizes MCC's original impact projections for Armenia and
    its revisions after correcting the errors it identified.

    Figure 5: MCC Revisions to Impact Projections for Armenia, Based on Its
    Corrections of Analytic Errors:

    [See PDF for image]

    This figure is a table depicting the following data:

    Armenia:

    Income/growth: Annual income in rural areas in 2010;
    Original calculation: $36 million increase;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Income/growth: Annual income in rural areas in 2015;
    Original calculation: $133 million increase;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Income/growth: Real income from agriculture in rural areas at end of
    compact;
    Original calculation: 5 percent increase;
    MCC's revised calculation: 3 percent increase;
    Absolute change: -2 percentage points;
    Percent change: -40%.

    Income/growth: Real income from agriculture in rural areas in 2013[A];
    Original calculation: 23 percent increase;
    MCC's revised calculation: 9 percent increase;
    Absolute change: -14 percentage points;
    Percent change: -61%.

    Poverty: Rural poverty rate[B];
    Original calculation: 6 percentage point decrease;
    MCC's revised calculation: 3 percentage point decrease;
    Absolute change: -3 percentage points;
    Percent change: -50%.

    Beneficiaries:
    Original calculation: 750,000;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Source: GAO analysis of MCC data.

    [A] MCC stated projections of real income increases in Armenia using an
    index, with the baseline set at 100 in 2005. Thus, MCC's initial
    projection of an increase from 2005 baseline index of 100 to 123 in
    2013 corresponds to a 23 percent increase in real income, which was
    later revised to 109, or a 9 percent increase, for the same year.

    [B] MCC stated projections of poverty rate reductions in Armenia using
    a baseline of 32 percent. Thus, MCC's initial projections of a
    reduction in the poverty rate from 2004 baseline of 32 percent to a
    target of 26 percent in year 2013 corresponds to a decline of 6
    percentage points, which was later revised to a reduction to 29
    percent, or a decline of 3 percentage points, for the same year.

    [End of figure]

    * El Salvador. MCC determined that it had made an error in its formula
    for projecting per capita income increases and overestimated income
    increases in its projections of poverty rate reduction for El Salvador.
    In correcting these errors, MCC lowered its projections of income and
    poverty impact. Figure 6 summarizes MCC's original impact projections
    for El Salvador and its revisions after correcting the errors it
    identified. MCC also presented alternative methods for identifying
    beneficiaries, which further reduces the results of its compact-level
    projections. (See app. IV.)

    Figure 6: MCC Revisions to Impact Projections for El Salvador, Based on
    Its Corrections of Analytic Errors:

    [See PDF for image]

    This figure is a table depicting the following data:

    El Salvador:

    Income/growth: Incomes within the region within 5 years;
    Original calculation: 18 percent increase;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Income/growth: Incomes within the region within 10 years;
    Original calculation: 26 percent increase;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Income/growth: Annual per capita income of beneficiaries within 5
    years;
    Original calculation: $148 increase;
    MCC's revised calculation: $123 increase;
    Absolute change: -$25;
    Percent change: -17%.

    Income/growth: Annual per capita income of beneficiaries within 10
    years;
    Original calculation: $230 increase;
    MCC's revised calculation: $189 increase;
    Absolute change: -$41;
    Percent change: -18%.

    Poverty: Number of persons for whom poverty is alleviated[A];
    Original calculation: 150,000 persons;
    MCC's revised calculation: 145,000 persons;
    Absolute change: -5,000 persons;
    Percent change: -3%.

    Poverty: Poverty rate in the Northern Zone within 5 years;
    Original calculation: 11 percentage point decrease;
    MCC's revised calculation: 10 percentage point decrease;
    Absolute change: -1 percentage point;
    Percent change: -9%.

    Poverty: Poverty rate in the Northern Zone within 10 years;
    Original calculation: 17 percentage point decrease;
    MCC's revised calculation: 15 percentage point decrease;
    Absolute change: -2 percentage point2;
    Percent change: -12%.

    Beneficiaries:
    Original calculation: 850,000;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Source: GAO analysis of MCC data.

    Notes: For El Salvador, MCC calculated with-and without-project
    scenarios and cited these figures in its public documents. To estimate
    compact impact attributable to MCC, we calculated the differences
    between with-and without-project scenarios.

    MCC also presented alternative methods for identifying beneficiaries of
    two education projects, which affect the results of MCC's compact
    impact analysis. (See app. IV.)

    [A] MCC's spreadsheet calculations show a figure of more than 160,000
    Salvadorans lifted out of poverty in year 10 of the project. However,
    MCC's compact summary stated that "the program is projected to directly
    alleviate the poverty of over 150,000 Salvadorans." The compact
    summary's projection of 150,000 is used here.

    [End of figure]

    * Mozambique. MCC determined that it had not updated formula links in
    its analytic spreadsheets for Mozambique. Correcting this error led MCC
    to revise the projected income increase upward by $4 million for 2015
    and downward by $13 million for 2025. In addition, MCC corrected both
    the formula and the data[Footnote 28] used to calculate the effect of
    this income increase on poverty in Mozambique, presenting two
    alternative approaches for estimating poverty elasticity.[Footnote 29]
    In making these changes, MCC lowered the projected decline in the
    poverty rate as well as the number of persons likely to be lifted out
    of poverty because of the compact. For example, MCC originally
    projected a 7 percent reduction in Mozambique's poverty rate in 2015;
    using the alternative approaches for estimating poverty elasticity, MCC
    projected a poverty rate reduction of either 0.6 percent or 2 percent
    in 2015. Likewise, MCC originally projected that 270,000 people would
    be lifted out of poverty in Mozambique in 2015; using the alternative
    approaches, MCC projected that either 27,000 persons or 56,000 persons
    would be lifted out of poverty in 2015. Figure 7 summarizes MCC's
    original impact projections for Mozambique and its revisions after
    correcting errors and presenting alternative approaches for estimating
    poverty elasticity.

    Figure 7: MCC Revisions to Impact Projections for Mozambique, Based on
    Its Corrections of Analytic Errors:

    [See PDF for image]

    This figure is a table depicting the following data:

    Mozambique:

    Income/growth: Income in 2015;
    Original calculation: $75 million increase;
    MCC's revised calculation: $79 million increase;
    Absolute change: +$4 million;
    Percent change: +5%.

    Income/growth: Income in 2025;
    Original calculation: $180 million increase;
    MCC's revised calculation: $167 million increase;
    Absolute change: -$13 million;
    Percent change: -7%.

    Poverty: Poverty rate in 2015;
    Original calculation: 7 percent decrease;
    MCC's revised calculation: 0.6 decrease (lower estimate); 2 percent
    decrease (higher estimate);
    Absolute change: -6.4 percentage points (lower estimate); -5 percentage
    points (higher estimate);
    Percent change: -91% (lower estimate); -66% (higher estimate).

    Poverty: Poverty rate in 2025;
    Original calculation: 16 percent decrease;
    MCC's revised calculation: 0.7 decrease (lower estimate); 3 percent
    decrease (higher estimate);
    Absolute change: -15.3 percentage points (lower estimate); -13
    percentage points (higher estimate);
    Percent change: -96% (lower estimate); -83% (higher estimate).

    Poverty: Number of persons lifted out of poverty in 2015;
    Original calculation: 270,000 persons;
    MCC's revised calculation: 27,000 persons (lower estimate); 56,000
    persons (higher estimate);
    Absolute change: -243,000 persons (lower estimate); -214,000 persons
    (higher estimate);
    Percent change: -90% (lower estimate); -79% (higher estimate).

    Poverty: Number of persons lifted out of poverty in 2025;
    Original calculation: 440,000 persons;
    MCC's revised calculation: 32,000 persons (lower estimate); 43,000
    persons (higher estimate);
    Absolute change: -408,000 persons (lower estimate); -397,000 persons
    (higher estimate);
    Percent change: -93% (lower estimate); -90% (higher estimate).

    Beneficiaries:
    Original calculation: 5 million by 2015;
    MCC's revised calculation: No change;
    Absolute change: 0;
    Percent change: 0.

    Source: GAO analysis of MCC data.

    Notes: For Mozambique, MCC's public documents stated projections of
    poverty (1) with the MCC compact and (2) without the MCC compact. To
    estimate compact impact attributable to MCC, we calculated the
    differences between with-and without-project figures.

    In addition to correcting its formula used to calculate the effect of
    increased income on poverty in Mozambique, MCC presented alternative
    approaches for estimating poverty elasticity. MCC's poverty impact
    projections using both alternatives are shown here.

    [End of figure]

    Revised Projections Would Not Have Changed Funding Decisions, but
    Future Compact Impact Projections Will Undergo Review:

    According to MCC officials, the revisions to its initial calculations
    of compact impact on income and poverty for Armenia, El Salvador, and
    Mozambique would not have changed MCC's decision to recommend each
    compact to the MCC Board. The officials emphasized that MCC's economic
    impact projections are one of many aspects of the due diligence process
    that inform its compact investment decisions. A senior official stated
    that MCC's compact-level analysis is separate from the decision about
    whether to invest in specific projects; the two levels of analysis are
    connected but play different roles in MCC's decision making. MCC
    officials also told us they used conservative data and assumptions to
    project compacts' impact on income, growth, and poverty. However,
    future public statements would reflect MCC's corrections and revised
    analyses.[Footnote 30]

    In addition, MCC officials said that although lead economists
    independently review the economic projections performed by others on
    the transaction team for each country, calculations and assumptions
    performed by the lead economists for the compacts we reviewed did not
    undergo a final check for accuracy and validity. The officials stated
    that such a review might have caught the errors that MCC later
    identified and corrected.[Footnote 31] They further stated that, with
    fewer compacts undergoing due diligence in the future, MCC will have
    the staff capacity to ensure that such reviews are performed. MCC
    officials told us they have begun to implement this peer review for
    compacts currently under development. However, MCC has not documented
    its procedures for conducting these reviews, so it is unclear what
    criteria and level of detail the peer review includes.

    MCC Used Varying Methods to Project Compact Impact on Income and
    Poverty:

    MCC used different methods to project the four compacts' impact on
    income and poverty, limiting the estimates' comparability and
    replicability. Although the method chosen can affect the results, MCC
    has not provided preferred methods, or guidelines for selecting a
    method, for these calculations.

    MCC Used Varying Methods for Economic Growth Projections:

    MCC used a different method to estimate impact on economic growth for
    Armenia, El Salvador, and Mozambique than it used for Lesotho. As a
    result, the estimates are not comparable.

    * For Armenia, El Salvador, and Mozambique, MCC estimated the impact of
    compact projects on aggregate income or income growth.[Footnote 32]
    This method entailed summing the total benefits of individual compact
    projects and adding them to the income that would have prevailed
    without MCC.

    * For Lesotho, MCC estimated the impact of the compact on the country's
    GDP growth rate using the published results from a World Bank
    simulation model. According to MCC, the World Bank's model showed that
    a public and private investment of about $100 million would lead to an
    increase in GDP growth rate of 3.75 percentage points. Reasoning that
    its projected investment in the Lesotho compact is analogous to the
    investment in the World Bank model, MCC extrapolated from the World
    Bank's estimate to project that the compact, if successfully
    implemented, would nearly double Lesotho's GDP growth rate.[Footnote
    33]

    According to MCC, its use of GDP growth impacts based on the published
    World Bank model's results was appropriate, in that the model was
    consistent with the scale and type of public infrastructure investments
    proposed under the compact. MCC also noted that the World Bank model
    provided more information than did MCC's analyses for the other
    compacts.[Footnote 34] However, our analysis of the World Bank report
    [Footnote 35] shows some differences between MCC's and the World Bank's
    interventions. For example, whereas MCC's compact comprises only
    increased public investment in infrastructure, the World Bank report's
    model projects the combined effect of three elements: increase in
    garment exports, growth in commercial agriculture, and increase in
    public investment in infrastructure.[Footnote 36] Because the model's
    three elements have different and interlinked effects, MCC's
    extrapolation of the effect of one of these elements, public investment
    in infrastructure, to its compact requires a number of assumptions that
    cannot be validated.

    Although MCC guidelines call for projecting compacts' impact on
    economic growth starting from the project level, MCC has not provided
    preferred methods, or guidelines for selecting a method, for estimating
    compacts' impact on GDP growth rate.

    MCC Used Varying Methods to Project Poverty Reduction:

    MCC estimated compacts' impact on the poverty rate for Armenia, El
    Salvador, and Mozambique based on the responsiveness, or elasticity, of
    the poverty rate to changes in income.[Footnote 37] MCC uses several
    income measures in this analysis. For Armenia, MCC estimated poverty
    reduction based on its elasticity with respect to agricultural value
    added to the local economy; for El Salvador, with respect to increased
    annual national GDP; and for Mozambique, with respect to increased
    annual regional GDP. [Footnote 38]

    Each of these measures can be a valid measure of income and
    corresponding elasticity of the poverty rate to income changes.
    However, the types of data used to estimate poverty elasticity can
    significantly affect poverty impact projections. For example, in
    projecting the Mozambique compact's impact on poverty, MCC initially
    used an ad hoc estimate of elasticity rather than an elasticity based
    on historical income and poverty data specific to Mozambique; according
    to MCC officials, the ad hoc elasticity was consistent with
    conservative cross-country estimates. In a response to our questions
    about its original analysis, MCC projected revised poverty impacts for
    Mozambique in two ways, in both cases correcting for analytic errors
    discussed above. For one revised projection, MCC used its initial
    elasticity based on cross-country data. For the second projection, MCC
    used an alternative elasticity calculated from Mozambique data, which
    resulted in lower poverty reduction estimates.[Footnote 39] The
    resulting estimates for the number of persons to be lifted out of
    poverty were 56,000 versus 27,000 persons by 2015 and 43,000 versus
    32,000 persons by 2025, respectively.

    MCC's guidelines call for estimating compacts' impact on reducing the
    poverty rate but do not discuss or prioritize various possible methods
    for estimating poverty reduction.[Footnote 40] Because the methods
    chosen for each compact can vary, the resulting estimates of poverty
    reduction may not be comparable or replicable across countries.

    MCC Used Different Methods to Estimate Beneficiaries and Is Taking
    Steps to Strengthen Guidance:

    MCC used two different methods to estimate the number of beneficiaries
    it reported for the compacts we reviewed: (1) summing beneficiaries
    from the individual projects, with adjustments for double counting and
    timing of benefits, and (2) counting the intervention area's entire
    population as beneficiaries.

    * For Armenia and Mozambique, MCC counted the population of the areas
    where the compact is being implemented as project-specific
    beneficiaries and summed these project totals to produce the reported
    number of beneficiaries. MCC corrected for double counting of
    beneficiaries in areas where more than one project was implemented. For
    Mozambique, MCC projected a gradual increase in the number of
    beneficiaries as benefits accrue over time, whereas for Armenia it did
    not.

    * For El Salvador, MCC reported that the entire population of the
    intervention area would benefit,[Footnote 41] although the project-
    specific beneficiary counts did not sum to the area's population.

    * For Lesotho, assuming that the compact would have economywide impact,
    MCC reported the country's entire population as beneficiaries although
    estimates of project beneficiaries were available.

    MCC guidelines define program beneficiaries as individuals or groups
    that derive economic gains from MCC investments. However, the
    guidelines do not explicitly define economic gain or provide criteria
    for counting beneficiaries based on the amount of accrued benefits or
    the degree of exposure to compact-provided services. As a result, it is
    unclear whether MCC's transaction teams would count as equal
    beneficiaries a person using a compact-related benefit once and a
    person using the same benefit regularly or whether MCC would apply an
    equivalence scale to adjust for varying use of the benefit. For
    example, for El Salvador education programs, MCC first counted as
    beneficiaries all individuals entering the programs--assuming they
    would experience some income increases, and later revised the method to
    include only individuals completing the programs, assuming they would
    experience measurable income increases. The transaction team's choice
    of method for defining beneficiaries, in turn, affects the compact-
    level projections of impact on income and poverty. For example, MCC's
    estimate of the number of persons to be lifted out of poverty by the
    compact as a whole ranges from 145,000 in the first case to 83,000 in
    the second, depending on the assumptions used. (See app. IV for more
    details.)

    According to MCC officials, MCC is in the process of making its
    guidance regarding program beneficiaries more operational by
    establishing criteria for defining beneficiaries. The officials also
    noted that increases in income are considered a starting point for
    measuring economic gains and therefore determining the number of
    compact beneficiaries.

    Conclusions:

    As MCC works in some of the world's poorest countries, it has taken on
    the difficult challenge of projecting its compacts' likely ERR, and
    economic growth and poverty effects. However, our analysis shows that
    analytic errors affected the results, and the choice of analytical
    method can change the results of economic projections. Without a
    consistent approach and preferred methods for its due diligence
    economic analyses, MCC's transaction teams and individual team members
    have used different methods to identify compact results. This
    heterogeneity means that different teams could reach different
    conclusions about the worthiness of individual projects or compacts
    based on the method chosen. MCC has refined its guidance over time, and
    continues to do so. Further refinements that include formal procedures
    for reviewing the results of due diligence economic analyses and
    establishing a consistent approach with preferred methods--which could
    be modified if required by specific country conditions--would help MCC
    reduce the likelihood of errors and provide a common MCC lens for
    projecting compacts' ERRs and impacts. This in turn would enhance the
    reliability and comparability of the information MCC uses internally
    for decision making as well as the information it provides to Congress
    and the public for oversight of MCC's activities.

    Recommendations for Executive Action:

    To improve the reliability and comparability of its projected ERR and
    economic impacts, we recommend that the CEO of MCC take the following
    actions:

    * Adopt and implement written procedures for a secondary independent
    review of the methods and results of its economic analyses.

    * Improve MCC's guidelines by identifying a consistent approach with
    preferred methods for projecting compacts' impact on income and
    poverty.

    Agency Comments and Our Evaluation:

    MCC provided comments regarding a draft of this report, which we have
    reprinted, with our response, in appendix V. MCC also provided
    technical clarifications, which we have incorporated as appropriate.

    In commenting on a draft of this report, MCC concurred with our
    recommendations and outlined related steps that it is taking--including
    developing standard practices and templates, initiating an independent
    peer review process, and posting its economic analyses on the Internet.
    MCC commented that there is not a single cost-benefit standard practice
    and that in many cases the inconsistencies we identified were
    technically appropriate. We acknowledge that there is not a single
    standard practice; however, we maintain that consistent analytic
    approaches are needed to ensure the reliability of MCC's analyses.
    According to MCC, the choice of analytic time frames was in some cases
    based on the judgment of the individual lead economist rather than on
    established criteria. The steps that MCC has stated it is taking--using
    a default time frame for analyses and subjecting the time frame
    decision to peer review--will help to enhance the comparability across
    compacts of the time frame of ERR calculations. MCC also stated that it
    disagreed with what it saw as our report's implication that ERRs are
    disconnected from income and poverty effects. However, we assert that
    ERRs do not provide information about MCC's impact on any specific
    population group, including the poor. Also, ERRs are a relative measure
    of benefits in relation to costs that can fluctuate owing to changes in
    costs alone--and therefore cannot be considered absolute measures of
    impact on income.

    We are sending copies of this report to interested congressional
    committees as well as the Chief Executive Officer of MCC. We will also
    make copies available to others upon request. In addition, this report
    will be available at no charge on the GAO Web site at [hyperlink
    http://www.gao.gov].

    If you or your staff have any questions about this report, please
    contact David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact
    points for our Offices of Congressional Relations and Public Affairs
    may be found on the last page of this report. GAO staff who made major
    contributions to this report are listed in appendix VI.

    Sincerely yours,

    Signed by:

    David Gootnick:
    Director:
    International Affairs and Trade:

    [End of section]

    Appendix I: Objectives, Scope, and Methodology:

    At the request of the chairman of the House Committee on Foreign
    Affairs, we assessed the Millennium Challenge Corporation's (MCC)
    economic analyses, including its projections of the compact's economic
    rate of return (ERR), changes in income and poverty, and the number of
    beneficiaries. To carry out this review, we reviewed MCC compacts with
    four countries: Armenia, El Salvador, Lesotho, and Mozambique. We
    selected these countries based on their percentage of total compact
    funding, the recentness of the compact, and geographical
    representation. These four compacts make up a total of about $1.56
    billion in compact assistance. When we began our work in August 2007,
    this represented about 41 percent of the $3.85 billion MCC had set
    aside for 13 compacts[Footnote 42] and included the most recent
    compacts signed in Eurasia, Latin America, and Africa. We chose to
    include two African countries in acknowledgement of MCC's focus on
    Africa, which accounted for 7 of MCC's 13 compact countries and more
    than half of the total compact value.

    MCC's compact-level projections are influenced by the specific data
    points and assumptions used in project-and activity-level analysis. We
    determined that the data we used were sufficiently reliable for the
    purposes of our analysis; however, within our scope of work, we did not
    independently evaluate the thousands of data points and assumptions at
    the project and activity levels or the sensitivity analyses used in
    MCC's economic analyses and therefore are not assessing all aspects of
    the validity of MCC's ERR or impact projections. We also did not assess
    MCC's progress in implementing these compacts, and therefore its
    progress toward achieving projected compact results.

    To assess MCC's compact ERR projections, we reviewed MCC's public and
    internal documents for statements of compact and project ERR. When we
    began our review, MCC had published the compact and project ERRs for
    only Armenia.[Footnote 43] However, for all four countries, MCC stated
    ERRs in the internal investment memo. Next, we consulted MCC's guidance
    for ERR analyses as well as the spreadsheets provided by MCC to support
    its calculations of ERR and document its work. We reviewed MCC guidance
    on calculating ERR and minimum ERRs and compared MCC's internal
    documentation and spreadsheets to elements of this guidance. We
    examined the spreadsheets to determine how MCC aggregated project-level
    ERRs into one compact-level ERR for the four countries in our review
    and identified the two approaches---summing net benefits and using
    weighted averages--that MCC used. To determine the effects of MCC's
    alternative approach on the compact ERR, we used MCC's cost and benefit
    data to calculate the alternative ERRs. We also used these data to
    calculate compact-and project-level ERRs over different time horizons
    to explore the sensitivity of the ERRs to differing time horizons. To
    determine how MCC approached same-sector projects in different
    compacts, we studied road and water projects because these were each in
    three of the four compacts. We then compared the broad approaches MCC
    used in assessing the costs and benefits associated with these
    projects. We interviewed MCC economists and other officials regarding
    MCC's ERR analysis to further discuss MCC's approaches and clarify
    aspects of MCC's analysis. We also consulted Office of Management and
    Budget (OMB) guidance and GAO guidance[Footnote 44] on establishing and
    implementing internal controls to inform our assessment of MCC's
    processes for developing and maintaining information for the purpose of
    management decision making.

    To assess MCC's projections of the number of compact beneficiaries, and
    changes in income and poverty, we first compiled and analyzed MCC's
    public documents for statements of compact impact. These documents
    included MCC's compacts, compact summaries, congressional notifications
    and budget justifications, and MCC's annual reports. MCC makes all of
    these documents available online.[Footnote 45] We identified statements
    that fell into four categories of economic projections: impact on
    national or regional income, impact on national or regional gross
    domestic product (GDP) growth rate, impact on national or regional
    poverty, and number of compact beneficiaries. MCC reviewed and
    concurred with our compilation and summary of these statements of
    compact impact. We also reviewed MCC's guidance on projecting these
    economic impacts. We reviewed the spreadsheets for each compact that
    MCC used to conduct and document its economic analyses. After an
    initial examination of these spreadsheets, we met with MCC economists
    and other officials to discuss MCC's methods and calculations for
    projecting compact impact. MCC officials also provided responses to our
    questions in written form. In these responses, MCC revised its initial
    calculations of its impacts in Armenia, El Salvador, and Mozambique and
    also presented alternative methods. We then reviewed the updated
    information that MCC submitted to us and calculated the magnitude of
    the difference in MCC's original statements and those supported by its
    revised analyses. In the case of Lesotho, we reviewed the World Bank
    Country Economic Memo regarding the economic growth model that MCC used
    to estimate compact impact in Lesotho. We also reviewed World Bank
    country assistance strategies and International Monetary Fund Article
    IV consultation reports and Poverty Reduction Strategy Papers to
    improve our contextual understanding of each country's compact program.
    Finally, we consulted OMB guidance and GAO guidance on establishing and
    implementing internal controls to inform our assessment of MCC's
    internal processes.

    We conducted this performance audit from August 2007 to June 2008, in
    accordance with generally accepted government auditing standards. Those
    standards require that we plan and perform the audit to obtain
    sufficient, appropriate evidence to provide a reasonable basis for our
    findings and conclusions based on our audit objectives. We believe that
    the evidence obtained provides a reasonable basis for our findings and
    conclusions based on our audit objectives.

    [End of section]

    Appendix II: MCC Minimum Acceptable ERRs:

    MCC has issued two definitions of the minimum acceptable ERR, which MCC
    refers to as the hurdle rate, for compacts and projects. In its initial
    guidelines, issued in April 2005, MCC defined the hurdle rate as the
    average of the country's real growth rates for the previous 3 years.
    [Footnote 46] The January 2006 guidelines did not set a hurdle rate.
    The November 2006 guidelines defined the hurdle rate as the greater of
    (1) two times the average real growth rate of GDP for the country for
    the most recent 3 years for which data are available or (2) two times
    the average real growth rate of GDP for all of the MCC eligible
    countries for each country for the most recent 3 years for which data
    are available.[Footnote 47] The November 2006 guidelines also state
    that the hurdle rate may not be higher than 15 percent.

    In setting the hurdle rate for each compact that we reviewed, MCC
    applied the definition of the rate from the guidelines current at the
    time. For Armenia and Mozambique, MCC used its April 2005 guidance. For
    El Salvador, according to MCC officials, and for Lesotho, MCC used the
    definition in the November 2006 guidelines. (See table 2.)

    Table 2: MCC Compact Hurdle Rates and Hurdle Rate Definition:

    Compact: Armenia;
    MCC hurdle rate[A]: 12.5 percent;
    Hurdle rate definition: April 2005 guidance: The country's average real
    growth rate for the past 3 years.

    Compact: Mozambique;
    MCC hurdle rate[A]: 8.76 percent;
    Hurdle rate definition: April 2005 guidance: The country's average real
    growth rate for the past 3 years.

    Compact: El Salvador;
    MCC hurdle rate[A]: 10.8 percent;
    Hurdle rate definition: November 2006 guidance: Based on two times the
    average real growth rate of GDP for all of the MCC eligible countries
    for each country for the most recent 3 years for which data are
    available.

    Compact: Lesotho;
    MCC hurdle rate[A]: 10.8 percent;
    Hurdle rate definition: November 2006 guidance: Based on two times the
    average real growth rate of GDP for all of the MCC eligible countries
    for each country for the most recent 3 years for which data are
    available.

    Source: MCC documents.

    [A] For Armenia, Mozambique, and Lesotho, the hurdle rate was stated in
    the investment memo. The investment memo for El Salvador did not state
    the hurdle rate, but MCC officials reported that a hurdle rate of 10.8
    percent, based on the November 2006 definition, was applied.

    [End of table]

    In two cases, the ERRs that MCC initially calculated did not meet the
    compact hurdle rates. MCC originally calculated the El Salvador
    Community Infrastructure Project ERR as slightly below the hurdle rate
    at 10.4 percent over 25 years. MCC's investment memo for Lesotho stated
    that the ERR for the Rural Water Project was under 6 percent.
    Subsequent revisions to the Lesotho analysis reduced the projected ERR
    to less than 1 percent. The Lesotho investment memo discusses the 6
    percent ERR but notes a different view within the transaction team that
    other benefits are not captured by the economic analyses.

    [End of section]

    Appendix III: Compact ERRs:

    We assessed the sensitivity of MCC's compact-level ERRs to the use of
    varying time frames and varying methods for calculating the compact-
    level ERR. We found in each case that the variance does affect the
    results, but does not change the ERRs to below the applicable hurdle
    rate.

    Time Frames:

    MCC used different time frames to calculate and report compact ERRs.

    * For Armenia, MCC calculated and reported a 20-year time frame for the
    compact and both of its projects.

    * For El Salvador, MCC calculated a 25-year ERR for most projects and
    for some projects reported the ERR as a 25-year ERR in its investment
    memo. MCC officials explained that their El Salvador country
    counterparts originally performed the analysis over 25 years and that
    MCC's lead economist determined this to be a reasonable approach.

    * For Lesotho, MCC used a 20-year ERR for all but one ERR calculation
    but did not state the time frame of the calculations in the investment
    memo.

    * For Mozambique, MCC used a 24-year ERR for most compact projects.
    According to MCC officials, MCC intended to present a 25-year ERR, but
    a delay in the beginning of compact implementation pushed the compact's
    time horizon 1 year into the future and reduced the time frame for the
    analysis to 24 years. The Mozambique investment memo does not state the
    number of years used for the time frame of the ERR calculation.

    Table 3 summarizes our comparison of the compact hurdle rates with the
    ERRs that MCC reported and our calculations of the 20-year ERRs. Our
    analysis shows that applying a 20-year time frame in place of varying
    time frames does not significantly affect the results of the ERR
    calculations by lowering it below the hurdle rate.

    Table 3: Comparison of MCC Compact ERRs Stated in Investment Memo with
    ERRs Calculated Using a 20-Year Time Frame:

    Armenia:
    Hurdle rate: 12.5 percent;
    ERR reported in investment memo[A]: 25 percent over 20 years;
    GAO calculation: 20-year ERR[B]: 27.4 percent.

    El Salvador[C]:
    Hurdle rate: 10.8 percent;
    ERR reported in investment memo[A]: 21 percent over 25 years;
    GAO calculation: 20-year ERR[B]: 17.0 percent[D].

    Lesotho:
    Hurdle rate: 10.8 percent;
    ERR reported in investment memo[A]: 16.3 percent, period of years not
    specified;
    GAO calculation: 20-year ERR[B]: 15.9 percent[E].

    Mozambique:
    Hurdle rate: 8.76 percent;
    ERR reported in investment memo[A]: 19.6 percent, period of years not
    specified;
    GAO calculation: 20-year ERR[B]: 17.2 percent.

    Source: GAO analysis of MCC economic analyses.

    [A] In some cases, MCC's underlying spreadsheets stated different ERRs
    than those reported in the investment memo. However; for Armenia, the
    revision to a 27.4 percent ERR occurred after the investment memo. We
    based our calculations on the spreadsheet calculations of the ERRs: for
    Armenia, 27.4 percent over 20 years; for El Salvador, 20.6 percent over
    25 years; for Lesotho, 16.4 percent over 20 years; and for Mozambique,
    18.7 percent over 24 years.

    [B] These alternative ERRs use the method used by MCC for the original
    ERR--that is, weighted averages for El Salvador and Lesotho and sums of
    net benefits for Armenia and Mozambique.

    [C] MCC provided updated spreadsheets for its education and water and
    sanitation projects that reduced some expected benefits and incorrectly
    entered some compact administration costs for another project. Using
    the weighted average method originally used by MCC, the ERR with these
    revisions would be 16.3 percent at 20 years.

    [D] The formula used to calculate the ERR for the Productive
    Development project in El Salvador returns an error when used to
    calculate the ERR for 20 years. However, since the total net benefits
    for the project at 20 years are negative, the ERR also would be
    negative. We used a zero ERR over 20 years in place of a negative ERR
    for this project in calculating the compact ERR. The total net benefits
    and ERR for the Productive Development project become positive at 21
    years.

    [E] MCC originally calculated the Lesotho compact-level ERR using a
    weighted average of ERRs, some of which did not match the ERRs in its
    underlying spreadsheet calculations and the investment memo. We have
    used the ERRs in these underlying spreadsheets to calculate the 20-year
    ERR. MCC also originally used a 30-year ERR for the Metolong Dam
    project to calculate the 20-year compact-level ERR. We have used the 20-
    year ERR for the Metolong Dam for this 20-year compact ERR calculation.

    [End of table]

    Methods:

    Figure 8 illustrates the two different methods--summing net benefits
    and calculating a weighted average--that MCC used to determine a
    compact-level ERR.

    Figure 8: MCC's Alternative Methods for Calculating Compact ERR:

    [See PDF for image]

    This figure is an illustration of MCC's alternative methods for
    calculating Compact ERR, as follows:

    Alternative 1: Sum of net benefits across all projects, for each year:

    Net benefits project A, plus Net benefits project B, plus Net benefits
    project C, equal Compact net benefits; Calculate compact ERR, yields
    Compact ERR.

    Alternative 2: Weighted average of project ERRs:

    ERR Project A, times Cost share A, equals Weighted project ERR A;
    ERR Project B, times Cost share B, equals Weighted project ERR B;
    ERR Project C, times Cost share C, equals Weighted project ERR C;
    Combined to calculate overall weighted average, yields Compact ERR.

    Source: GAO synthesis of MCC information.

    [End of figure]

    We compared the results of the compact ERR calculation using different
    methods and determined that the choice of method affects the results,
    but does not reduce the ERR below the hurdle rate. See table 4 for
    details of our analysis.

    Table 4: Compact ERR Using Alternative Methods:

    Country: Armenia;
    ERR reported in investment memo: 25 percent over 20 years;
    ERR from underlying spreadsheets[A]: 27.4 percent over 20 years;
    ERR using alternative method: 25.6 percent over 20 years;
    Difference: - 1.8.

    Country: El Salvador;
    ERR reported in investment memo: 21 percent over 25 years;
    ERR from underlying spreadsheets[A]: 20.6 percent over 25 years;
    ERR using alternative method: 19.6 percent over 25 years;
    Difference: -1.

    Country: Lesotho;
    ERR reported in investment memo: 16.3 percent, period of years not
    specified;
    ERR from underlying spreadsheets[A]: 16.4 percent over 20 years;
    ERR using alternative method: 15.4 percent over 20 years;
    Difference: -1.

    Country: Mozambique;
    ERR reported in investment memo: 19.6 percent, period of years not
    specified;
    ERR from underlying spreadsheets[A]: 18.7 percent over 24 years;
    ERR using alternative method: 17.3 percent over 24 years;
    Difference: -1.4.

    Source: GAO analysis of MCC economic analyses:

    [A] The ERR in MCC's underlying spreadsheets for Armenia, Lesotho, and
    Mozambique was different from that stated in its investment memos.
    However; for Armenia, the revision to a 27.4 percent ERR occurred after
    the investment memo. In order to determine the ERRs from an alternative
    method, we needed to use these spreadsheets; therefore they are the
    appropriate comparison for determining the magnitude of the change in
    results from using the alternative method. El Salvador's reported ERR
    was rounded in the investment memo.

    [End of table]

    [End of section]

    Appendix IV: Impact of Alternative Beneficiary Counts on El Salvador
    Compact Impact Projections:

    During the course of our engagement, MCC changed its method for
    determining the beneficiaries and benefits of education projects in El
    Salvador. In its initial calculations for both projects, MCC assumed
    that all students entering the programs would experience an increase in
    income. In its revised calculations for the formal education project,
    MCC assumed that all nongraduates would have zero income increases--an
    underestimate of the impact--because of lack of data. For the informal
    education project, MCC estimated an income increase of more than 200
    percent for those who obtain employment, based on a study conducted by
    the training institute in El Salvador. However, MCC assumed that many
    entering students would not complete the program or obtain employment
    and their income would therefore not increase. (See table 5.)

    Table 5: MCC's Compact-Level Impact Estimates for El Salvador, with
    Alternative Assumptions for Estimating Beneficiaries of Education
    Projects:

    Results of compact-level analysis:

    Incomes in the region: 5 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income: 18
    percent increase;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: 10 percent increase.

    Incomes in the region: 10 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income: 26
    percent increase;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: 13 percent increase.

    Annual per capita income of beneficiaries: 5 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income:
    $123 increase;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: $73 increase.

    Annual per capita income of beneficiaries: 10 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income:
    $189 increase;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: $97 increase.

    Number of persons for whom poverty is alleviated: 10 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income:
    145,000 persons;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: 83,000 persons.

    Poverty rate in the Northern Zone: 5 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income: 10
    percentage point decrease;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: 6 percentage point decrease.

    Poverty rate in the Northern Zone: 10 years;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals entering the program will experience increase in income: 15
    percentage point decrease;
    Assumption for estimating beneficiaries of education projects[A]:
    Individuals completing the program[B] will experience increase in
    income[C]: 8 percentage point decrease.

    Source: GAO analysis of MCC data.

    Note: These figures also reflect MCC's corrections of formula errors.

    [A] MCC's compact with El Salvador comprises a formal education project
    and an informal education project. We consider these two projects
    jointly for the purposes of this summary table.

    [B] For the informal education project, MCC estimated the number of
    individuals who complete the program and obtain employment.

    [C] The revised projections also reflect MCC changes to (1) the number
    of beneficiaries of the productive development project, and (2)
    projected income increases resulting from the water and sanitation
    project.

    [End of table]

    [End of section]

    Appendix V: Comments from the Millennium Challenge Corporation:

    Note: GAO comments supplementing those in the report text appear at the
    end of this appendix.

    Millennium Challenge Corporation:
    Reducing Poverty Through Growth:
    875 Fifteenth Street NW:
    Washington, DC 20005-2221:
    (202)521-3600:
    (202521-3700 (Fax):
    [hyperlink, http://www.mcc.gov]:

    May 23, 2008

    Mr. David B. Gootnick:
    Director, International Affairs and Trade:
    U.S. Government Accountability Office:
    441 G Street, NW:
    Washington, DC 20001:

    Dear Mr. Gootnick:

    Thank you for the opportunity to respond to GAO's draft report, MCC:
    Independent Reviews and Consistent Approaches Will Strengthen
    Projections of Program Impact.

    MCC appreciates GAO's recognition of the ground-breaking nature of
    MCC's projections of program impact, reflected in the report's
    conclusion: "As it works in some of the world's poorest countries, MCC
    has taken on the difficult challenge of projecting its compacts' likely
    ERR, and economic growth and poverty effects." MCC concurs with GAO's
    overall recommendations for enhanced guidance and a secondary
    independent review of economic analyses. Indeed, as GAO notes, MCC has
    already "taken steps to standardize elements of its economic analyses
    and centralize its records management" in addition to establishing an
    independent peer review process.

    Independent Reviews:

    Economic analysis, conducted for the purpose of informing MCC
    investment decisions, already undergoes a series of reviews as part of
    our standard practice. Much of the initial analysis is performed by
    professional counterparts in our partner countries and by private
    consultants hired by them or by MCC. In every case, MCC economists
    represent an independent review of that preliminary analysis.

    MCC supports an additional layer of review and, as the report notes,
    has already established an independent peer review process. This
    process, which is managed by the Chief Economist, provides a review of
    the models, formulae and parameters used to estimate the expected
    impact of programs. MCC has already used the new process to review
    forecasts for one compact program currently under consideration, and
    has begun the process for a second country. [See comment 1]

    MCC has taken the unprecedented step of making our economic analysis
    accessible on our public website, opening our work to the broader
    scrutiny of interested NGOs, academics, and private analysts, which we
    believe will generate useful feedback further strengthening our
    analysis. MCC has posted the economic models and resulting impact
    estimates for nine countries, along with explanatory descriptions and
    documentation. We will post the analysis for all other compacts before
    the end of this fiscal year. Independent observers, including senior
    officials at Center for Global Development and Bread for the World
    Institute, have lauded this initiative as setting a new standard for
    transparency in government. [See comment 1]

    Consistency in Technical Approaches:

    The report recognizes that "MCC has refined its guidance over time, and
    continues to do so." MCC is still a new and evolving agency, and we are
    continually refining our standard procedures, for economic analysis and
    in other areas. MCC's rigorous, transparent use of economic analysis to
    estimate the cost-effectiveness of potential interventions is
    exceptional within the international development community and U.S.
    foreign assistance agencies.

    We endorse GAO's conclusion that "establishing a consistent approach
    with preferred methods - which could be modified if required by
    specific conditions - would help MCC to reduce the likelihood of
    errors." MCC has an effort underway to develop standard practices and
    baseline templates for a number of core sectors prevalent in past
    proposals, including roads, water services & sanitation, and
    agriculture & irrigation. We are also considering such templates for
    education, land reform, and micromicrofinance. [See comment 1]

    Placing Consistency in Context:

    GAO analyzed more than 20 projects' economic models, most of which
    comprise numerous spreadsheets incorporating data from reams of
    academic research, country sources, and professional judgment. The
    report cited a number of "inconsistencies" in MCC practices, and GAO
    identified some as minor, with trivial effect on our estimates. In many
    cases, these "inconsistencies" are technically appropriate variations
    of standard practices that can be explained by the specific country
    context. [Footnote 48]

    There is simply no single cost-benefit "standard practice," because
    many of the technical details are by necessity context specific. In
    some cases, the data available allow reasonable estimation using
    different models, and MCC needs the flexibility, as the GAO report
    notes, to apply professional judgment as to when the attainment of
    uniformity in practice is either not cost-effective or will yield
    perverse results. For example, when existing data allow the use of a
    different model and the collection of new data needed for the standard
    model would require both significant time and cost, MCC might find the
    alternative model acceptable. Similarly, when application of a standard
    time horizon does not appropriately reflect a project whose useful life
    is either shorter or longer, MCC will prefer to "deviate" [See comment
    2] from consistent practice rather than inaccurately estimate the
    economic impact of the proposed investment. In all such cases, however,
    MCC will fully subject these decisions to peer review and document them
    for public viewing and any subsequent external assessment. MCC will
    also use sensitivity analysis to explore the implications of
    alternative assumptions or parameters. [See comment 1]

    MCC's ERRs Estimate the Impact on Local Incomes:

    GAO differentiates between ERRS and what the report refers to as MCC's
    "projections of compacts' impact on income and poverty," which implies
    that our ERRS are an aggregate measure of impact disconnected from
    welfare levels experienced by the low-income residents in our partner
    countries. This characterization is based on a common misunderstanding
    about how MCC calculates ERRs and what those numbers represent.

    MCC follows standard cost-benefit practices in most ways, but includes
    only incremental increases in incomes earned by households and domestic
    firms. By excluding other possible benefit streams that do not affect
    domestic incomes, MCC's ERRs represent a direct estimation of the
    magnitude by which local incomes will rise as a result of the MCC
    program. When MCC reports high returns for our projects, these
    estimates do not reflect a broad impersonal measure of economic
    activity, but rather a much more tangible estimate of the project's
    effect on people's lives. [See comment 3]

    Conclusion:

    MCC uses economic analysis and other technical work to direct taxpayer
    funds to investments that will generate significant benefits in well-
    governed developing countries. MCC's performance of cost-benefit
    assessments for virtually all proposed investments, posted on our
    public website, is unprecedented. This assessment represents a critical
    tool for accountability. For every proposed project, MCC assesses
    whether it is a wise investment of American taxpayer funds that will
    generate ample returns for our intended beneficiaries.

    It is instructive to compare MCC practices to those found in other
    foreign assistance agencies around the world. Most other aid agencies
    rarely, if ever, subject their projects to such technical scrutiny. MCC
    does it as a matter of practice, and our practices have generally been
    both reliable and transparent.

    Both our use of economic analysis as a tool for decision-making and our
    openness to public review are critical parts of the MCC model. MCC has
    established a new standard for project efficacy and transparency, and
    we will continue to review and enhance our practices to meet the high
    standards we have set for ourselves.

    Sincerely,

    Signed by:

    Rodney G. Bent:
    Deputy CEO:
    Millennium Challenge Corporation:

    [End of letter]

    The following are GAO's comments from the Millennium Challenge
    Corporation letter, dated May 23, 2008.

    GAO Comments:

    1. We recognize MCC's constructive responses to the issues identified
    in the course of our audit--including posting economic analyses on the
    Internet, developing standard practices and templates, and initiating
    an independent peer review process.

    2. We acknowledge that there is not a single standard practice;
    however, we maintain that consistent analytic approaches are needed to
    ensure the reliability and comparability of MCC's analyses. According
    to MCC, the choice of analytic time frames was in some cases based on
    the judgment of the individual lead economist rather than on
    established criteria. The steps that MCC has stated it is taking--using
    a default time frame for analyses and subjecting the time frame
    decision to peer review--will help to enhance the comparability of its
    of ERR calculations across compacts.

    3. We disagree that MCC's ERRs estimate the project's effect on
    people's lives. ERRs do not provide information about MCC's impact on
    any specific population group, including the poor. ERRs also are a
    relative measure of benefits in relation to costs and can fluctuate
    owing to changes in costs alone. For example, if costs increase and the
    income benefits remain the same, the ERR would decrease. Therefore,
    ERRs cannot be considered an absolute measure of income benefits.

    [End of section]

    Appendix VI: GAO Contact and Staff Acknowledgments:

    GAO Contact:

    David B. Gootnick, Director, 202-512-3149 or gootnickd@gao.gov:

    Staff Acknowledgments:

    In addition to the person named above, Emil Friberg, Jr. (Assistant
    Director), Todd M. Anderson, Gergana Danailova-Trainor, Reid Lowe,
    Michael Simon, and Seyda Wentworth made key contributions to this
    report. Also, C. Etana Finkler, Ernie Jackson, and Tom McCool provided
    technical assistance.

    [End of section]

    Footnotes:

    [1] An MCC compact is an agreement between the U.S. government, acting
    through MCC, and the government of a country eligible for MCC
    assistance. MCC's authorizing legislation, Public Law 108-199, limits
    compact duration to no more than 5 years.

    [2] The due diligence review also includes an evaluation of countries'
    consultative processes used to develop the proposal, donor
    coordination, and environmental and social impact, among other things.

    [3] Economic rate of return is the expected annual average return to
    the countries' firms, individuals, or sectors for each dollar that MCC
    spends on the project. For example, if MCC spent $100,000 on a project
    in year 1 and expected that the project would yield net benefits of
    $120,000 in year 2, the project's ERR for year 1 would be (120,000-
    100,000)/100,000=0.2, or 20 percent.

    [4] Transaction teams comprise MCC staff, personnel from other U.S.
    agencies, and consultants.

    [5] The investment committee consists of MCC's Chief Executive Officer
    (CEO), vice presidents, and other senior officials. The committee
    reviews the memo and decides whether to recommend proceeding to compact
    negotiations.

    [6] GAO, Millennium Challenge Corporation: Vanuatu Compact Overstates
    Projected Program Impact, [hyperlink, http://www.gao.gov/cgi-
    bin/getrpt?GAO-07-909] (Washington, D.C.: July, 11, 2007).

    [7] An internal control is an integral component of an organization's
    management that provides reasonable assurance that the agency is
    achieving: effectiveness and efficiency of operations, reliability of
    financial reporting, and compliance with applicable laws and
    regulations. For OMB guidance, see OMB circular A-123. For GAO
    guidance, see Standards for Internal Control in the Federal Government,
    [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1]
    (Washington, D.C.: November 1999).

    [8] We could not review MCC's calculations for the Lesotho compact,
    because MCC based its projections on a previous World Bank economic
    growth model rather than its own calculations.

    [9] MCC first issued guidelines for the compact development process in
    April 2005. MCC issued revised guidelines in January 2006 and, most
    recently, in November 2006.

    [10] MCC also must report to Congress prior to obligating funds.

    [11] The Secretary of State serves as MCC Board chair, and the
    Secretary of the Treasury serves as vice-chair. Other board members are
    the U.S. Trade Representative, the Administrator of the U.S. Agency for
    International Development (USAID), the CEO of MCC, and up to four
    Senate-confirmed public members who are appointed by the President from
    lists of individuals submitted by congressional leadership.

    [12] We recently reported on MCC's progress in developing and
    implementing compacts. See GAO, Millennium Challenge Corporation:
    Analysis of Compact Development and Future Obligations and Current
    Disbursements of Compact Assistance, [hyperlink, http://www.gao.gov/cgi-
    bin/getrpt?GAO-08-577R], (Washington, D.C.: Apr. 11, 2008).

    [13] MCC stated that these analyses do not capture all aspects of MCC's
    potential impact, such as the effect of government reforms or policy
    changes.

    [14] MCC's guidelines discuss this type of economic analysis under the
    heading "Beneficiary Analysis." For the purposes of our report, we
    refer to this type of analysis as projections of impact on income and
    poverty or simply impact projections.

    [15] In El Salvador, the water project is a component of the Community
    Infrastructure project.

    [16] This amount includes both obligations and commitments. As of March
    2008, MCC had provided a total of $5.5 billion for all 16 countries
    with signed compacts. When we began our work in August 2007, MCC had
    signed 13 compacts totaling $3.85 billion.

    [17] In some cases, the ERRs we refer to were calculated by MCC at the
    activity level.

    [18] The four compacts' minimum ERRs are 12.5 percent for Armenia, 10.8
    percent for El Salvador, 10.8 percent for Lesotho, and 8.76 percent for
    Mozambique. MCC set the minimum ERR for each compact applying
    definitions contained in the guidelines current at the time. The
    initial ERRs for all but two projects--the El Salvador Community
    Infrastructure project and the Lesotho Rural Water project--met the
    respective minimums for each country. See app. II for further
    discussion of MCC's minimum ERR.

    [19] Specifically, the Rio Lurio-Metoro Road segment ERR is 7.2
    percent, the Namialo-Rio Lurio ERR is 5.9 percent, and the Nampula-Rio
    Ligonha ERR is 6.3 percent.

    [20] MCC's April 2005 guidance stated that project ERRs should be
    defined "over the natural life of that component."

    [21] According to MCC, the urban water analysis was revised after the
    investment memo to phase in costs.

    [22] Salvage value is the estimated value of an asset at the end of its
    useful life.

    [23] MCC used an alternative method for calculating road project
    benefits in El Salvador. Because the existing road was nearly
    impassable, MCC thought its preferred method based on existing traffic
    estimates was a poor predictor of the project's impact. MCC instead
    substituted a measure of increases in land values as a proxy for income
    growth. MCC's transaction team reported both scenarios to the
    investment committee--the ERR would be 24 percent using a land value-
    based measure and between 13.8 percent and 14.7 percent using the
    traffic count method. The ERR in either method exceeds MCC's minimum
    ERR of 10.8 percent.

    [24] MCC's April 2005 guidance noted that overall compact ERRs may be
    calculated by using cost-weighted averages of project components,
    combining cost and benefit flows, or another approach--depending on the
    facts and circumstances of the compact. The January 2006 guidance and
    the November 2006 guidance do not define a procedure for calculating
    compact-level ERR.

    [25] See OMB Circular A-123, "Management's Responsibility for Internal
    Control," revised Dec. 21, 2004; and GAO, Standards for Internal
    Control in the Federal Government, [hyperlink, http://www.gao.gov/cgi-
    bin/getrpt?GAO/AIMD-00-21.3.1] (Washington, D.C.: November 1999).

    [26] MCC's November 2006 Guidelines for Economic and Beneficiary
    Analysis states that poverty may be defined according to country-
    specific definitions, such as the official poverty line, or according
    to international standards such as the World Bank's extreme poverty
    definition of $1.08 per capita per day in purchasing power parity, or
    $2 per day. In making its public statements of poverty impact in the
    countries we examined, MCC used country-specific definitions.

    [27] We reviewed MCC's impact analysis spreadsheets for these three
    compacts. For Lesotho, although MCC economists conducted ERR analyses
    for activities and projects under the compact, MCC ultimately based its
    compact-level projections on a previous World Bank economic growth
    model. As such, we were not able to assess MCC's compact-level impact
    projections for Lesotho.

    [28] In its initial projections of the impact of increased incomes on
    poverty in Mozambique, MCC used two types of income estimates - GDP and
    GDP per capita - to compare with-and without-project scenarios.

    [29] Poverty elasticity measures the extent to which economic growth
    reduces poverty by estimating the percentage change in poverty caused
    by a 1 percent change in income.

    [30] As of May 2008, MCC has posted spreadsheets on its Web site
    showing calculations of ERR for El Salvador projects. These
    spreadsheets do not include MCC's compact ERR calculation or
    calculations of compact impact on income and poverty.

    [31] According to OMB and GAO guidelines, an effective control
    environment for data processing may include edit checks. OMB Circular A-
    123 calls on federal agencies to establish management controls to
    ensure that reliable and timely information is maintained for decision
    making. See OMB Circular A-123, "Management's Responsibility for
    Internal Control," revised Dec. 21, 2004. GAO's Standards for Internal
    Control in the Federal Government cites edit checks as an example of a
    control activity used in information processing [hyperlink,
    http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1] (Washington,
    D.C.: November 1999).

    [32] Income growth is defined as the percentage change of income from
    year to year.

    [33] Specifically, MCC projected that the compact could nearly double
    GDP growth by the end of the 5-year compact implementation period
    (using the International Monetary Fund's baseline of 2.6 percent). MCC
    also stated that the acceleration of GDP growth was expected to
    continue, propelling growth toward 7 percent per annum within 5 years
    after compact completion.

    [34] The World Bank model tracks flows of all transactions, from sector
    to sector, within the economy. In most countries, the economists could
    not use such analysis either because no well-calibrated model exists or
    because the MCC package is not easily incorporated into existing
    models.

    [35] The World Bank, Lesotho Country Economic Memorandum: Growth and
    Employment Options Study, Report No. 35359-LS, Apr. 21, 2005.

    [36] An alternative scenario developed by the World Bank involves an
    approximate $50 million investment in infrastructure by the government
    of Lesotho. MCC did not choose to extrapolate the results of this
    scenario.

    [37] MCC did not publicly report an estimate of poverty impact for
    Lesotho.

    [38] Poverty analysis generally involves measuring economic welfare of
    individuals and constructing poverty lines to determine the number of
    people deemed poor--the poverty rate--and the depth of poverty--the
    poverty gap. Viable measures of economic welfare include, for example,
    income per capita, consumption per some "standardized" adult, food
    share of total expenditures, and nutritional indicators. For each
    compact, MCC measured poverty by the country's headcount poverty rate-
    -that is, the number of people with incomes below the poverty line.

    [39] Cross-country data are data drawn from multiple countries. MCC's
    memorandum additionally noted the possibility of using a mixed approach
    that assumes a nonlinear relationship between poverty and income, using
    different values for specific ranges of poverty.

    [40] MCC's November 2006 guidelines state that impact on poverty should
    be measured in terms of both the poverty rate and the poverty gap.
    However, MCC did not estimate changes in the poverty gap for any of the
    four compacts we reviewed. According to the guidelines, "the poverty
    gap is calculated as the sum of money required to bring all poor
    households up to the poverty line, and the effect of an MCC investment
    on the poverty gap would reflect incremental income to poor households
    in aggregate. The poverty rate, in contrast, would not reflect, for
    example, significant improvements in income levels for households
    remaining below the poverty line." As a result of estimating the impact
    on the poverty rate but not on the poverty gap, MCC estimates its
    compacts' impacts on the number of beneficiaries lifted out of poverty
    but does not evaluate potential impact on the severity of
    beneficiaries' poverty.

    [41] In MCC's public documents, MCC cited the entire population of the
    Northern Zone of El Salvador (850,000 people) as the total number of
    compact beneficiaries. In addition, in its 2006 annual report, MCC
    characterizes these beneficiaries as poor. However, according to MCC's
    investment memo, 450,000 (53 percent) of these people are poor. The
    investment memo further defines poverty in the context of its
    Productive Development Project in two ways: (1) relative poverty as
    defined by El Salvador's General Directorate for Statistics and Census
    and (2) more than half the population of the Northern Zone living on
    less than $2 dollar per day and more than 25 percent living on less
    than $1 per day. However, the investment memo does not specify the
    poverty definition used to generate the estimate of 450,000 poor
    people.

    [42] As of March 2008, MCC had 16 signed compacts.

    [43] These internal documents are restricted from public dissemination
    based on MCC policy, but MCC made them available to us for analysis. As
    of April 2008, MCC had posted its spreadsheets showing calculations of
    project-level ERR for El Salvador projects on its Web site, see
    [hyperlink, http://www.mcc.gov/programs/err/]. As of April 2008, MCC
    has not released the supporting ERR spreadsheets for Armenia, Lesotho,
    and Mozambique, but MCC officials have told us they plan to release all
    such spreadsheets.

    [44] See OMB circular A-123. For GAO guidance, see Standards for
    Internal Control in the Federal Government, [hyperlink,
    http://www.gao.gov/cgi-bin/getrpt?GAO/AIMD-00-21.3.1] (Washington,
    D.C.: November 1999).

    [45] See [hyperlink, http://www.mcc.gov/about/reports/] and [hyperlink,
    http://www.mcc.gov/countries/index.php].

    [46] MCC's April 2005 guidelines noted that the hurdle rate definition
    will be revised based on MCC's subsequent experience and data on the
    experiences of developing countries in general. Although other
    development agencies, such as the World Bank and the Asian Development
    Bank, link their hurdle rate to the notion of opportunity cost of
    capital rather than GDP growth rate, in practice the results are
    similar, in that the resulting hurdle rate generally remains between 10
    percent and 12 percent. According to MCC officials, they considered
    other measures but decided on a method that was tied to economic growth
    and did not require extensive and debatable data analysis--MCC's
    current hurdle rates are generally between 10 percent and 15 percent.

    [47] MCC's November 2006 guidelines note that the hurdle rates will be
    set once per year, using data available in the September edition of the
    International Monetary Fund's World Economic Outlook Database for the 3
    previous years.

    [48] In one example, local counterparts developed initial calculations
    for a road project using a 25-year time horizon instead of the 20-year
    horizon used by MCC in most other countries. MCC's economist reviewed
    the analysis and found the difference unimportant to the outcome. The
    economist decided it was better to accept the high-quality work done
    locally than to insist on revisions solely for the sake of attaining
    uniformity of practice.

    [End of section]

    GAO's Mission:

    The Government Accountability Office, the audit, evaluation and
    investigative arm of Congress, exists to support Congress in meeting
    its constitutional responsibilities and to help improve the performance
    and accountability of the federal government for the American people.
    GAO examines the use of public funds; evaluates federal programs and
    policies; and provides analyses, recommendations, and other assistance
    to help Congress make informed oversight, policy, and funding
    decisions. GAO's commitment to good government is reflected in its core
    values of accountability, integrity, and reliability.

    Obtaining Copies of GAO Reports and Testimony:

    The fastest and easiest way to obtain copies of GAO documents at no
    cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each
    weekday, GAO posts newly released reports, testimony, and
    correspondence on its Web site. To have GAO e-mail you a list of newly
    posted products every afternoon, go to [hyperlink, http://www.gao.gov]
    and select "E-mail Updates."

    Order by Mail or Phone:

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent
    of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
    more copies mailed to a single address are discounted 25 percent.
    Orders should be sent to:

    U.S. Government Accountability Office:
    441 G Street NW, Room LM:
    Washington, D.C. 20548:

    To order by Phone:
    Voice: (202) 512-6000:
    TDD: (202) 512-2537:
    Fax: (202) 512-6061:

    To Report Fraud, Waste, and Abuse in Federal Programs:

    Contact:

    Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]:
    E-mail: fraudnet@gao.gov:
    Automated answering system: (800) 424-5454 or (202) 512-7470:

    Congressional Relations:

    Ralph Dawn, Managing Director, dawnr@gao.gov:
    (202) 512-4400:
    U.S. Government Accountability Office:
    441 G Street NW, Room 7125:
    Washington, D.C. 20548:

    Public Affairs:

    Chuck Young, Managing Director, youngc1@gao.gov:
    (202) 512-4800:
    U.S. Government Accountability Office:
    441 G Street NW, Room 7149:
    Washington, D.C. 20548:

    *** End of document. ***