Millennium Challenge Corporation: Compact Implementation
Structures Are Being Established; Framework for Measuring Results
Needs Improvement (28-JUL-06, GAO-06-805).
In January 2004, Congress established the Millennium Challenge
Corporation (MCC) to administer the Millennium Challenge Account.
MCC's mission is to reduce poverty by supporting sustainable,
transformative economic growth in developing countries that
create and maintain sound policy environments. MCC has received
more than $4.2 billion in appropriations, and, as of May 2006, it
had disbursed $22.4 million to four countries whose signed MCC
compacts have entered into force. For the first three countries
with compact entry into force--Madagascar, Cape Verde, and
Honduras--GAO was requested to examine (1) key aspects that MCC
reviewed, and the criteria it used, in its due diligence
assessments; and (2) the structures that have been established
for implementing the compacts.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-805
ACCNO: A57625
TITLE: Millennium Challenge Corporation: Compact Implementation
Structures Are Being Established; Framework for Measuring Results
Needs Improvement
DATE: 07/28/2006
SUBJECT: Accountability
Developing countries
Economic analysis
Economic growth
Evaluation criteria
Federal corporations
Financial management
Fiscal policies
Performance measures
Program evaluation
Program management
Program goals or objectives
Program implementation
Cape Verde
Honduras
Madagascar
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-805
* Report to the Chairman, Committee on Foreign Relations, U.S. Senate
* July 2006
* MILLENNIUM CHALLENGE CORPORATION
* Compact Implementation Structures Are Being Established;
Framework for Measuring Results Needs Improvement
* Contents
* Results in Brief
* Background
* MCC Principles
* MCC Structure
* Eligibility for MCC Assistance
* MCC's Compact Development
* Compact Signature and Entry-into-Force Time Frames
* MCC Guidance
* MCC Assessed Five Key Aspects of Proposals While Developing Guidance,
but Limitations Affected the Accuracy of Economic Analyses
* MCC Assessed the Consultative Process as Guidance Evolved
* MCC Assessed Project Coherence While Developing Specific Guidance
* MCC Screened Projects for Environmental and Social Impact,
Conditioning Funding on Detailed Assessment
* MCC Evaluated Country Commitment and Capacity and Estimated
Project Impact to Assess Sustainability
* Limitations in Assumptions and Data, as Well as Country
Involvement, Affected the Accuracy of Economic Analyses
* MCC-Approved Compact Implementation Structures Are Not Yet Complete;
Weaknesses in Monitoring and Evaluation Framework May Limit
Measurement of Results
* Countries Manage Projects with MCC Review, but Have Faced Delays
in Staffing Oversight and Management Structures
* MCC Has Oversight Staff in Each Country, while Countries
Retain Management Responsibility
* Oversight and Management Entities Have Faced Staffing Delays
* MCC Has Made Progress in Ensuring Fiscal Accountability, Although
Countries' Systems Are Still Being Developed
* MCC Has Made Progress in Fiscal Accountability at the
Compact Level
* MCA-Madagascar and MCA-Cape Verde Have Implemented Internal
Controls for Fiscal Accountability, but Their Systems Are
Not Yet Fully Developed
* MCA-Madagascar
* MCA-Cape Verde
* Financial Audits for MCA- Madagascar and MCA-Cape Verde
* MCC-Approved Country Procurement Systems Have Effective
Characteristics, but Are Early in Their Implementation
* MCC Allowed Countries to Propose Procurement Agents and
Systems, but Retained Approval Rights
* Countries' Procurement Systems Have Strengths, but MCC and
Country Systems Are Not Fully Established
* MCC Has Established Country Monitoring and Evaluation Frameworks,
but Weaknesses May Limit Measurement of Results
* Countries' Programs Include Monitoring and Evaluation
Frameworks
* Weaknesses in Monitoring and Evaluation Frameworks May Limit
MCC's Ability to Measure Results
* Ensuring Baseline Data Availability and Quality
* Linking Monitoring Plans to Economic Analyses
* Accounting for Uncertainty in Achieving Target Values
* Using Randomized Controlled Trials in Compact Countries
* Conclusions
* Recommendations
* Agency Comments and Our Evaluation
* MCC Actions in Response to April 2005 GAO Recommendations
* MCC Has Developed a Strategic Plan to Enhance Accountability and
Completed a 2006 Performance Plan
* MCC Has Made Significant Progress in Establishing Internal
Controls
* MCC Has Taken Steps to Develop Human Capital, but Does Not Track
Allocation of Human Capital to Key Activities
* MCC Has Taken Steps to Define Corporate Governance
* Scope and Methodology
* MCC Projects in Madagascar and Cape Verde
* Summary of MCC Procurement Agents, Standards, and Provisions in
Madagascar, Cape Verde, and Honduras
* Comments from the Millennium Challenge Corporation
* GAO Comments
* Comments from the Department of State
* GAO Comments
* GAO Contacts and Staff Acknowledgments
Report to the Chairman, Committee on Foreign Relations, U.S. Senate
July 2006
MILLENNIUM CHALLENGE CORPORATION
Compact Implementation Structures Are Being Established; Framework for
Measuring Results Needs Improvement
Contents
Tables
Figures
Abbreviations
July 28, 2006Letter
The Honorable Richard G. Lugar Chairman Committee on Foreign Relations
United States Senate
Dear Mr. Chairman:
In January 2004, Congress established the Millennium Challenge Corporation
(MCC) to administer the Millennium Challenge Account (MCA) for foreign
assistance.1 MCC's mission is to reduce poverty by supporting sustainable,
transformative economic growth in partnership with developing countries
that create and maintain sound policy environments. MCC carries out its
mission by funding projects or activities in developing countries that
demonstrate a commitment to MCA objectives. MCC expects to raise incomes
and lift thousands out of poverty in countries receiving MCC assistance.
MCC has received appropriations for fiscal years 2004 to 2006 totaling
more than $4.2 billion, about $3.8 billion of which has been set aside for
compact assistance; as of May 2006, it had committed $1.5 billion and
disbursed $22.4 million in compact assistance.2 The President has
requested an additional $3 billion in MCC funding for fiscal year 2007.
In April 2005, we reported on MCC's first year of operations, focusing on
its process for determining country eligibility, progress in developing
compacts, coordination with stakeholders, and progress in establishing
management and accountability structures.3 We made several recommendations
to the Chief Executive Officer (CEO) of MCC and to the Secretary of State,
in her capacity as board chair, on ways to improve MCC's strategic
planning, internal controls, and human capital and corporate governance
policies. At that time, MCC's assessment-"due diligence"-of eligible
countries' compact proposals4 had resulted in its signing one compact for
$110 million with Madagascar; by the end of May 2006, MCC had signed
compacts with seven more countries.5 As of the end of May 2006, six
compact countries-Madagascar, Cape Verde, Honduras, Georgia, Vanuatu, and
Nicaragua-had signed the supplemental agreements that MCC requires before
compacts can "enter into force," when MCC begins to disburse funds for
compact implementation. MCC has disbursed funds to four of these six
countries-Madagascar, Cape Verde, Honduras, and Georgia-to begin
implementing their compacts.
At your request, we examine in this report MCC's procedures and structures
for developing and implementing compacts. We focused our work on the first
three countries to sign compacts: Madagascar, Cape Verde, and Honduras.
Specifically, we reviewed
o the key areas that MCC examined in its due diligence assessments of the
three countries' compact proposals and the criteria that MCC used in these
assessments and
o the implementation structures that MCC and the three countries have
established for oversight and management, fiscal accountability,
procurement, and monitoring and evaluation of compacts.
We also report on MCC's response to our April 2005 recommendations (see
app. I). In conducting our work, we analyzed MCC's process for evaluating
eligible country proposals, including MCC's guidance documents for
eligible countries, policies and procedures, economic analysis, and
documentation of its assessment of proposals submitted by Madagascar, Cape
Verde, and Honduras. In addition, we reviewed MCC's compacts with the
three countries and the supplemental agreements required for those
compacts to enter into force.6 We also reviewed the countries'
implementation plans, such as those for monitoring and evaluation. We
supplemented our evaluation of due diligence and implementation documents
with interviews with MCC officials and site visits to Madagascar and Cape
Verde, in January and February 2006. We selected these two countries
because they had advanced further than Honduras in filling key positions
and beginning compact implementation. These site visits provided us with a
firsthand perspective of the status of the program and the challenges
these countries faced in developing their compacts and in beginning to
implement their programs. Furthermore, we examined MCC's planning
documents, policies, and procedures and interviewed senior MCC officials.
We conducted our review from June 2005 through May 2006 in accordance with
generally accepted government auditing standards. (See app. II for
additional details of our scope and methodology.)
Results in Brief
MCC undertook a wide range of activities in its due diligence of the
Madagascar, Cape Verde, and Honduras proposals, while at the same time
developing guidance on key aspects of the countries' proposals. In
conducting its assessments, MCC applied criteria from general guidance
issued in 2004, which was shortly before the countries began developing
their proposals, and from more specific guidance issued in 2005 and 2006.
MCC's analyses of the projects' economic impact had limitations in their
use of assumptions and data and degree of country involvement. We focused
on MCC's assessment of five key aspects: the consultative process during
proposal development, project coherence, environmental impact,
institutional and financial sustainability, and economic analyses of the
projects' impact on economic growth and poverty reduction:
o Consultative process. MCC obtained the views of government, civil
society, and private sector officials to determine whether the countries'
consultative process had been timely, participatory, and meaningful. MCC
officials told us that, on the basis of these assessments, they had
returned proposals that were not sufficiently founded on the consultative
process. However, in assessing the proposals, MCC used criteria contained
in guidance issued after or shortly before the countries submitted their
proposals.
o Project coherence. MCC assessed whether the countries' proposed projects
were linked to their compact goals and whether the projects addressed key
impediments to achieving those goals. MCC's initial published guidance to
the countries did not inform them that projects should be linked to
compact goals. When MCC issued more specific guidance, in November 2005,
it had already signed compacts with all three countries. Generally,
projects were linked to compact goals in the three countries that we
reviewed.
o Environmental and social impact. MCC reviewed the probable environmental
and social impact of proposed projects. For projects that MCC deemed
likely to have a negative impact, it required further assessment as well
as an impact management plan. In conducting its assessments, MCC used
criteria in guidelines issued in March 2005.
o Project sustainability. MCC assessed whether projects could be sustained
institutionally and financially after the compacts expired. MCC reviewed
(1) the countries' commitment to sustaining the projects by making policy
changes and providing financial resources, (2) the countries'
institutional capacity to achieve project objectives, and (3) the
projects' likelihood of being financially self-sustaining based on MCC's
economic analysis.
o Economic analyses. To predict the projects' impact on economic growth,
MCC calculated their economic rate of return-that is, for each dollar that
it spent, the dollar benefit the country is likely to receive. MCC used an
economic model that included data and assumptions about project
beneficiaries' expected behaviors. However, the data and assumptions used
in some of MCC's economic analyses may not accurately reflect the
countries' socioeconomic conditions. As a result, MCC cannot be assured
that it has selected projects that will achieve the compacts' goals. In
the two countries we visited, Madagascar and Cape Verde, country
representatives were not closely involved in MCC's economic analyses. This
limited involvement in the process resulted in countries' having little
understanding of the underlying economic framework on which the compacts
were based.
MCC and the countries have made progress in implementing structures for
oversight and management, fiscal accountability, and procurement for
Madagascar, Cape Verde, and Honduras, but some of these structures are not
yet complete. In addition, although MCC has established monitoring and
evaluation frameworks, the frameworks have weaknesses that could affect
MCC's ability to ensure accountability for results.
o Oversight and management. MCC and the countries have established
structures that allow for country management with MCC review. MCC
maintains a small staff in each country, which refers matters requiring
approval to MCC headquarters in Washington, D.C. Country management
structures include the following: a steering committee, a stakeholder
committee, and a management unit. Staffing for all three country
organizations remained incomplete for months after the compacts entered
into force. This incomplete staffing at entry into force limits the
ability of the countries to achieve their compact objectives within the
fixed time period of the compact.
o Fiscal accountability. MCC has established a framework of internal
controls to incorporate fiscal accountability into MCC-funded projects. In
Madagascar, we found that underlying accountability systems, policies and
procedures, and internal controls are less fully developed than those in
Cape Verde, which built on existing government structures and systems.
Because the accountability systems in both countries are still under
development, both countries face risks in their key financial processes
and activities. In January 2006, the MCC Inspector General (IG) issued
guidelines for the countries' financial audits, and Madagascar and Cape
Verde are preparing for these audits.
o Procurement. In each of the three countries, the MCC-approved systems
have characteristics that typify effective procurement systems-for
example, integrity, openness, and accountability. These systems are still
largely untested, and some MCC and country staff and procedures are not
yet in place.
o Monitoring and evaluation. Compact countries are required to prepare and
implement a monitoring and evaluation framework, including plans that
document the necessary data collection, data quality reviews, analysis,
and interim and final reporting of results. MCC has conditioned some
disbursements on the countries' achieving performance targets. However,
MCC's ability to track and account for results may be limited by
weaknesses in the frameworks related to the availability and quality of
baseline data, ensuring linkage between the economic models and the
monitoring and evaluation plans, and accounting for uncertainty in setting
targets and measuring progress. In addition, although MCC has retained
five U.S. research organizations to independently analyze the results of
MCC compacts, MCC has not completed the research designs needed to use its
preferred methodology of randomized controlled trials prior to project
implementation. Not completing the designs prior to project implementation
potentially limits MCC's ability to perform randomized controlled trials.
We are recommending that the CEO of MCC (1) adopt procedures that ensure
greater involvement of compact country stakeholders in developing the
economic model used to assess projects' likely impact and (2) to the
extent practical and cost-effective, improve MCC's monitoring and
evaluation capabilities by obtaining more accurate and reliable baseline
data, ensuring a clear linkage between MCC's economic analyses and
monitoring and evaluation frameworks, better accounting for uncertainty in
setting targets and achieving outcomes, and ensuring the timely
development of the needed research designs for randomized controlled
trials prior to project implementation.
In commenting on a draft of this report, MCC generally agreed with our
findings, conclusions, and recommendations. The Department of State
commented that it considered some of the findings of our report as
reflecting minor or transitory problems, and that the report should note
the improvements that MCC made during and since the period it covers.
Throughout this report, we have described MCC and compact progress through
May 2006. We have reprinted MCC's and State's comments, with our
responses, in appendixes V and VI. We also incorporated technical comments
from MCC in our report where appropriate.
Background
MCC Principles
In pursuit of its mission to reduce poverty by supporting economic growth,
MCC has identified and defined the following three key principles to guide
its actions:7
o Reward good policy. "Using objective indicators, countries are selected
to receive assistance based on their performance in governing justly,
investing in their citizens, and encouraging economic freedom."
o Operate in partnership. "Countries that receive MCA assistance are
responsible for identifying the greatest barriers to their own
development, ensuring civil society participation, and developing a
multi-year MCC compact."
o Focus on results. "MCA assistance goes to those countries that have
developed well-designed programs with clear objectives, benchmarks to
measure expected results, procedures to ensure fiscal accountability for
the use of MCA assistance, and a plan for effective monitoring and
objective evaluation of results. Programs are designed so that recipient
countries can sustain progress after the funding under the compact has
ended."
MCC Structure
MCC is a government corporation that is managed by a CEO appointed by the
President with the advice and consent of the Senate and is overseen by a
Board of Directors (MCC Board). The Secretary of State serves as board
chair, and the Secretary of the Treasury serves as vice-chair. Other board
members are the U.S. Trade Representative, the Administrator of the U.S.
Agency for International Development (USAID), the CEO of MCC, and up to
four Senate-confirmed public members who are appointed by the President
from lists of individuals submitted by congressional leadership.
Eligibility for MCC Assistance
The Millennium Challenge Act of 2003 requires MCC to select countries as
eligible for MCA assistance each fiscal year. Countries with per capita
income at or below a set threshold may be selected as eligible for
assistance if they pass MCC indicator criteria and are not statutorily
barred from receiving U.S. assistance. MCC uses 16 indicators divided into
three categories: Ruling Justly, Encouraging Economic Freedom, and
Investing in People.8 To be eligible for MCA assistance, countries must
score above the median relative to their peers on at least half of the
indicators in each category and above the median on the indicator for
combating corruption. MCC used these quantitative indicators, as well as
the discretion implicit in the Millennium Challenge Act, to select 17
countries as eligible to apply for MCA compact assistance for fiscal years
2004 and 2005. For fiscal year 2006, MCC identified 23 countries9 as
eligible for assistance-the 17 previously selected and 6 additional
countries, which included lower-middle-income countries eligible for the
first time in fiscal year 2006.10
MCC's Compact Development
After MCC selects eligible countries, they may begin a four-phase process
that can lead to the entry into force of compacts (see fig. 1). Each phase
of this process is discussed after figure 1.
Figure 1: Summary of the MCC Compact Development and Implementation
Process and Status of Countries Eligible to Apply for Compacts, as of June
2006
aMCC must notify congressional appropriations committees 15 days prior to
obligating funds.
bCompact negotiations begin after the MCC investment committee approves a
Consultation Memorandum prepared by the MCC transaction team. The
memorandum is based on the transaction team's determination that the
country proposal has sufficient information to justify entering into
negotiations with the country. MCC must consult with and report to the
appropriate congressional committees 15 days prior to the start of compact
negotiations.
cThe names of newly eligible countries in fiscal year 2006 are italicized.
dThe MCC Board suspended Gambia's eligibility on June 16, 2006.
1.Country proposal development. Eligible countries are invited to submit
compact proposals, which are to be developed in consultation with members
of civil society, including the private sector and nongovernmental
organizations (NGO). The eligible country also identifies an "accountable
entity" to manage the programs funded by MCC.11 Eligible countries
submitting proposals are not guaranteed funding; instead, MCC assesses
proposals through its due diligence review. As of May 2006, 14 of the 17
countries selected as eligible in fiscal year 2004 or 2005, and 1 of the 6
countries selected as eligible for the first time in fiscal year 2006, had
submitted proposals accepted by MCC for due diligence review.
2.MCC's due diligence review. MCC determines whether the proposal that an
eligible country has submitted meets MCC criteria to ensure that proposed
programs will be effective and funds will be well-used. Due diligence
primarily occurs between MCC's acceptances of an "opportunity memo" and an
"investment memo." MCC assembles a transaction team of MCC staff,
personnel from other U.S. agencies, and consultants to conduct a
preliminary assessment of a country's proposal and reports the team's
findings in an opportunity memo to the MCC investment committee. The MCC
investment committee consists of MCC's CEO, vice presidents, and other
senior officials. If the opportunity memo is approved, the transaction
team launches a detailed due diligence review. The team assesses the
country proposal, reports its findings, and makes recommendations based on
its assessment in an investment memo to the MCC investment committee. As
of May 2006, MCC was conducting due diligence analyses of seven eligible
country proposals.
3.Compact negotiation and MCC Board approval. MCC may enter into compact
negotiations with the eligible country before the investment memo is
completed. If compact negotiations are successful, MCC staff formally
submit the compact for MCC Board approval. Once the board approves the
compact, MCC and the eligible country may sign it. As of March 2006, MCC
had signed compacts with 8 of the 17 countries determined eligible in
fiscal years 2004 and 2005. (See fig. 2.) MCC commits the full amount of
the compact funding at signing but obligates and begins to disburse funds
to implement projects only after the compact has entered into force. Under
the Millennium Challenge Act, compacts may remain in force no longer than
5 years. The compacts stipulate that, with limited exceptions, all funds
must be spent during that time.
4.MCC and compact country complete entry-into-force requirements. MCC's
compact with each country identifies the following supplemental agreements
that MCC and the country's accountable entity must complete before the
compact can enter into force.12
o The disbursement agreement sets out the "conditions precedent"13 and
other requirements for disbursements from MCC and redisbursements to any
person or entity. These conditions include performance targets for
projects outlined in the compact.
o The procurement agreement sets forth guidelines for all procurements of
goods, works, and services financed with MCC funding.
o Compact term sheets for supplemental agreements, which vary by country,
and include documents such as a governance agreement, fiscal agent
agreement, form of implementing entity agreement, and form of bank
agreement.
After compacts enter into force, MCC may begin the disbursement of funds
and countries may begin implementing projects. In the first eight
compacts, approximately 53 percent of funding went to transportation and
other infrastructure projects; 22 percent went to agriculture and rural
development; 13 percent went to other project types; and 12 percent went
to program management, monitoring, and evaluation. (See fig. 2.)
Figure 2: MCC-Eligible Countries with Signed Compacts, as of May 2006
Note: Numbers may not add due to rounding of source data. See appendix III
for photographs of projects we visited in Cape Verde and Madagascar.
aThe Access to Markets Project in Benin is a major construction project at
the Port of Cotonou and includes associated studies and institutional
strengthening.
bThe Irrigated Agriculture Project in Armenia includes the repair of
irrigation infrastructure.
cThe Justice Program in Benin includes institutional strengthening and
infrastructure components (construction of new courthouses).
Compact Signature and Entry-into-Force Time Frames
The length of time from country eligibility selection to compact signature
has varied, with proposal development and due diligence generally
requiring the most time (see fig. 3). For the six countries whose compacts
had entered into force as of the end of May 2006, completing the steps
necessary for entry into force after compact signing took approximately 3
to 4 months for Madagascar, Cape Verde, and Honduras; approximately 7
months for Georgia; 2 months for Vanuatu; and about 10 1/2 months for
Nicaragua. For the two countries whose compacts had not entered into force
as of the end of May 2006, 3 months had elapsed since compact signature
for Benin, and 2 months had elapsed for Armenia (see fig. 3).
Figure 3: Number of Days Elapsed from Eligibility to Compact Signature or
Entry into Force, as of May 2006
aBenin and Armenia have not yet entered into force.
MCC Guidance
MCC has issued guidance and policies for its compact development process
in several stages. Before publishing its initial guidance in May 2004, MCC
provided countries with preliminary guidance addressing fiscal
accountability and monitoring and evaluation. Figure 4 shows the evolution
of MCC's published guidance relative to the end of the due diligence
process with the investment memo.
Figure 4: Timeline of MCC Guidance Issuance and Investment Memo Dates
aMCC issued interim Environmental Guidelines on March 4, 2005, and final
Environmental Guidelines on January 23, 2006.
bEconomic analysis guidelines and compact assessment and approval
guidelines were reissued on January 23, 2006.
cConsultative process guidance was reissued on January 6, 2006.
dAccording to MCC officials, MCC provided an initial 2-page guidance to
eligible countries in December 2004, but it did not publish the guidance
on the MCC Web site. MCC issued an updated version of the guidance on
January 6, 2006.
MCC Assessed Five Key Aspects of Proposals While Developing Guidance, but
Limitations Affected the Accuracy of Economic Analyses
MCC undertook a wide range of activities in its due diligence of the
Madagascar, Cape Verde, and Honduras proposals, while at the same time
developing guidance on key aspects of the countries' proposals.14 During
due diligence, MCC primarily considered criteria related to the proposals'
consultative process, project coherence, environmental impact,
institutional and financial sustainability, and economic analyses.15 MCC
generally approved proposals that were based on a consultative process and
returned proposals that lacked adequate consultations; however, MCC did
not publish detailed criteria for the consultative process until 1 year
after selecting the countries as eligible for MCC assistance. In assessing
project coherence, MCC approved projects that were linked to the overall
proposal objectives and rejected projects that were not, although its
assessments used criteria that it had not yet published in its guidance.
Additionally, MCC screened projects for likely environmental impacts and
considered factors important for institutional and financial
sustainability. Finally, MCC conducted economic analyses to assess the
projects' likely impact on economic growth. However, limitations in
assumptions and data may have affected the analyses' accuracy and led MCC
to select projects that would not achieve its goals. Also, a lack of
country involvement in the analyses does not reflect the MCC principle of
working in partnership with countries and may have limited the countries'
understanding of the process.
MCC Assessed the Consultative Process as Guidance Evolved
MCC's due diligence for Madagascar, Cape Verde, and Honduras assessed
whether the countries had consulted with public and private sector and
civil society stakeholders during proposal development. MCC officials told
us that before beginning due diligence for the three countries, they
assessed the consultative process in proposal drafts and, on the basis of
this review, returned proposals that were not sufficiently founded on a
consultative process. For example, according to the officials, MCC did not
accept one country's proposal because, although the country had consulted
with stakeholders, its proposal did not reflect the priorities identified
during the consultative process.
MCC documents indicate that during its due diligence for two of the three
countries, MCC obtained the views of government, civil society, and
private sector officials on how the governments conducted the consultative
process. In addition, MCC assessed any previous experience the country had
had with a consultative process16 For all three countries, MCC also
reviewed factors such as the date, frequency, and locations of the
process. MCC's documentation of these assessments in Madagascar, Cape
Verde, and Honduras indicated that key stakeholders had generally agreed
on the proposed compact priorities. However, the MCC assessments noted
some weaknesses in the Madagascar and Honduras governments' management of
the process:
o In Madagascar, some groups expressed concern to MCC that the government
had provided short notice (less than 10 days) for consultative meetings,
and that this might have limited rural groups' participation.
Additionally, the government did not communicate to participants its
rationale for accepting or rejecting projects. However, despite these
shortcomings, MCC noted "widespread agreement and enthusiasm for
the...primary components of [the] proposal" among the business community,
local and international NGOs, civil society, and donors.
o In Honduras, MCC found, on the basis of discussions with local civil
society organizations and international NGOs, that weaknesses-such as the
large size of meetings-had limited effective participation in the
consultative process for the poverty reduction strategy, which formed the
basis of Honduras's MCC proposal. MCC's assessment also indicated that the
government had not directly asked the consulted groups to identify
obstacles to growth. However, MCC found that these groups concurred with
the priorities identified in the country proposal. According to MCC
officials, they followed up with the Honduran government to address the
weakness noted by MCC in Honduras's consultative process. As a result,
according to MCC documentation, the Honduran government conducted
additional consultative sessions with civil society organizations and
donors.
MCC's due diligence of the countries also noted requirements for
additional consultations during project implementation. For example, in
some cases, countries were to undertake consultations with local
stakeholders to identify project sites and conduct environmental
assessments.
Our discussions with representatives of civil society groups and donors in
Madagascar and Cape Verde indicated that they generally concurred with the
compact proposals.17 In Madagascar, a representative from a key civil
society organization noted weaknesses in the government's conduct of the
consultative process similar to those recorded by MCC. Nevertheless,
stakeholders on the Madagascar advisory council, which includes various
civil society and other organizations, said that the compact proposal
generally accounted for their views, especially in comparison with other
donor programs.
As a relatively new organization, MCC conducted due diligence reviews
while it was developing consultative process guidance:
o Evolving guidance. While developing their proposals, the countries had
access to general criteria in MCC's 2004 guidance; however, in assessing
the proposals, MCC applied the more specific criteria contained in its
detailed 2005 guidance. As figure 5 shows, the 2005 guidance was issued 1
year after MCC announced the eligibility of Madagascar, Honduras, and Cape
Verde. According to MCC's Guidance for Developing Proposals for MCA
Assistance in FY 2004, "...each proposal is expected to reflect the
results of an open consultative process, integrating governmental
interests with those of the private sector and civil society." The 2004
guidance required that proposals include a description of the consultative
process, such as how the proposal takes into account local-level
perspectives of the country's rural and urban poor, including women, and
of private and voluntary organizations and the business community.
Additionally, the guidance required the country to list all key
participants, such as government and nongovernmental officials, who played
a significant role in developing the proposal.
MCC's 2005 Guidance on the Consultative Process more specifically requires
eligible country governments to involve their citizens in identifying
obstacles to economic growth and developing and prioritizing the
development strategies and programs that will be included in the compact
proposal. The 2005 guidance further states that an adequate consultative
process should be timely, participatory, and meaningful. MCC's guidance
also took into account the country's experience in using a consultative
process to develop other national or poverty reduction strategies. If the
compact was built on these consultations, MCC required some additional
consultations to provide justification of country priorities in the MCA
proposal.
Figure 5: Issuance of Key MCC Criteria for the Consultative Process
Relative to Completion of Due Diligence for Madagascar, Honduras, and Cape
Verde
o Incomplete documentation. MCC's documentation of its due diligence for
the three countries presents summary findings, rather than an analysis of
the extent to which the countries consulted with the rural and urban poor.
For example, MCC's due diligence documentation indicates that the
countries' governments included women's groups and rural sector groups in
their consultative process; however, the documentation does not indicate
the extent to which these groups represented the poor. Additionally, in
Madagascar and Honduras, MCC's documentation does not indicate how MCC
assessed the extent to which the consulted groups informed compact
proposal priorities.
MCC Assessed Project Coherence While Developing Specific Guidance
In keeping with MCC's emphasis on results, due diligence for the three
countries also assessed whether proposed projects were linked to one or
more of the country's compact goals, and whether the projects addressed
key impediments or constraints to achieving these goals. On the basis of
its due diligence assessment, MCC rejected projects that were not linked
to key constraints in two of the three countries we reviewed.
Specifically, MCC rejected tourism and preschool education projects
proposed by Honduras because they were not linked to the impediments to
growth that emerged from the consultative process.18 MCC also rejected
projects to construct feeder roads, which connect some watershed areas to
the markets, and provide access to electricity in the rural areas of Cape
Verde, because MCC's due diligence did not indicate that these projects
addressed key constraints in Cape Verde.19
In reviewing MCC's due diligence for the three countries, we found that
MCC did not issue guidance stating that proposed projects should be linked
to the compact goal until after it had concluded its due diligence
assessments. MCC's 2004 guidelines for proposal development broadly
instruct eligible countries to identify priority areas, such as health or
education, and their expected goals for each priority area over the term
of the proposed compact. The guidance also asks the countries to show how
these strategic goals are related to the economic growth and poverty
reduction of the country. MCC's November 2005 MCC Compact Assessment and
Approval Guidelines more specifically indicates that MCC will assess how
the project addresses compact goals. However, MCC issued the
November 2005 guidance after completing its due diligence for the three
countries.20 (See fig. 6.)
Figure 6: Issuance of Key MCC Criteria for Project Coherence Relative to
Completion of Due Diligence for Madagascar, Honduras, and Cape Verde
MCC Screened Projects for Environmental and Social Impact, Conditioning
Funding on Detailed Assessment
MCC's due diligence for the three countries included a review of the
probable environmental and social impact of projects that met its economic
analysis and other criteria.21 For projects that it deemed likely to cause
adverse environmental and social impact, MCC required impact assessments
or environmental analyses, including an impact management plan.
On the basis of its assessment, MCC assigned each project to category A,
B, or C during due diligence to reflect its likely impact and assessments
required.22 For example, MCC assigned all projects in the Madagascar
proposal to category C, because MCC determined that these projects were
not likely to have adverse environmental and social impact. In contrast,
MCC assigned infrastructure projects in Cape Verde's and Honduras's
proposals to category A or B. For example, the highway expansion project
in Honduras was assigned category A, because it involves the clearing of
rights-of-way that will require compensation for more than 200 affected
people. The port expansion project in the Cape Verde proposal was also
assigned to category A, because it entails dredging and construction in
and around an existing port.
Additionally, MCC assessed whether environmental impact assessments had
been conducted for category A or B projects. For projects lacking
environmental impact assessments, MCC conditioned project funding on the
completion of such assessments as well as on the development of mitigation
plans in consultation with affected groups. For projects that other
organizations had assessed for environmental impact, MCC used a U.S.
agency or a contractor to evaluate the assessment and determine its
adequacy. For example, in Honduras, another donor had already conducted
the environmental impact assessment for the highway segments proposed for
MCC funding. As part of MCC due diligence, the U.S. Army Corps of
Engineers reviewed these assessments and made recommendations that were
incorporated in the existing assessments. However, in Cape Verde, the MCC
contractor conducting due diligence found another organization's
assessment of the port project's environmental impact to be inadequate. As
a result, MCC required a new environmental assessment, along with plans to
manage adverse impact, as a precondition for funding the project. MCC has
allocated funds for this analysis in the compact budget for Cape Verde.
MCC's 2004 proposal development guidelines do not address projects'
environmental and social impact. In assessing environmental impact, MCC
applied criteria from the Millennium Challenge Act of 2003, which
prohibits MCC from funding projects that are "likely to cause a
significant environmental, health, or safety hazard." MCC also used
criteria laid out in its March 2005 interim Environmental Guidelines,
which state that MCC will not fund projects that lack the appropriate
screening or analysis for
environmental impact.23 The guidance also states that the country has
primary responsibility for conducting and monitoring environmental
assessments. (See fig. 7.)
Figure 7: Issuance of Key MCC Criteria for Environmental Impact Relative
to Completion of Due Diligence for Madagascar, Honduras, and Cape Verde
MCC Evaluated Country Commitment and Capacity and Estimated Project Impact
to Assess Sustainability
In keeping with its emphasis on sustainable progress, MCC's due diligence
examined whether the three countries' proposed projects could be sustained
after their compacts expired. In assessing project sustainability, MCC
reviewed each country's policy and regulatory environment and commitment
to reforms and financing of future maintenance costs; it also reviewed
expected results from MCC-funded projects. In addition, MCC considered the
countries' institutional capacity to sustain proposed projects as well as
other donors' roles in strengthening countries' capacity.
o Country commitment. For specific projects in Madagascar, Honduras, and
Cape Verde, MCC reviewed policy or regulatory reforms, and host country
financial commitment, and required some changes or commitment as a
precondition to funding. For example, MCC determined that agricultural
projects proposed by Cape Verde and Honduras would not be sustainable
without the governments' commitment to policy reform. To ensure adequate
operations and maintenance of proposed irrigation systems, MCC required
that the Cape Verde government implement fees that reflect the costs of
scarce water resources and recommended that the Honduran government also
institute such reforms. In the case of Madagascar, MCC reviewed policy
reforms in land management and financial sectors that would benefit
MCC-funded activities.
o Country capacity. In Madagascar, MCC determined that the government had
limited institutional capacity to achieve the project objectives of
increasing land security and promoting financial intermediation to
increase rural savings and extension of credit. To build the government's
capacity, MCC budgeted funding for (1) staff recruitment and training at
Madagascar's land management department to support the land security
project and (2) finance, management, and production training for rural
producers and microfinance institutions to support the financial
intermediation project. In addition, MCC considered the role of other
donors in strengthening the countries' institutional capacity. For
example, while assessing road projects, MCC considered the World Bank's
road sector initiative, which includes institutional capacity-building in
Cape Verde. In Honduras, MCC considered World Bank-funded and
Inter-American Development Bank-funded programs intended to develop the
transportation ministry's management capacity and maintenance contracting
capacity.
o Project impact. MCC also relied on the assumptions used in its analysis
of projects' economic impact to determine the sustainability of
agricultural sector projects for all three countries we reviewed. For
example, MCC expects that as a result of MCC-funded technical assistance
or credit to farmers and rural entrepreneurs, recipients will be able to
generate enough income to afford these services by paying fees to
providers. In cases where sustainability depends on achieving MCC's
projected impact, the soundness of MCC's economic analysis, discussed in
the next section of this report, will also be an important factor.
In its sustainability assessments, MCC generally adhered to guidance
issued in 2005, rather than to guidance from 2004. MCC's May 2004 proposal
development guidance included a general requirement for a strategy to
sustain progress after the compact's expiration. MCC's November 2005 MCC
Compact Assessment and Approval Guidelines did not require such a
strategy, but the guidelines required identification of factors
contributing to institutional and financial sustainability for each
project. (See fig. 8.)
Figure 8: Issuance of Key MCC Criteria for Project Sustainability Relative
to Completion of Due Diligence for Madagascar, Honduras, and Cape Verde
Limitations in Assumptions and Data, as Well as Country Involvement,
Affected the Accuracy of Economic Analyses
During its due diligence reviews for Madagascar, Cape Verde, and Honduras,
MCC analyzed proposed projects' probable impact on the country's economic
growth and poverty reduction.24 This analysis was intended both to assess
whether these projects would achieve MCC's goals and to provide a basis
for monitoring their progress and evaluating their impact.
To predict each project's impact on economic growth, MCC calculated an
economic rate of return (ERR) -that is, the expected annual average
return to the country's firms, individuals, or sectors for each dollar
that
MCC spends on the project.25 In calculating projects' ERRs,26 MCC used an
economic model that includes the following elements (see fig. 9):
o MCC's annual expenditures for the project,
o the project's annual benefits to the country,
o predicted net benefits27 of the project, and
o the projected ERR.
Figure 9: Illustration of MCC's Methodology for Calculating an Economic
Rate of Return: Advanced Crop Irrigation Project
ERRs for the proposed projects in Madagascar, Cape Verde, and Honduras
varied considerably, ranging from 116 percent for Madagascar's land
titling project to 10 percent for a watershed management and agriculture
support project in Cape Verde;28 the median ERR for projects in the three
countries was 14 percent. However, because MCC analyzed the Madagascar and
Honduras proposals before publishing its first set of economic analysis
guidelines, the ERRs did not significantly affect those countries' overall
choice of projects. (See fig. 10.)29 MCC's finding of a low ERR for part
of Cape Verde's watershed management and agriculture support project
resulted in the country's dropping an irrigation activity on one island.
Figure 10: Issuance of Key MCC Criteria for Economic Analysis Relative to
Completion of Due Diligence for Madagascar, Honduras, and Cape Verde
Note: The 2006 guidance states that because of economic analyses'
confirmed tendency to be overly optimistic about project benefits, MCC
prefers that evidence about project impact be based on evaluations of
similar, completed projects. For a discussion of the accuracy of such
economic analyses, see Gerhard Pohl and Dubravko Mihaljek, "Project
Evaluation and Uncertainty in Practice: A Statistical Analysis of
Rate-of-Return Divergences of 1015 World Bank Projects," World Bank
Economic Review, Volume 6 (1992): 255-277. With respect to poverty
reduction, the guidelines state that poverty reduction should be measured
by the expected decrease in the poverty gap, defined simply as the amount
of money that, when transferred to poor people, brings everyone's income
up to the poverty line.
We found limitations in the assumptions and data that MCC used in its
analyses as well as in the countries' involvement in the analyses. These
limitations may negatively affect the accuracy of the analyses and the
countries' understanding of the analysis process, respectively.
o Assumptions and data. Some of the assumptions and data that MCC used in
its analyses do not fully reflect the countries' socioeconomic
environment. As a result, MCC cannot be assured that the projects it
approved, partly on the basis of these analyses, will achieve the
compacts' goals. For example, in calculating the ERR for the Madagascar
land titling project, MCC assumed that local small farmers would use newly
titled land as collateral for loans, invest the borrowed funds in
agricultural activities, and benefit from the increased income from those
activities We discussed this assumption with focus group participants in
Madagascar, including MCA and U.S. government officials, senior Madagascar
government officials representing ministries affected by the compact, and
bank representatives. Our discussions suggested that MCC's analysis may
have been overly optimistic in assuming that small farmers would mortgage
40 percent of their newly titled land in Madagascar's uncertain market,
which does not offer insurance.30 While MCC assumed that 40 percent of the
newly titled land will be collateralized, focus group participants
believed this was overly optimistic. As a result, MCC may have calculated
an unrealistically high ERR for the land titling project. Furthermore, the
project may be more likely to benefit farmers with large, secure
landholdings and investors from outside the farming community, rather than
the local small farmers it was intended to help.31
Similarly, MCC's economic model for a part of Madagascar's finance
project, the modernization of the National Savings Institution, may not
accurately reflect the institution's financial condition. MCC's model for
the institution's modernization uses the institution's April 2003 net
profits (1.4 billion Malagasy francs) as a baseline for estimating the
benefits of computerizing the bank. However, according to institution
officials, 2003 is not a representative year for the bank, and it
experienced unusually large net losses (13 billion Malagasy francs) after
an economic crisis in 2002. As a result, MCC may have inaccurately
estimated the project's likely impact on the banking system.
o Country involvement. In the two countries we visited, country
representatives were not closely involved in MCC's economic analyses of
the proposed projects. This constrained the contribution of the analysis
process to enhancing country partnership, and stakeholders' understanding
of MCC's economic analysis-including the data analyzed, assumptions used
in the analysis, and the expected outcomes. According to country officials
in Madagascar and Cape Verde, MCC developed the economic models and
selected data with little assistance from country representatives.
According to the Cape Verde proposal team, they provided some data but did
not participate in the actual analysis and did not have a clear
understanding of the process and the criteria MCC used to assess proposed
projects. We discussed these issues with MCC officials, who told us that
the countries' degree of involvement depended on the capability and
willingness of the countries' proposal development team to actively
participate in the analyses.
MCC-Approved Compact Implementation Structures Are Not Yet Complete;
Weaknesses in Monitoring and Evaluation Framework May Limit Measurement of
Results
MCC gave Madagascar, Cape Verde, and Honduras authority to propose and
develop the implementation structures that they will use to manage
compacts, although MCC retains authority over a number of management
decisions. To govern their programs, the countries have created management
units under the direction of a steering committee, but they have had
difficulty in filling key positions. The countries also have established
structures for ensuring fiscal accountability and for managing
procurements that appear to be effective; however, implementation is still
at a very early stage, and some required elements of these structures are
not yet in place. Finally, the countries have established frameworks for
monitoring and evaluating the performance of MCC projects. However, the
frameworks have weaknesses related to the inadequacy of baseline data,
linkage of monitoring plans with economic models, methods of addressing
uncertainty in achieving stated targets, and the timeliness of research
designs for randomized controlled trials. These weaknesses may limit MCC's
ability to track and account for program results.
Countries Manage Projects with MCC Review, but Have Faced Delays in
Staffing Oversight and Management Structures
MCC has established a limited presence in Madagascar, Cape Verde, and
Honduras and has approved oversight and management structures that allow
the countries to direct their compact programs, subject to MCC review.
However, these structures were not fully staffed until months after the
countries' compacts entered into force. According to MCC officials, the
delay in staffing has shortened the available time in which to achieve
compact goals. MCC has adopted a policy that implements its authority to
provide funding to hire officials prior to compact signature, which may
reduce delays in hiring in the future.
MCC Has Oversight Staff in Each Country, while Countries Retain Management
Responsibility
Although MCC maintains a limited presence in each country and retains some
review and approval authorities, the countries have authority to manage
and oversee the compact program using MCC funds. In each country, MCC has
directly hired a resident country director and a small staff. The director
serves as MCC's public face and manages its relationship with the compact
country, guiding and overseeing the country's efforts to complete needed
plans and reports, implement accountability mechanisms, and make
appropriate use of resources. The director also supervises management of
the MCC office and staff, site visits by MCC staff and technical teams,
and engagement with MCC headquarters. MCC retains authority and approval
over several key elements, including quarterly disbursements, plans such
as those for monitoring and evaluation, procurements above certain
thresholds, and the hiring of key employees. According to MCC officials,
resident country directors review matters that require MCC approval,
sometimes making a recommendation, before forwarding the matters to MCC
headquarters.
Each of the three countries has established structures to provide
oversight and management of their compact programs and to ensure
stakeholder input (see fig. 11). Although these structures have most
elements in common, the countries have the flexibility to design these
structures to fit their needs.
Figure 11: Compact Country Oversight and Management Structure
Note: This figure represents a composite of the Madagascar, Cape Verde,
and Honduras oversight and management structures. However, in Honduras,
stakeholder input is obtained through representatives on the steering
committee, rather than through a stakeholder committee.
The three countries have generally included the following three key
oversight and management entities in their structures:
o The steering committee represents the country government and interfaces
directly with MCC. The committee is ultimately responsible for the
oversight and results of the MCC compact, oversees management unit
employees, and approves and signs off on key decisions and reporting to
MCC. Steering committee members vary by country but have included
government executives, such as representatives of the Prime Minister's
office; representatives of key ministries affected by compact projects;
and representatives of stakeholder groups, such as civil society and the
private sector. The MCC resident country director serves as a nonvoting
member of the steering committee. According to MCC guidelines, MCC must
approve the individual members of the steering committee.
o In Madagascar and Cape Verde, a stakeholder committee meets periodically
to advise and inform the steering committee regarding compact
implementation and to serve as the official liaison between interested
parties and the steering committee. Stakeholder committee members include
representatives of civil society, the private sector, and NGOs. MCC
approves the type of organization that should be represented on the
committee (e.g., civil society) but does not approve the specific
individuals. MCC reserves the right to approve all changes in the
committee's membership.
o The management unit is directed by the steering committee and has
principal responsibility for overall compact management and
implementation.32 The compact lists key management positions responsible
for daily program operations. The steering committee hires
employees for these positions, but MCC retains the authority to approve
the country's choices.33
Oversight and Management Entities Have Faced Staffing Delays
In each of the three countries, key compact management positions have
remained unfilled after the compacts have entered into force. Madagascar
and Cape Verde did not hire key officials until several months after their
compacts' entry into force in July and October 2005, respectively. As of
April 2006, Honduras's hiring of key officials was not yet complete,
although the compact entered into force in September 2005. (See table 1.)
This incomplete staffing at entry into force limits the ability of the
countries to achieve their compact objectives within the fixed time period
of the compacts.
Table 1: Hiring Dates for Key Positions Identified in Madagascar, Cape
Verde, and Honduras Compacts, as of April 2006
Hiring date
Key position Madagascar Cape Verdea Honduras
(entry into force (entry into (entry into force
force
July 27, 2005) Sept. 29, 2005)
Oct. 17, 2005)
Managing director: Incumbent January 1, 2006 The first person
participated in to hold this
Responsible for the compact position, from
overall operation of development and November 2005 to
the management unit, served as interim March 2006, left
including all director prior to after the change
communications with being confirmed on in ruling party
third-parties and November 22, 2005. from the January
the public, elections. The
including other current managing
donors. Certifies director was hired
all quarterly March 6, 2006.
financial and
performance reports,
budgets, and any
other documents
presented to the
steering committee.
Manager of November 22, 2005 March 1, 2006 April 10, 2006
administration and
finance:
Responsible for
ensuring that
reporting is
prepared, and that
all accounting
records are
maintained in a form
acceptable to MCC.
Manager of November 22, 2005c January 1, 2006 Selection in
procurement: b progress.
Serves as a liaison
with the procurement
agent and for the
preparation of
periodic reporting
to the steering
committee regarding
procurement
activities.
Key position Madagascar Cape Verdea Honduras
(entry into force (entry into (entry into force
force
July 27, 2005) Sept. 29, 2005)
Oct. 17, 2005)
Manager of January 19, 2006 January 1, 2006 Selection in
monitoring and progress.
evaluation:
Responsible for
setting up the data
collection,
analysis, and
reporting systems
for the overall
program, and, in
turn, for training
and assisting
project managers in
their implementation
project monitoring
systems for each
project.
Manager of Position does not March 1, 2006 April 1, 2006
environmental and exist at
social assessment: MCA-Madagascar.
Ensures that
environmental and
social mitigation
measures are
followed for all
activities of the
program, in
accordance with the
provisions set in
the compact and
other documents.
Managers of Land Tenure: Watershed Rural Development:
individual compact Incumbent Management and Selection in
projects: participated in Agricultural progress.
compact Support:
Project managers development and Transportation:
should have sectoral served as interim January 1, 2006
expertise relevant manager prior to April 1, 2006
for their respective being confirmed on Infrastructure:
projects. Project November 22, 2005. January 1, 2006
managers are
responsible for the Finance: Private Sector
day-to-day oversight Development:
and management of November 22, 2005 January 1, 2006
implementing
entities.d Agricultural
Business
Investment:
November 22, 2005.
A new manager took
over this position
on January 19,
2006.
Source: MCC compacts and hiring data.
aAccording to MCC, in Cape Verde, some of the key staff "became engaged"
shortly after Compact signature-though the hiring process was not
complete. These positions, including the senior economist, project
managers, and monitoring and evaluation manager, were retained through a
provisional employment contractual arrangement made with the Ministry of
Finance.
bThis position is referred to as the "procurement specialist" in Honduras.
cIncumbent serves as manager of procurement and of administration and
finance.
dThe Cape Verde compact, for example, defines an "implementing entity" as
a government affiliate, nongovernmental organization, or other public or
private sector entity or persons to which MCA-Cape Verde may provide
funding, directly or indirectly, to implement or carry out the projects or
any other activities in furtherance of the compact.
The apparent reasons for these difficulties in hiring key personnel vary
by country. For example:
o In Madagascar, according to U.S., MCC, and country officials, finding
technically qualified candidates for key positions has presented a
challenge compounded by competition for their availability. According to
the Managing Director of MCA-Madagascar, hiring her MCA team was difficult
because "all the good people in Madagascar are taken."
o In Honduras, a postelection change in government has delayed staffing
the program management unit. After a recent national election, the
director of the management unit resigned to allow the new administration
to appoint a director. According to MCC, all other management unit staff
appointed under the previous government have been retained.
o In Honduras, the monitoring and evaluation director position and one of
the project director (Rural Development) positions have been difficult to
fill. MCA-Honduras recompeted both of these positions because the first
attempt did not produce suitable candidates.
In October 2005, after the signature of its first six compacts, MCC
adopted a policy implementing the authority given it by section 609(g) of
the Millennium Challenge Act of 2003 to make grants to facilitate the
development and implementation of compacts. MCC's policy includes a
provision where, if certain conditions are met, it may fund an eligible
country's request for "management support payments" for salaries, rent,
and equipment for the country's MCA technical team prior to compact
signature. MCC has used this authority in four countries. Experienced
officials may therefore be hired and in place prior to compact signature
and the delays in hiring may be reduced in the future.
MCC Has Made Progress in Ensuring Fiscal Accountability, Although
Countries' Systems Are Still Being Developed
MCC has established a framework of internal controls to incorporate fiscal
accountability into compacts between MCC and recipient countries. The MCC
fiscal accountability framework includes an accountable entity (MCA),
which is responsible for internal control and at least an annual audit of
the program's financial and procurement transactions, with semiannual
audits initially expected for countries starting implementation.34
MCA-Madagascar and MCA-Cape Verde both have made
significant progress in implementing the framework's internal controls.35
However, MCA-Madagascar's newly formed system is somewhat less mature than
that established by MCA-Cape Verde. While MCA-Madagascar has established
internal control processes, some processes that are routinely practiced
are not formally documented, and it has not yet converted to a fully
automated financial management system. MCA-Cape Verde was established
partly within the country's Ministry of Finance, and it continues to
develop internal controls, put in place key finance and administration
personnel, and make use of existing information system platforms with the
characteristics of advanced and integrated systems. Neither country has
yet had an MCA unit in operation long enough to require an annual audit,
but officials at both MCAs told us that they expected their first audits
to be completed within the period established by the MCC IG.
MCC Has Made Progress in Fiscal Accountability at the Compact Level
MCC has made progress in defining the mechanisms and processes of internal
control that provide reasonable assurance of fiscal accountability at the
compact level. MCC incorporated these internal controls into a fiscal
accountability framework for the compacts between MCC and recipient
countries. To help countries implement this framework, MCC also developed
guidance36 and a series of charts that identify the characteristics of
each element of the framework, responsibilities of key personnel, and
examples of alternative methods of achieving fiscal accountability. Figure
12, which is based on these charts, provides an overall view of the
framework.
Figure 12: Elements of the MCC Fiscal Accountability Framework
Although MCC identifies the country's government as ultimately responsible
for meeting its fiscal accountability requirements, the country's
accountable entity is responsible for ensuring that basic internal control
functions-such as funds control and documentation, cash management,
disbursement controls, and timely and meaningful reporting-are
established as specified in the country's compact.37 The fiscal
accountability framework also requires that the accountable entity obtain
at least an annual audit of financial and procurement transactions.
Each country's compact requires the accountable entity to develop its own
fiscal accountability plan (FAP), which typically includes a transparent
process for ensuring that open, fair, and competitive procedures will be
used to administer grants or cooperative agreements and procurement.
Required elements of the FAP include funds control, internal controls,
accounting standards and systems, reporting, public availability of
financial information, cash management, procurement and contracting, the
role of independent auditors, and the roles of fiscal and procurement
agents.
MCA-Madagascar and MCA-Cape Verde Have Implemented Internal Controls for
Fiscal Accountability, but Their Systems Are Not Yet Fully Developed
MCA-Madagascar and MCA-Cape Verde have made significant progress in
implementing internal controls for fiscal accountability, although their
systems are not yet mature or fully developed. In line with their compacts
and supplemental agreements with MCC, both countries have put in place
such elements of the fiscal accountability framework as bank agreements,
fiscal agents, procurement agents, and financial reporting mechanisms;
however, controls called for in the framework, such as proper and
consistent accounting and authorization of transactions, are still in
development. Because the controls are not yet fully operational, we
focused on those that have been put in place in relation to an internal
control maturity model that provides a scale of development.38 We found
that both MCAs differ in the maturity of their internal controls.
Madagascar is using a newly formed entity; therefore, the maturity of
MCA-Madagascar's internal control is still at a developing stage. MCA-Cape
Verde used an existing government institution with established processes
and systems as the accountable entity and, therefore, represents a higher
level of internal control maturity. Neither country has yet been required
to provide any financial statement audits, but officials at both MCAs told
us that they expected their audits to be completed within the period
specified in their compacts.
MCA-Madagascar
Although MCA-Madagascar has established important internal control
processes, some have not been formally documented, and processes in
several areas are in the early stages of development. (See table 2.) The
government of Madagascar proposed, and MCC agreed, to establish
MCA-Madagascar as a new unit of government with the sole purpose of
managing implementation of the MCC compact. The MCA has established
finance and administrative operations based on the MCC fiscal
accountability framework, with segregation of duties39 over such functions
as work-order approval, vendor selection, and authorization to disburse
funds. Standard operating procedures for these processes are outlined in
the MCA's FAP, which was approved by MCC in January 2006. However, the FAP
includes limited detail on asset management, and sections in the
management of tax exemption, information technology management,
communications, and audits serve mainly as placeholders with information
"to follow." Although MCA-Madagascar acquired and is testing an automated
accounting system, current use of a manually based system leaves
operations vulnerable to human omission and error. The FAP makes reference
to MCC's procurement agreement guidelines but does not formally
incorporate procurement processes.
Table 2: Status of MCA-Madagascar Internal Controls, as of February 2006
Internal control Status
Key capability o Overall, a fiscal accountability framework is in
attributes place and functioning.
o The FAP was approved by MCC in January 2006.
o Duties are segregated, with oversight of fiscal
and procurement agent activities performed by
MCA-Madagascar officials.
o Bank agreements have been established.
o Expenditure process is in place.
o MCC conducts quarterly visits to review fiscal
activities.
o Staff has been hired.
Control efforts under o Formal documentation that covers the internal
way control structure for all significant transactions
and events (i.e., standard operating procedures) is
being developed on the basis of existing documents
and practice.
o Procurement has not yet been formally incorporated
into the FAP. At the time of our review, the
procurement manual was considered to be an annex to
the FAP.
Potential residual o Accounting system carries the inherent risks of a
risks manually based system.
o The MCC IG risk assessment of the MCA-Madagascar's
financial operations identified high vulnerabilities
in several critical areas-procurement, cash
management, and disbursements-that may adversely
impact financial operations.
o Lack of a disaster recovery or back-up plan could
result in the loss of critical data.
o Inadequately addressed data security could result
in fraud, waste, and abuse.
Source: GAO analysis of MCA-Madagascar systems and documents.
Although internal controls with and without formal documentation were in
operation in MCA-Madagascar at the time of our review in February 2006, a
risk assessment performed by the MCC IG in December 2005 found
vulnerabilities in high-risk areas that could affect the MCA's financial
operations. The assessment was performed during MCA-Madagascar's
implementation phase to detect early-on any vulnerabilities that could
prevent the MCA from establishing effective financial operations. The IG
reported that the MCA had high vulnerabilities in several critical
areas-including procurement, cash management, and disbursements-that may
adversely impact its financial operations, and that it had not implemented
all of the key controls described in its compact with MCC.40 The IG also
reported that the MCA's FAP was interim and not comprehensive in
establishing the internal controls needed to effectively manage the funds
provided by MCC.
MCA-Madagascar's internal controls are in development. It is putting in
place a control framework defined in MCC's fiscal accountability
framework, but the MCA currently has basic internal control policies and
processes that are not fully documented or implemented. Due to the
developmental stage of MCA-Madagascar's internal controls, residual risks
currently exist in its key financial processes and activities.
MCA-Cape Verde
MCA-Cape Verde is developing its internal controls on the basis of the
Cape Verde government's existing structure and information system
platforms, which have the characteristics of advanced and integrated
systems (see table 3). The MCA was established in the Ministry of Finance,
and management is focused on implementing its interim FAP, which was
approved by MCC in December 2005. The MCA is using ministry staff, and
most key finance and administration personnel are in place. To support the
financial processing requirements set out in Cape Verde's compact, the MCA
is also using the ministry's financial management information system and
the basic accounting system that the Cape Verde government has used for
several years. Cape Verde's fiscal agent, which is responsible for the
MCA's financial processing, consists of Ministry of Finance staff with
experience in the budget, treasury, and financial processes and in the use
of the accounting system. The MCA's expense transaction flows are
documented, and key duties and responsibilities are segregated.
Table 3: Status of MCA-Cape Verde Internal Controls, as of February 2006
Internal control Status
Key capabilities o Fiscal agent and key finance and
administration personnel are mostly in place.
o Basic accounting system appears to be fully
operational.
o The interim FAP is in place.
o Transparency of operations is enhanced by Cape
Verde's Web site and management oversight.
o Information system platform was preexisting in
the Ministry of Finance and is fully
operational.
o Expenditure flows for MCA office transactions
are documented.
o Procurement and payment authorizations are
segregated.
Control efforts under way o Currency conversion capability for accounting
system is being implemented.
o Conversion to commitment-based budgeting
system is under way.
o Accounting policies and procedures manuals are
being formalized.
o Outside audit entity is yet to be contracted.
o The FAP is being updated and finalized.
o Enhanced transparency initiative is to be
completed, which includes management tools.
Potential residual risks o Currency conversion calculations and some
conversion schedules are being done manually,
which could lead to potential errors or
omissions.
o Complications may result from converting to an
enterprisewide and commitment-based budgeting
system.
o Potential conflicts of interest could arise
due to the different ministries that currently
have operational responsibilities and may also
participate in the oversight of the
effectiveness and efficiency of MCA-Cape Verde's
operations.a
o Lack of a disaster recovery plan could result
in the loss of critical data.
Source: GAO analysis of MCA-Cape Verde systems and documents.
aMCC officials have stated that they believe this risk is mitigated
through clearly defined roles and responsibilities, autonomy of the
ministries, and specific controls over procurement approvals.
MCA-Cape Verde had internal controls across the entity, documented
transaction flows, sources of risk identified, and risk-mitigating control
processes that were mostly documented at the time of our review. MCA-Cape
Verde also had several internal control efforts under way, including the
development of policies and procedures manuals. As these efforts continue,
the MCA continues to have some residual risks that are related to specific
processes.
Financial Audits for MCA-Madagascar and MCA-Cape Verde
MCA-Madagascar and MCA-Cape Verde are preparing for financial statement
audits as required in their compacts. In January 2006, the MCC IG issued
guidelines for MCA financial audits and a standard statement of work for
the MCAs to use in their financial audits and posted lists of MCC-approved
audit firms for Madagascar and Cape Verde.41 According to MCC IG revised
guidance issued in January 2006, compacts are to have an audit completed
no later than 90 days after the first anniversary of the entry into force
of the compact or such other periods as parties may agree in writing. The
MCC IG recently informed us that the audits for MCA-Madagascar and
MCA-Cape Verde are to be completed by September 30, 2006, for the period
from entry into force until June 30, 2006. The MCC IG also told us that it
waived the first audit to be completed for MCA-Madagascar as stipulated in
the compact documents because the MCA did not have any financial activity
for the period to be audited.
Although we were able to assess the countries' progress in establishing a
system of internal controls to provide the financial accountability
required by MCC, it is too early to assess the effectiveness of those
controls over the funding provided to them. Each country has made
progress, but they are at different stages of maturity in their underlying
accountability systems, policies and procedures, and internal controls.
Oversight mechanisms that support program effectiveness as compact
projects disburse funds throughout multiple countries and promote the
segregation of procurement and payment processes will be critical for the
continual monitoring of progress, the development of key accountability
mechanisms that function as intended, and the mitigation of risks to
acceptable levels.
MCC-Approved Country Procurement Systems Have Effective Characteristics,
but Are Early in Their Implementation
Consistent with MCC guidance, each of the three countries-Madagascar, Cape
Verde, and Honduras-proposed its own procurement system, subject to MCC
review and approval during due diligence. Although the approved systems
have characteristics that we have found typical of effective procurement
systems, their effectiveness has not yet been tested by many procurements.
In addition, some of the staff and procedures needed to implement the
systems are not yet in place at MCC headquarters or in compact countries.
MCC Allowed Countries to Propose Procurement Agents and Systems, but
Retained Approval Rights
To capitalize on existing country knowledge and experience, MCC has given
the countries flexibility to choose their procurement agents and
standards, with the choice subject to MCC approval during due diligence.
Madagascar and Cape Verde are using private and public sector procurement
agents, respectively, and Honduras is using a combination of public and
private sector agents.42 (See app. IV.) In contrast to donors such as the
World Bank, MCC also allowed the countries to propose their own standards
for managing procurements.43 (See app. IV for details of the countries'
procurement agents and standards.) MCC required the countries to adhere to
"procurement principles" that include equal access to procurements,
competition for awards, and transparency of the process.44 Madagascar,
Cape Verde, and Honduras have used either existing World Bank standards or
their own laws to govern MCC procurements. In its procurement agreement
with each country, MCC included modifications to the country's selected
standards to reconcile them with U.S. law and MCC principles. For example,
MCC required that countries not include preferences for domestic suppliers
in solicitations paid for with MCC funds.
Although MCC's agreements with the three countries give the countries a
number of authorities over procurements, MCC retains certain approval
rights. MCC's fiscal accountability framework describes procurement as one
of the highest-risk areas of fiscal accountability. MCC's compact,
disbursement agreement, and procurement agreement with each of the
countries describe the relationship, roles, and authority of MCC, the
procurement agent, and the compact country (see app. IV for details).
Although these agreements have some common elements, each agreement is
unique to the individual country. To determine when MCC approval of
individual actions is appropriate, MCC included review thresholds in the
procurement agreements keyed to procurement size and methods. Above cost
thresholds, which vary among countries, MCC must approve items such as the
procurement method, terms of reference, and selection. Below thresholds,
the compact country may conduct procurements in keeping with the
procurement plan without additional MCC oversight. These thresholds are a
risk management tool that maximizes MCC control for larger transactions
but leaves discretion to the compact country for smaller transactions.45
Countries' Procurement Systems Have Strengths, but MCC and Country Systems
Are Not Fully Established
Although the three countries' procurement systems vary, each has
characteristics that we have previously identified as typical of effective
international procurement systems.46 These characteristics are similar to
the principles, such as equal access, competition, and transparency, that
MCC applies in its review of the systems during due diligence. However,
MCA officials in Madagascar and Cape Verde told us that they had completed
few procurements that would test the systems in practice. Furthermore, MCC
headquarters has not yet finished hiring its procurement staff, and
procurement systems in Madagascar and Cape Verde are not yet fully
established.
o Incomplete staffing. As of May 2006, MCC headquarters had hired its
senior director of procurement but had not yet hired five director-level
procurement staff in its Accountability Department. According to MCC
officials, the function of these vacant positions is currently being
filled by six intermittent personal service contractors, and MCC has
offered one of these contractors a full-time position.47 During our site
visits in January and February 2006, staff in Madagascar and Cape Verde
reported that the time frames for MCC review of procurements had been
satisfactory. However, if MCC staffing does not increase as the countries
submit more procurement decisions for approval, MCC may have difficulty in
conducting timely reviews of these decisions.
o Incomplete systems. During our site visits, we found that some elements
of the procurement systems documented in MCC's agreements with Madagascar
and Cape Verde were not yet in place or had not functioned smoothly.
o Neither country had yet established the procurement bid protest body
required by MCC or put in place automated systems for procurement tracking
and management, although both reported plans to do so in 2006. Madagascar
also had not established a process for reviewing contractor change order
requests.
o During our visit to Madagascar, a senior MCC official discovered that
the country's procurement plan for months 2 through 4 did not reflect
current project work plans. As a result, the procurement agent was
preparing for procurements that were no longer needed. Members of the
Madagascar management team told us that they would establish new
procedures to coordinate future work plan changes with the procurement
agent.48
o In Cape Verde, the procurement review commission lacked office space and
was concerned about being able to handle the number of reviews required of
it once procurements begin in earnest. The commission is comprised of
members who are fully employed elsewhere and are not permitted to delegate
their work. The members have worked nights and weekends or have negotiated
with their supervisors to be released from their other jobs to perform the
commission's work. According to MCC, subsequent to our site visit, Cape
Verde has taken actions to mitigate the risks to the efficient operation
of the commission.49
o Both Madagascar and Cape Verde reported difficulties with an MCC
requirement that documents be in both English and the local language. Cape
Verdean procurement review commission members are not required to speak
English but are expected to review documents prepared in English. Cape
Verde was considering hiring a full-time translator to address this
need.50 In Madagascar, the translation of a French document prepared by
the implementer of the finance project and requested by MCC delayed the
approval of needed project procurement.
MCC Has Established Country Monitoring and Evaluation Frameworks, but
Weaknesses May Limit Measurement of Results
Each of the three countries' (Madagascar, Cape Verde, and Honduras)
programs includes a monitoring and evaluation framework that includes
plans for data collection, data quality reviews, analysis, and interim and
final reporting of results. We found several weaknesses in the monitoring
and evaluation frameworks that could affect MCC's ability to track and
account for program results.
Countries' Programs Include Monitoring and Evaluation Frameworks
Each of the three countries' programs includes detailed plans for
monitoring and evaluating program results. MCC approved Madagascar's and
Cape Verde's plans in November 2005 and April 2006, respectively, while
Honduras's plan existed in April as a detailed draft but had not yet been
approved. According to MCC officials, Honduras's plan is not final pending
the staffing of the monitoring and evaluation director position to ensure
country understanding and buy-in. In accordance with MCC
guidance, the countries' plans include separate components for monitoring
and evaluation: 51
o The monitoring component includes, among other things, key indicators
that are linked as closely as possible to the variables identified in the
economic analysis of the country's proposed projects. These indicators are
to be used throughout implementation to assess whether the program is
likely to achieve the desired results. The monitoring component also
identifies baseline and target values for each indicator and includes
plans for periodic performance reports and data quality reviews. MCC
guidelines state that monitoring can be on select indicators to minimize
reporting requirements.52
o The evaluation component identifies, among other things, the methodology
that will be used to assess the program's impact, such as randomized
controlled trials, and describes plans for collecting baseline, interim,
and final data on program results.
Countries' monitoring and evaluation plans use, to varying degrees, the
economic analysis of the proposed projects to identify indicators for
monitoring progress toward project objectives, calculate targets for each
indicator, and evaluate the achievement of compact goals. The economic
relationships specified in the models, such as the relationship between
improved infrastructure and farm output, provide a basis for tracking
project success. The monitoring framework also includes setting target
values for indicators. Figure 13 illustrates indicators at various levels
for a rural development project in Honduras.
Figure 13: Indicator Structure for Honduras's Rural Development Project
Although the specifics of the countries' plans vary, the monitoring
component of each plan calls for the country's MCA to submit periodic
performance reports and data quality assessments to MCC. Performance
reports will include quarterly assessments to alert the countries and MCC
to any problems, periodic audits that analyze performance at all compact
levels, and annual reports that consolidate the quarterly reports and
recommend adjustments. Most performance data will be gathered by project
implementers, either country government employees or contractors; the
plans allow for contracting with other entities to prepare the reports. In
addition, each plan calls for third-party data quality reviews over the
course of the compact. For example, in Madagascar, data quality
assessments are planned to occur every 6 months during the first year and
annually thereafter.53
Weaknesses in Monitoring and Evaluation Frameworks May Limit MCC's Ability
to Measure Results
In reviewing the frameworks for monitoring and evaluation in the three
countries, we identified several challenges that MCC faces in ensuring
accountability for results. These challenges include (1) ensuring the
availability and quality of baseline data, (2) establishing clear links
between the economic model and the monitoring and evaluation framework,
(3) accounting for the degree of uncertainty in expected outcomes, and (4)
using randomized controlled trials in compact countries.
Ensuring Baseline Data Availability and Quality
Baseline data are essential to measuring the results of MCC-funded compact
projects. Although MCC has taken steps to collect baseline data for
monitoring and evaluation, problems with data availability and quality may
lead to challenges in measuring the progress and impact of the countries'
projects over time. MCC officials told us that they worked with their
country counterparts to set up a Management Information System that can
meet the requirements for collecting performance data. In addition, MCC
evaluated the technical capabilities of the country staff and the
information system the country proposes to use for data management
purposes. Finally, MCC budgeted funding for surveys in Madagascar and Cape
Verde to collect baseline data when it was not available. However, we
found that some of the baseline data in the countries' monitoring plans
were not complete, and that some of the data MCC collected were not
reliable.
o Baseline data availability. In some instances, the countries' monitoring
and evaluation plans lack complete baseline data against which to measure
progress. For example, two activity and final project indicators in
Madagascar's plan-"volume of production covered by warehouse receipts in
zones" and "volume of microfinance institution lending in the
zones"-currently lack baselines because the intervention zones have not
yet been selected. Moreover, although the collection of performance data
is closely linked to project implementation, Madagascar's plan contains no
intermediate outcome indicators and target values, thereby making it
difficult to effectively track project progress. (See fig. 13 for an
example of the role of intermediate outcome indicators in the monitoring
structure of Honduras.)
o Baseline data quality. MCC may face challenges in ensuring the quality
of the baseline data that it uses to monitor and evaluate program impact
and, as a result, may have difficulty in accurately measuring the impact
of compact projects. MCC officials told us that it has been difficult to
obtain accurate and reliable baseline data against which to measure
program results. In some countries, MCC has funded surveys to obtain the
needed baseline data. However, even with the additional resources provided
by MCC, obtaining baseline data has been a challenge. For example, we
found significant data quality problems associated with one of three
surveys that MCC funded to collect baseline data in Madagascar. Our
interviews with Madagascar and USAID officials who oversaw the survey
revealed that the survey results, which were used to estimate average land
values, are flawed in that they do not reflect recent significant changes
in Madagascar's currency.54 Madagascar's compact goal is to increase
household income, as measured by the percentage of increase in land
values.55 Because of the survey error, the land value estimates may not be
sufficiently reliable to evaluate the project impact and the compact as a
whole.56
Linking Monitoring Plans to Economic Analyses
Linking the indicators used to monitor and evaluate progress to the data
and assumptions used in MCC's due diligence economic analyses will also
present a challenge. In reviewing the draft plan for Honduras, we found
consistent linkages between the indicators for monitoring and evaluation
and the variables and assumptions used in the economic model. However, in
the plans for Madagascar and Cape Verde-the first two plans that MCC
approved-we found instances where MCC did not sufficiently link the
monitoring plans to the economic models, which may hamper its ability to
effectively measure project results. For example:
o After signing its compact with Cape Verde, MCC changed the interim
targets for seven indicators. In two cases, neither MCC nor Cape Verde was
able to identify the methodology used to select the indicators in the
monitoring and evaluation plan.57 According to MCC officials, they decided
that the assumptions in the economic analysis were a poor basis for
constructing the monitoring indicators and, therefore, chose other
indicators and estimated the targets. According to MCC officials, the
inability to identify the methodology, in conjunction with updated
baselines and revised work plans in the country resulted in MCC's and Cape
Verde's agreeing to reduce the interim targets that had been established.
o MCC's economic analysis for Madagascar's Land Tenure Project, which is
approximately one-third of the compact budget, did not identify the
expected benefits from the separate project activities. Therefore, we
could not track the linkage between the activities in the model to those
in the monitoring and evaluation plan.
o MCC's approved monitoring and evaluation plan does not include tracking
the results of two finance project activities-modernizing the National
Savings Institution and opening bank branches-although they were included
in economic analysis calculations during due diligence. According to an
MCC official, these two finance project activities will be tracked at a
higher level of aggregation-the finance project level-and monitored by
tracking the number and value of new accounts. However, this approach may
not adequately capture the outputs and benefits from the institution's
modernization and could potentially confound the effect of one activity
(modernizing the institution) with that of other activities. For example,
while an increase in the number and value of new accounts could result
from the two finance project activities, it could also result from an
overall increase in savings if customers invest in government savings
bonds, issued recently and prior to the start of the MCC compact.
Accounting for Uncertainty in Achieving Target Values
Although the countries' monitoring and evaluation plans acknowledge the
uncertainty of achieving indicator target values, MCC project monitoring
does not adequately address (1) the effect of potential variations in
uncertainty on the range of acceptable target values or (2) the
plausibility of target values. As a result, some targets specified in the
countries' monitoring plans may not be achieved.
MCC disbursement agreements include as a condition precedent that, if an
indicator value observed during compact implementation does not fall
within 10 percent of the agreed-on target values, MCC may withhold
disbursements. MCC applies this 10 percent margin to all projects,
regardless of type (e.g., agriculture or infrastructure). However, our
analysis suggests that several factors could cause indicator values for
many projects to fall outside the 10 percent range.
o External factors. Uncertainty associated with external factors varies by
country and project. For example, according to previous GAO work, external
factors that could affect project implementation might include political
instability, the lack of commitment of political leaders to necessary
reforms, the magnitude of assistance from other bilateral and multilateral
donors, weather conditions that affect crop yields, and the instability of
international markets. 58 These factors could cause indicator values to
fall outside the 10 percent range used across all countries and
projects.59
o Time factor. The 10 percent range also does not account for the increase
in the uncertainty of targets over time-for example, target values for
year 5 of a compact are likely to be less precise than those for year 1.
Therefore, to the extent that MCC bases its disbursement decisions on
results falling within a common range, it may not fully account for
variations in uncertainty across projects and over time.
According to MCC guidelines, the economic analyses and monitoring and
evaluation plans should, as much as possible, be clearly linked. However,
limitations in (1) the economic analyses due to problems with data
quality, assumptions, and lack of country involvement and (2) the
consistency between the economic analyses and the monitoring and
evaluation plans constrain MCC's ability to set plausible targets. If
targets are overly optimistic, countries may fail to reach them and MCC
may not be justified in halting disbursements because the countries failed
due to unattainable targets. Conversely, setting too conservative a target
may not prompt the country to fully utilize MCC resources. A lack of
plausible targets may lead to MCC's making ad hoc decisions regarding the
consequences of missing targets and applying judgment subjectively and
inconsistently in setting or modifying targets. MCC officials told us that
they would use a missed target as a cue to discuss with the country's MCA
its reasons for missing the target, and, on the basis of those
discussions, they would determine whether to use their authority to
withhold funding. According to MCC officials, senior management approval
would be needed to significantly modify targets; however, MCC currently
has no policy or documentation that defines a "significant modification"
of targets, especially for targets used as conditions precedent to
disbursements.
Using Randomized Controlled Trials in Compact Countries
MCC has retained research organizations to help the countries evaluate
program impact using, as appropriate, randomized controlled trials, but
MCC's involvement of these organizations after project implementation
begins may limit their ability to evaluate impact accurately.60 These
organizations' scope of work may include training MCC and compact country
staff; designing the trials and data collection; and proposing appropriate
methodologies for, and analyzing results from, impact evaluations.
According to MCC monitoring and evaluation officials, MCC has begun
designing impact evaluations by identifying those program components that
can and cannot be evaluated using randomized controlled trials, which MCC
has indicated are its preferred method of impact evaluation.61 According
to MCC officials, MCC considers evaluations using randomized controlled
trials as "rigorous" and evaluations using other methodologies as
"standard."
MCC has hired an independent consultant experienced in impact evaluations
to work with compact countries to assess the appropriateness of using
randomized trials to evaluate MCC's projects. When these assessments are
completed, the five research organizations will be invited to compete to
conduct randomized trials after compact implementation begins.62 However,
at that point, the organizations will not have had an opportunity to
assess the design of the countries' evaluation strategy, including the
adequacy and reliability of the baseline data. Without the involvement of
these organizations before implementation of the relevant project(s)
begins, MCC may not be able to ensure that they have the necessary data
and have established appropriate research designs for their work. MCC
officials told us that they thus far had not involved the five
organizations because rigorous evaluations were turning out not to be
feasible in some cases, or because the tasks were not large enough to
warrant the use of the five research organizations.63
Conclusions
MCC continues to mature and evolve as an institution, taking on the
ambitious task of creating new, country-managed organizations while
developing processes to oversee what are expected to be relatively large
amounts of foreign assistance. Toward that end, MCC has taken positive
steps with regard to establishing policies and procedures for MCA
organizations. However, it has taken time to complete the numerous
agreements necessary for compacts to enter into force. This could continue
to present challenges, given that MCC is working simultaneously with a
number of nations to develop and implement compacts.
As MCC moves forward, partnering with countries to develop well-founded
economic assumptions will be crucial to establishing a foundation for the
work of MCC and its partners. Furthermore, holding countries accountable
for results requires, to the extent practical and cost-effective:
collecting reliable and accurate baseline data, linking economic analyses
to monitoring plans, addressing the uncertainty associated with program
results, and ensuring the timely development of the research design for
randomized controlled trials.
Recommendations
Because of the central role of reliable economic analyses and the
importance of partnering with countries in achieving MCC goals and
ensuring accountability for MCC programs, we recommend that the Chief
Executive Officer of MCC take the following two steps:
o Ensure that MCC officials, in partnership with country representatives,
perform economic analyses that more fully reflect the countries'
socioeconomic environment and are better understood by country public and
private sector representatives.
o To the extent practical and cost-effective, improve MCC's monitoring and
evaluation capabilities by
o obtaining more accurate and reliable baseline data needed to permit
tracking progress during compact implementation;
o ensuring a clear linkage between MCC's economic analyses and monitoring
and evaluation frameworks;
o developing policies, procedures, and criteria for establishing targets
and for adjusting those targets if unforeseen events affect outcomes; and
o taking steps to ensure the timely development of the needed research
design for randomized controlled trials, if they are undertaken, prior to
project implementation.
Agency Comments and Our Evaluation
We received written comments on a draft of this report from MCC and the
Department of State. In commenting on a draft of this report, MCC
generally agreed with our findings, conclusions, and recommendations. MCC
noted that our discussion of the evolving guidance provided to eligible
countries in 2004 and 2005 was a result of the complex process of engaging
with eligible countries while simultaneously developing policies and
procedures. MCC stated that this criticism should not be valid beyond the
initial instances covered in this report. We recognize that MCC was
simultaneously addressing a number of issues during this period, but we
felt it was important to discuss the evolving nature of guidance to
provide a balanced perspective regarding the process eligible countries
had to follow to sign compacts.
MCC also commented (1) that our characterization of data quality issues in
Madagascar cited only a single survey and (2) that it differed with us on
the appropriate level of aggregation in linking economic models and
monitoring and evaluation plans. We note that there was one other instance
of poor data quality in Madagascar, but that we focused on the
Agricultural Productivity Survey because of the importance of accurately
tracking land values to monitor results. We agree that disaggregation may
not always be feasible, but note that aggregation poses some challenges
that could limit the effectiveness of monitoring and evaluation. We
reprinted MCC's comments, with our responses, in appendix V. We also
incorporated technical comments from MCC in our report where appropriate.
State commented that some of our findings reflect minor or transitory
problems and provided specific observations regarding MCC's evolving
guidance, the assumptions used in MCC's economic models, country
participation, staffing delays, fiscal accountability structures, and the
use of randomized controlled trials.
State noted that MCC's guidance could be expected to evolve, given the
newness of the organization, and that informal guidance from MCC was
always available to eligible countries. We agree that guidance could be
expected to evolve, but we sought to provide a balanced perspective by
noting instances where MCC's verbal guidance may not have been sufficient
to assist countries in submitting proposals that met MCC's criteria.
State questioned the findings from our Madagascar focus groups and our
finding that MCC conducted economic analyses with limited country
involvement. Our focus groups resulted in discussions of the assumptions
used in the Madagascar economic analysis with a broad range of country
stakeholders representing U.S. and Malagasy agencies and organizations
involved in implementing the compact. Furthermore, MCC agreed in its
comments that, in some cases, the level of country engagement on economic
analysis could be improved. MCC outlined specific steps that it had taken
to increase country involvement in economic analyses.
Regarding staffing delays and fiscal accountability structures, State
commented that we offered no suggestions to MCC for things they could have
done differently. We added additional material to the report to elaborate
on the steps MCC has taken to reduce delays in staffing key positions. In
regard to fiscal accountability, we agree that different countries will
differ in their maturity of internal control. However, evaluating maturity
is key to properly assessing risk and establishing effective oversight
mechanisms.
Finally, State disagrees with what it terms our "reliance" on randomized
controlled trials to measure success. This comment misconstrues our
findings. We did not rely on or advocate this methodology, but rather are
commenting on MCC's use of randomized controlled trials as its preferred
method of impact evaluation.
We have reprinted State's comments, with our responses, in appendix VI.
We are sending copies of this report to interested congressional
committees as well as the Secretary of State, the Secretary of the
Treasury, the CEO of MCC, and the Administrator of USAID. We will also
make copies available to others upon request. In addition, this report
will be available at no charge on the GAO Web site at http://www.gao.gov .
If you or your staff has any questions about this report, please contact
David Gootnick at (202) 512-3149 or g [email protected]. Contact points
for our Offices of Congressional Relations and Public Affairs may be found
on the last page of this report. GAO staff who made major contributions to
this report are listed in appendix VII.
Sincerely yours,
David Gootnick Director International Affairs and Trade
Jeanette Franzel Director Financial Management and Assurance
Appendix I: MCC Actions in Response to April 2005 GAO Recommendations
The Millennium Challenge Corporation (MCC) has taken a number of steps to
address our April 2005 recommendations regarding its strategic planning,
internal controls, and human capital and governance policies, although
some aspects of its organizational structure are not yet complete. MCC has
prepared a strategic plan for fiscal years 2006 through 2011, and an
annual performance plan for fiscal year 2006. MCC also has strengthened
its internal controls and taken steps to implement some fiscal
accountability mechanisms, established an audit committee within its
governing board, and completed several required audits and reviews.
However, MCC has not documented all financial control activities and
continues to face risks from the poor interfacing of its systems with
those of the Department of the Interior's National Business Center (NBC).
MCC has reassessed its staffing model, developed a plan for recruitment,
and implemented an improved performance management system; but, it does
not systematically track the use of staff resources to verify its human
capital model. Finally, MCC has approved a corporate governance policy and
taken steps to improve board involvement in planning, management, and
communication; but it has not yet fully addressed risk management for the
corporation.
MCC Has Developed a Strategic Plan to Enhance Accountability and Completed
a 2006 Performance Plan
Consistent with our recommendation to enhance corporate accountability,
MCC completed a strategic plan, approved by the Office of Management and
Budget (OMB) in November 2005. In April 2006, MCC completed an annual
performance plan that provides goals and benchmarks for assessing its
performance in fiscal year 2006. This annual performance plan will enable
MCC to report in the future on its progress in meeting its goals.1 As part
of the annual performance plan process, MCC also has developed goals and
benchmarks for its individual departments that support the attainment of
the corporate goals identified in the strategic plan. (See table 4.)
Table 4: MCC Response to GAO Recommendations for Corporatewide
Accountability
Summary of GAO MCC actions MCC actions not
recommendation completed completed or in
progress
Implement a strategic X Completed N/A
plan strategic plan,
November 2005.
Establish annual X Annual N/A
performance plans and performance plan
goals completed in
April 2006.
Use performance measures X Annual N/A
to monitor progress in performance plan
meeting both strategic completed in
and annual performance April 2006.
goals
Report internally and Completed annual
externally on progress performance plan
in meeting strategic and provides benchmarks
annual performance goals against which to
measure MCC's fiscal
year 2006 performance.
Sources: GAO interviews and analysis of MCC documents.
MCC Has Made Significant Progress in Establishing Internal Controls
In response to our April 2005 recommendations, MCC has made significant
progress in establishing internal controls over program and administrative
operations at both the MCC headquarters and compact levels.2 MCC has made
progress in each of the five components of internal controls discussed in
our April 2005 testimony:3 (1) control environment, (2) risk assessment,
(3) control activities, (4) monitoring, and (5) reporting.
o Internal control environment. MCC has initiated several measures to
establish a positive internal control environment, including documenting
its organizational structure for the Administration and Finance and
Accountability structures. The Fiscal Accountability area, which is a
component of the Accountability structure, currently consists of a
managing director and several directors; however, other positions to
support the fiscal oversight of MCC compacts have yet to be filled. A
formal ethics program has also been established for MCC headquarters.
o Risk assessment. MCC is developing a process for assessing risks facing
the corporation and its programs. To this end, MCC has hired a third-party
consulting firm to support the implementation of processes, based on OMB
A-123, Management's Responsibility for Internal Control, December 2004,
criteria, to address risks associated with domestic and foreign
operations.
o Control activities. MCC has instituted several control activities to
reduce risk. MCC has submitted its Strategic Plan under the Government
Performance and Results Act (GPRA) of 1993 and has completed required
audits and reviews, such as those required under the Federal Managers
Financial Integrity Act (FMFIA) of 1982 and the Federal Information
Security Management Act (FISMA) of 2002.4 MCC is in the process of
addressing the various material weaknesses, reportable conditions,
findings, and recommendations identified in the audits. For example, MCC
continues to address the formal documentation of control activities for
the financial reporting process at MCC headquarters. MCC also contracted
with a third-party service-provider, NBC, to maintain its accounting
system and the recommended Statement on Auditing Standards No. 70 (SAS 70)
review5 was performed; however, MCC is still addressing issues reported as
a result of the review, such as the need to address manual processes that
present inherent risks associated with accounting systems and related
processes. For example, MCC forwards daily to NBC a package containing
source documentation that is used by NBC to record transactions to the
general ledger. Similarly, information for MCC's travel expenses is
prepared at MCC headquarters office and daily communicated to NBC through
manual processes. MCC is aware of the need to address these issues and is
working with NBC to mitigate or eliminate the manual processes.
o Monitoring. MCC has taken steps to ensure ongoing monitoring and
periodic testing of control activities. MCC's investment committee
embodies functions of monitoring and testing and operating as an integral
part of MCC's internal control program by overseeing the program units'
compliance with both the procedural and substantive elements required by
its internal processes. Also, MCC conducted its first comprehensive survey
of internal controls, performed by outside consultants, in conjunction
with its annual audit. In addition, MCC has formed formal review panels to
monitor the progress of addressing findings from internal and external
reviews. For example, consistent with OMB guidance, MCC formed an FMFIA
Management Review Panel to assess the results of the internal control
survey along with the findings of MCC's independent financial auditors.
Similarly, MCC implemented a specific procedure to address recommendations
from reviews and audits performed by its Inspector General (IG).
o Reporting. MCC has made progress in establishing a process for assessing
and reporting on the operating effectiveness of its internal controls. MCC
established a formal board-level audit committee whose responsibilities
include (1) financial controls; (2) the integrity of the reporting
process; and (3) performance of the independent audit process. In
addition, MCC's FMFIA Management Review Panel assessed the results of the
internal control survey, along with the findings of MCC's independent
financial auditors. The panel identified four material weaknesses and
actions that MCC will be taking in future months to resolve them, which
the acting Chief Executive Officer certified on November 7, 2005.
MCC officials told us that the development of new internal control
policies and procedures and the revision of those already in place is a
continuing process as MCC continues to mature as an organization. Table 5
summarizes MCC's progress in addressing our April 2005 recommendations.
Table 5: MCC Response to GAO Recommendations for Internal Controls over
Program and Administrative Operations
Summary of GAO MCC actions completed MCC actions not
recommendation completed or in
progress
Complete the
development and
implementation of
overall plans and
related time frames
for actions needed
to establish the
following:
A positive and X Documented the o Positions within
supportive internal organizational the accountability
control environment X structure. area remain vacant.
X Documented a corporate o Continued efforts
strategy. to address audit
X findings of material
Implemented a procedure weaknesses and
X for addressing IG audit reportable
findings. conditions.
X
Established formal
ethics program,
including training for
all MCC employees.
Conducted audits
required under FISMA
and FMFIA.
Submitted Strategic
Plan under GPRA.
A process for X Identified 14 priority o Risk assessment
ongoing risk internal control areas process is still
assessment relating to being developed.
administrative and
program risks. o Outside consulting
firm to support MCC
risk analysis based
partly on OMB A-123
criteria.
Control activities X Internal Controls o MCC continues to
and procedures for Strategy Group work on implementing
reducing risk, such identified 14 key and formally
as measures to internal control areas documenting some
mitigate risk for MCC to focus procedures and
associated with efforts. control activities.
contracted
operational and
administrative
services
Ongoing monitoring X Established formal N/A
and periodic testing review panels to
of control monitor progress of
activities efforts to address
findings from internal
and external reviews
and formally documented
a process for
addressing MCC IG
recommendations.
A process for X Created a board-level o MCC continues
assessing and audit committee to efforts to establish
reporting on the monitor MCC's financial a comprehensive
effectiveness of controls, integrity of database to track
internal controls the financial reporting recommendations.
and addressing any process, and
weaknesses performance of the
identified audit process.
Sources: GAO interviews and analysis of MCC documents.
MCC Has Taken Steps to Develop Human Capital, but Does Not Track
Allocation of Human Capital to Key Activities
MCC has taken steps and is continuing to further develop its human capital
systems by (1) assessing its staffing needs; (2) improving its
recruitment, development, and retention systems; and (3) implementing a
performance management system linking compensation to individual
contributions toward corporate goals. However, despite having plans to
increase its staff by an additional 38 percent between May and September
2006, MCC does not systematically assess its staffing needs, has not
developed a human capital plan, and has not yet fully implemented its
improved performance management system, as follows:
o Staffing. Although MCC has completed an assessment of its human capital
needs since our April 2005 recommendation, it does not systematically
track the use of staff time on an ongoing basis. MCC's updated assessment
of its human capital needs shows that it plans to increase its staffing
from 218 staff, as of May 2006, to approximately 300 staff, as of
September 2006.6 MCC also has created an organization chart that includes
the specific approved positions for each department under the new
300-person staffing model, and it is hiring for many of these positions
using limited-term appointments to provide greater flexibility in filling
future needs.7 According to MCC officials, MCC made its case to OMB for
increasing staffing to 300 persons on the basis of an analysis of MCC
staffs' recollection of the amount of time they spent developing the
Georgia compact, which was thought to have been the most complex compact
development process to date. However, MCC officials felt that this
analysis may not have fully captured the amount of time spent by some
departments in developing the compact. Retrospective analysis was
necessary because MCC does not track employees' time on mission-related
projects on an ongoing basis. Without such data, MCC management is not
able to systematically assess the staffing requirements needed to carry
out MCC's mission and consistently align its human capital with its
changing needs.
o Recruitment and retention. MCC has identified priorities and committed
resources for recruitment and is developing a human capital plan to
address retention and training. To support its effort to hire
approximately 82 additional staff between May and September 2006, MCC has
retained an outside consultant firm to work on-site and help with
recruitment and has identified positions as first- or second-tier hiring
priorities. MCC officials also told us that they are developing an overall
human capital plan that will include planned activities and a time frame
for identifying critical skills and competencies for MCC's key positions.
The officials stated that the human capital plan also will include a
strategy for staff retention and will address staff training. Currently,
MCC has developed procedures for providing employees with outside
training. MCC intends to develop a comprehensive training plan following
the completion of the human capital plan, a draft of which it expects to
circulate to MCC senior staff for their review and comment by September
30, 2006.
o Performance management. In keeping with our recommendation, MCC has
established a performance-based compensation framework. MCC provided us
with documentation of its employee ratings process, showing that employee
expectations and performance reviews were keyed to organizational goals.
However, according to MCC officials, MCC did not incorporate the
departmental performance plans for the year into the performance framework
for 2006 until March 2006, as the annual performance plan neared
completion. MCC is shifting from a calendar year to a fiscal year
performance evaluation schedule to better align employee compensation with
its annual corporate goals. MCC anticipates that its strategic plan,
annual performance plan, department plans, and individual performance
goals will be fully synchronized beginning in fiscal year 2007. (See table
6.)
Table 6: MCC Response to GAO Recommendations for Human Capital
Infrastructure
Summary of GAO MCC actions MCC actions not
recommendation completed completed or in
progress
Establish an
effective human
capital
infrastructure,
including:
A thorough and X MCC has reassessed o MCC currently does
systematic assessment its human capital not track the amount
of the staffing X needs at of time its employees
requirements and approximately 300 spend on specific
critical skills staff, and OMB has tasks. Without this
needed to carry out approved this model. information, MCC
MCC's mission cannot systematically
MCC has completed an assess its staffing
updated organization needs on an ongoing
chart identifying basis.
the positions to be
filled under the new
human capital model.
A plan to acquire, X MCC has identified o MCC has not
develop, and retain priorities for completed an overall
talent that is X hiring based on its human capital plan but
aligned with the 300-staff model. plans to circulate a
corporation's X draft to MCC senior
strategic goals MCC has retained an staff for review and
outside consultant comment by September
firm in support of 30, 2006.
its effort to
increase staff from o MCC plans to include
218 in May 2006 to a retention strategy
approximately 300 at and address training
the end of September in the human capital
2006. plan.
MCC has developed a o MCC also plans to
procedure for develop a
employee training. comprehensive training
plan following the
completion of the
human capital plan.
A performance X MCC has established o MCC is shifting from
management system a performance a calendar year
linking compensation framework that links schedule to a fiscal
to employee compensation to year schedule of
contributions toward employee performance evaluation
the achievement of expectations and to better align
MCC's mission and performance reviews employee compensation
goals to MCC goals. with its annual
corporate goals.
o MCC anticipates that
its strategic plan,
annual performance
plan, department
plans, and individual
performance goals will
be fully synchronized
beginning in fiscal
year 2007.
Sources: GAO interviews and analysis of MCC documents.
MCC Has Taken Steps to Define Corporate Governance
The MCC Board of Directors (MCC Board) has taken steps to define the scope
of its corporate governance and oversight.8 The MCC Board has approved a
corporate governance policy developed by MCC with the involvement of staff
from MCC Board agencies.9 According to MCC officials, the policy
incorporates guidance on governance matters provided by board members at
their previous meetings, and the board participated in formulating MCC's
strategic plan before approving it in December 2005. The board has
established a board-level audit committee and a charter for that
committee. According to MCC officials, to address risk, MCC is recruiting
for the position of risk specialist and has used a contractor to support
MCC risk analysis. Finally, to improve communication with stakeholders in
eligible countries, MCC has published and distributed updated guidelines
for compact development and eligibility. MCC also has developed a series
of open forums where input is sought from groups with an interest in MCC.
(See table 7.)
Table 7: MCC Response to GAO Recommendations for Corporate Governance
Summary of GAO MCC actions MCC actions not
recommendation completed completed or in
progress
Recommended that the
Secretary of State ensure
that the MCC Board considers
and defines the scope of its
responsibilities with
respect to corporate
governance and oversight of
MCC...including oversight of
the following:
Executive management X The MCC Board has N/A
approved a corporate
governance policy.
The formulation and X The board N/A
execution of corporate participated in
strategies formulating MCC's
strategic plan and
approved the plan.
Risk management and audit X The MCC Board has MCC has not yet
and assurance processes formed an audit hired a risk
committee and specialist for
approved a charter the corporation
for that committee. but has used an
outside
consulting firm
to support MCC
risk analysis.
Communication and X MCC has published N/A
coordination with corporate and distributed
stakeholders X updated guidance on
compact development
and implementation.
MCC has developed an
outreach program of
open forums.
Sources: GAO interviews and analysis of MCC documents.
Appendix II: Scope and Methodology
At the request of the Senate Committee on Foreign Relations, we examined
the structures and procedures MCC has developed in consultation with
compact countries to manage compacts. Specifically, our work focused on
(1) the key areas that MCC examined in its due diligence assessments of
proposals for Madagascar, Cape Verde, and Honduras, and the criteria that
MCC used in these assessments, and (2) the form and adequacy of the
implementation structures that MCC and compact countries have put in place
for governance, procurement, fiscal accountability, and monitoring and
evaluation. In addition, we reviewed MCC's progress in responding to our
April 2005 recommendations on its corporate management and accountability
structures (see app. I).
To accomplish our objectives, we reviewed MCC's documentation of its
processes and agreements, supplemented by interviews with MCC officials.
We focused our review for objectives 1 and 2 primarily on the first three
countries with signed compacts-Madagascar, Cape Verde, and Honduras. These
countries' compacts were the first to enter into force. To further our
analysis for objectives 1 and 2, we also visited Cape Verde and Madagascar
in January and February 2006. We selected Cape Verde and Madagascar for
our site visits because they had advanced further than Honduras in filling
key positions and beginning compact implementation. While in Cape Verde
and Madagascar, we interviewed a number of MCC, Millennium Challenge
Account (MCA), and government officials and visited project sites.
To identify MCC's evaluation criteria and process for evaluating eligible
country proposals in due diligence, we reviewed MCC guidance and the
record of MCC analysis contained in (1) MCC's "due diligence books," which
are its internal records of how it assessed proposals submitted by
Madagascar, Cape Verde, and Honduras, and (2) investment memos, which are
MCC's analyses based on due diligence and internal recommendations to its
investment committee. These documents are restricted from public
dissemination due to their sensitive nature, but MCC made them available
to us for analysis. We have coordinated with MCC on describing the
information from these books in general terms without disclosing sensitive
information. We used MCC's definition of the due diligence process as
beginning with MCC's opportunity memo and ending with the acceptance of
the investment memo by MCC's investment committee. Our review, therefore,
may not capture some changes and decisions made by MCC or eligible
countries during proposal development and compact negotiations. To
evaluate MCC's assessments of proposals' consultative process, project
coherence, environmental and social impact, and institutional and
financial sustainability, we relied primarily on MCC's data and analysis
contained in the due diligence books and, to some extent, in the
investment memos. We compared MCC's analysis in these documents with
criteria outlined in MCC's guidance. We were able to perform only limited
independent verification of the use and adequacy of these criteria during
our site visits. With regard to its economic analyses, MCC also made
available to us the spreadsheet models it used to develop the economic
rate of return calculations that formed the basis for its evaluations of
the suitability of country-proposed projects for MCC funding. We
independently analyzed these spreadsheets and validated their logic and
conclusions on the basis of a review of economic literature and practices.
In addition, in Madagascar, we conducted a series of focus groups with
country officials to assess the data and logic used by MCC in developing
their economic analysis.
To assess MCC's compact implementation structures, we reviewed the
compacts with Madagascar, Cape Verde, and Honduras and the supplemental
agreements required for those compacts to enter into force. We
supplemented this review with our site visits to Madagascar and Cape
Verde. In all cases, our ability to analyze the adequacy of these
structures was limited by their relative newness and limited use in actual
implementation. We addressed the following four areas of MCC's
implementation structures:
o To determine the form of governance structures and key positions in
these three countries, we reviewed the requirements of MCC compacts and
supplemental agreements. We determined the progress of the country
organizations in Madagascar, Cape Verde, and Honduras in filling these
positions and establishing these structures by analyzing MCC's reported
staffing and status information. We independently assessed MCC's progress
in our site visits to Madagascar and Cape Verde. We discussed factors
affecting the filling of these positions through discussions with MCC and
compact country officials.
o To assess the adequacy of the countries' fiscal accountability
structures, we reviewed MCC's overall fiscal accountability framework and
the operations in place in Madagascar and Cape Verde. We assessed the
adequacy of these structures according to the criteria contained in GAO's
Standards for Internal Control in the Federal Government.1 We assessed
MCC's and the countries' implementation of these structures by using
criteria in the Internal Controls Capability Maturity Continuum developed
by the independent risk consulting company, Protiviti, Inc. We
independently verified the existence of the structures described in the
plan and discussed its strengths and weaknesses during the site visits.
o To assess MCC's procurement structures, we reviewed the MCC fiscal
accountability framework, and the implementing procurement documents for
the first three compact countries. We then assessed the adequacy of MCC's
framework, using criteria identified in previous GAO reports on
international procurement. To determine the status and factors affecting
the implementation of this framework in Cape Verde and Madagascar, we
interviewed compact country officials and obtained documentation of
procurement procedures.
o To determine the form of monitoring and evaluation structures in the
three countries with entry into force, we reviewed the requirements of MCC
compacts and supplemental agreements. We assessed the status of the
country organizations in Madagascar, Cape Verde, and Honduras to establish
these structures by analyzing the staffing and status information provided
to us by MCC. We independently assessed MCC's progress in our site visits
to Madagascar and Cape Verde. Additionally, we reviewed the scope of work
of the independent U.S. evaluation contractors retained by MCC, and
closely reviewed the monitoring and evaluation plan for Madagascar-the
only plan approved by MCC prior to April 2006. We also reviewed the plan
for Cape Verde and the draft plan for Honduras. We assessed the adequacy
of the Madagascar plan against the criteria of data quality, and
consistency with the economic model and logic identified in MCC's due
diligence review of projects. We also applied general principles of
economic logic, such as treatment of uncertainty in data, to assess how
uncertainty was incorporated into MCC's monitoring and evaluation
framework.
To review MCC's progress in responding to GAO's April 2005 recommendations
(see app. I), we examined MCC documents, such as its strategic plan,
planning documents, policies, procedures, and human capital documents. In
December 2005, MCC provided us with a letter outlining the steps that the
corporation had taken in response to our recommendations. Using this as a
basis for discussion, we held additional meetings with MCC officials and
received additional documentation of MCC's responses. We also reviewed the
findings of the MCC IG analysis of the functions of the Corporation and
met with the IG to determine the steps that MCC had taken in response to
IG findings related to our recommendations.
We conducted our review from June 2005 through May 2006 in accordance with
generally accepted government auditing standards.
Appendix III: MCC Projects in Madagascar and Cape Verde
Figure 14: Jatropha Plantation, outside Antsirabe, Madagascar, Targeted
for Assistance by the MCA-Madagascar Agricultural Business Investment
Project
Figure 15: Antananarivo Land Records Storage Room, Programme National
Foncier Offices
Note: Digitization of these land records is being supported by MCA funds.
Figure 16: Port of Praia, Cape Verde
Note: The quay of the current port can only accommodate one large
international vessel. Under the MCA program, the quay will be expanded to
almost double its current length, enabling three ships to dock at a time.
Figure 17: The Road from Assomada to Rincao, Cape Verde
Note: Using MCA funds, about half of the road will be converted from
cobblestone to asphalt. The cobblestones from the first half will be used
to pave the second section of the road leading up to Rincao.
Appendix IV: Summary of MCC Procurement Agents, Standards, and Provisions in
Madagascar, Cape Verde, and Honduras
Table 8: Procurement Agents and Standards in Madagascar, Cape Verde, and
Honduras
Country Type of procurement Procurement agent Procurement standards
agent
Madagascar Nongovernmental GTZa Madagascar Law Number
2004-009 Dated July 26,
2004 on Public
Contracts subject to
listed exceptions in
Procurement Agreement.
Cape Verde Governmental Cape Verde World Bank Guidelines:
accountable entity Procurement Under
International Bank for
Reconstruction and
Development Loans and
International
Development Association
Credits, May 2004, as
modified for
nonconsultant services,
works and goods.
World Bank Guidelines:
Selection and
Employment of
Consultants by World
Bank Borrowers, May
2004, as modified, for
consultants.
Ministry of
Infrastructure and
Transportation rules
apply to procurements
below $55,000 for
transportation
projects.
Honduras Hybrid governmental An outside project World Bank Guidelines:
and nongovernmental manager for the Procurement Under
transportation International Bank for
project, and a Reconstruction and
government agency, Development Loans and
the Fondo Nacional International
de Desarollo Rural Development Association
Sostenible, for the Credits, May 2004, as
Rural Development modified for
Project.b nonconsultant services,
works, and goods.
World Bank Guidelines:
Selection and
Employment of
Consultants by World
Bank Borrowers, May
2004, as modified, for
consultants.
Source: GAO analysis of MCC procurement documents.
aDeutsche Gesellschaft fu:r Technische Zusammenarbeit is a private company
owned by the German federal government.
bNational Fund for Sustainable Rural Development.
Table 9: Procurement Provisions, Roles, and Responsibilities Stated in MCC
Agreements with Madagascar, Cape Verde, and Honduras
Procurement-related Compact Disbursement Procurement
provisions agreement agreement
Document Outlines MCC's Includes Lists specific
approval rights conditions rules and
and documents the precedent to procedures that
responsibility of disbursements of the procurement
compact country funds from MCC to agent will
governments to the compact follow in
ensure oversight country that MCC managing
of procurement. can use to procurement for
enforce the the compact
requirements in country.
the procurement
agreement and
compact.
Compact country roles o Provide o Submit a report o Submit
and responsibilities oversight, ensure comparing actual reports; and
the removal or procurement obtain MCC
disclosure of any activity with approvals
conflicts of procurement plans before
interest, and for the second undertaking
conduct audits on most recent procurements
at least an annual disbursement above
basis. period as a thresholds.
condition
o Ensure precedent to o Report
transparency by disbursements. deviations from
establishing a Web approved
site for the o In Madagascar, procurement
program and provide a plan to MCC.
posting semiannual review Cape Verde and
procurement report on Madagascar to
documents online.a compliance with report
the procurement deviations
o Create guidelines for valued at more
procurement review MCC review. than $10,000.
bodies in both Honduras to
Cape Verde and o In Cape Verde, report all
Honduras. establish a deviations.
procurement
review commission o Provide a
and its charter, procurement
prior to plan
receiving MCC semiannually.
disbursements.b
o MCA-Honduras
must create and
approve a bid
challenge system
acceptable to MCC
prior to
receiving funds
for the second
quarter of
operations.
MCC role and approval o Approve the o Review and o Approve
authority members of Cape approve periodic procurements
Verde's plans and above
procurement review reports. thresholds.
commission and
Honduras's o Review and
procurement approve any
supervisor deviations from
agreement.c approved plan
above certain
o Authority to thresholds.
terminate the
compact if the o Review and
compact country approve the
government or compact
another party country's
materially procurement
breaches the plan, including
compact or any proposed
supplemental procurements
agreement, such as and methods of
a procurement procurement.
agreement.
o Requires the
signing of a
procurement
agreement prior to
entry into force.
Source: GAO analysis of MCC documents.
aIn Cape Verde, MCC compact funding is also supporting the creation of an
electronic procurement system. First, MCC is funding the establishment and
implementation of a public e-procurement system for use in compact
procurements. Second, it is funding the expansion of this system to all
other units of the government.
bAs conditions precedent to any disbursements for procurement management,
MCC has required that the government of Cape Verde will (1) request, as a
matter of national interest that requires urgent resolution of the
National Assembly, that the passage of a harmonized public procurement
law, with a supporting regulatory framework, be placed on the legislative
agenda and (2) establish a long-term program, with support in the national
budget, for the establishment of local procurement training capacity
(facilities and personnel), which will be sustainable without donor
funding within a period of 5 years.
cIn Honduras, the compact also requires MCC approval of a procurement
operations manual.
Appendix V: Comments from the Millennium Challenge Corporation
The following are GAO's comments on the Millennium Challenge Corporation
letter dated July 7, 2006.
GAO Comments
1.MCC stated that "MCC's mandate to engage directly with eligible
countries shortly after its creation did not allow for the possibility of
developing and vetting policies and procedures in advance of such
engagement." We recognize that MCC was simultaneously addressing a number
of issues during its initial years of operation-selecting eligible
countries, setting up MCC as an organization, and developing guidance and
policies while also working with countries to finalize compacts. In the
context of MCC as an evolving organization, we felt it was important to
discuss the evolving nature of guidance to provide a balanced perspective
regarding the process that eligible countries had to follow to sign
compacts.
2.MCC stated that our concern regarding baseline data quality stems from
one survey in Madagascar. However, we found issues with data reliability
in Madagascar with both the Agricultural Productivity Survey and Household
Survey. In the second survey, the error was smaller; therefore, we focused
on the Agricultural Productivity Survey to illustrate the more significant
example. The measurement of land values is very important for Madagascar's
monitoring and evaluation plan. Both the compact-level goal of increasing
household income and the program objective of increasing investment in
rural Madagascar are measured in terms of land values. In each of the
zones, the expected increase in household income is estimated at 5 percent
of the average land value, and the expected increase in investment is
estimated at 27 percent of the average land value. With a large error in
baseline data, it will be difficult to accurately track progress toward
the compact-level goal and program objective.
3.MCC noted that not every benefit can be isolated and measured during
implementation, and that disaggregating may not be feasible. We recognize
that not all outcomes of the economic analysis can be directly tracked
with indicators at different levels of monitoring or for impact
evaluation. There are trade-offs between cost and level of detail.
However, as we note in this report, aggregation poses some challenges that
could limit the effectiveness of monitoring and evaluation.
Appendix VI: Comments from the Department of State
The following are GAO's comments on the Department of State letter dated
July 10, 2006.
GAO Comments
1.In referring to our discussion about MCC's guidance on project
coherence, State Department commented that it would be erroneous to
indicate that countries were wholly unaware of the need to design their
compact programs to target compact goals, and that informal guidance from
MCC could have been received at any time. We agree that countries could
have requested informal guidance from MCC. According to MCC, they did
provide verbal guidance to countries regarding the need to link projects
to compact goals. However, MCC rejected tourism and preschool education
projects proposed by Honduras because they were not linked to the
impediments to growth that emerged from the consultative process. MCC also
rejected projects in Cape Verde because MCC's due diligence did not
indicate that these projects addressed key constraints in Cape Verde.
Given that these two countries submitted projects that did not meet
project coherence criteria, MCC's verbal guidance may not have been
sufficient to direct countries to submit proposals meeting these criteria.
We recognize that MCC was simultaneously addressing a number of issues
during its initial years of operation-selecting eligible countries,
setting up MCC as an organization, and developing guidance and policies
while also working with countries to finalize compacts. In the context of
MCC as an evolving organization, we felt it was important to discuss the
evolving nature of guidance to provide a balanced perspective regarding
the process that eligible countries had to follow to sign compacts.
2.State noted that we did not detail the methodology and composition of
our Madagascar focus groups. We summarized the composition of the focus
groups in the letter of this report. We met with these 29 individuals,
including representatives of MCA-Madagascar and Madagascar ministries,
during the course of our week in Madagascar and reviewed with them the
assumptions used in MCC's economic model for its projects. As previously
discussed in this report, we found that these officials were not closely
involved in MCC's economic analysis. According to MCC and Madagascar
officials, an MCC economist met with four Madagascar representatives in
Paris, France, during compact negotiations and reviewed the model with
them at that time. (MCC officials did travel to Madagascar during due
diligence to assess proposed projects.) When we discussed this with MCC
officials, they stated that not traveling to Madagascar to develop the
model was a mistake, and one that has not been repeated.
3.State commented that we do not rebut MCC's point that countries' degree
of involvement depended on their capability and willingness to
participate. (See comment 2.) In its letter, which is reprinted in
appendix V, MCC agreed that in some cases, the level of country engagement
on economic analysis could be improved. MCC also now requires that
countries engage an economist on the core country team, provides examples
of the economic rate of return analysis, and has MCC economists make early
technical guidance trips to work with the core country team and the core
country team economist. These procedures were not in place when MCC
conducted the economic analysis for Madagascar, Cape Verde, or Honduras.
By not thoroughly involving the countries in this analysis, MCC risks not
meeting its stated principle of focusing on results. In MCC's
results-based framework, the economic model is the basis for determining
performance targets against which funding is conditioned. MCA-Madagascar
may find that the targets are unrealistic if the targets have not been
thoroughly discussed with, and understood by, Madagascar officials.
4.State noted that the time for the in-country MCA organizations to hire
staff compares favorably with general government hiring. Management unit
officials are hired by a compact country's MCA program steering committee,
not the U.S. government. As such, it is not "general government hiring"
and should not be compared with it.
5.State correctly observed that we did not offer MCC a recommended
alternative to its taking the time to find good candidates for positions.
We did not make a recommendation for addressing this issue because MCC has
adopted a policy implementing the authority given it by section 609(g) of
the Millennium Challenge Act of 2003 to make grants to facilitate compact
development and implementation. MCC's current policy includes a provision
where, if certain conditions are met, it may fund an eligible country's
request for "management support payments" for salaries, rent, and
equipment for the country's team prior to compact signature. We have added
a discussion of this policy in the letter of this report to clarify this
point.
6.In referring to our discussion about the different maturity of internal
control between Madagascar and Cape Verde, State pointed out that
differences in the maturity of internal control between the countries are
to be expected, given that some countries have historically more
well-developed systems. We agree that different countries will differ in
their maturity of internal control due to historical differences in the
underlying systems and accountability infrastructures. The current
maturity of internal control in each country, however, is a key
consideration in assessing risk and establishing effective oversight
mechanisms to deal with the unique risks in the key financial processes
and activities of each country.
7.State referred to "GAO's reliance on project evaluation criteria that
make use of randomized controlled trials to measure success." This comment
misconstrues our findings on the use of randomized controlled trials. We
do not rely on these criteria. We are commenting on MCC's use of
randomized controlled trials, which MCC has indicated are its preferred
method of impact evaluation, and on the potential challenges to the use of
this approach. State's comments regarding cost-benefit analysis would be
more appropriately addressed to MCC.
8.State correctly noted that we did not capture the change to one
indicator for fiscal year 2006. We have updated the list of MCC indicators
in footnote 8.
9.State suggested rewording MCC's expected results to "MCC expects to
permanently raise incomes...." We did not use the word "permanently"
because in the three compacts that we focused on for this report, it was
not used to describe MCC's expected results at the country level.
Appendix VII: GAO Contact and Staff Acknowledgments
David B. Gootnick, Director, 202-512-3149
In addition to the person named above, Phil Herr (Assistant Director),
Claude Adrien, Cara Bauer, Gergana Danailova-Trainor, Jeanette Franzel,
Keith Kronin, Reid Lowe, Norma Samuel, Mona Sehgal, and Michael Simon made
key contributions to this report. Also, Tim DiNapoli, David Dornisch,
Etana Finkler, Ernie Jackson, Debra Johnson, Bruce Kutnick, Janice
Latimer, Charlotte Moore, and Celia Thomas provided technical assistance.
(320370)
www.gao.gov/cgi-bin/getrpt? GAO-06-805 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact David Gootnick at (202) 512-3149 or
[email protected].
Highlights of GAO-06-805 , a report to the Chairman, Committee on Foreign
Relations, U.S. Senate
July 2006
MILLENNIUM CHALLENGE CORPORATION
Compact Implementation Structures Are Being Established; Framework for
Measuring Results Needs Improvement
In January 2004, Congress established the Millennium Challenge Corporation
(MCC) to administer the Millennium Challenge Account. MCC's mission is to
reduce poverty by supporting sustainable, transformative economic growth
in developing countries that create and maintain sound policy
environments. MCC has received more than $4.2 billion in appropriations,
and, as of May 2006, it had disbursed $22.4 million to four countries
whose signed MCC compacts have entered into force. For the first three
countries with compact entry into force-Madagascar, Cape Verde, and
Honduras-GAO was requested to examine (1) key aspects that MCC reviewed,
and the criteria it used, in its due diligence assessments; and (2) the
structures that have been established for implementing the compacts.
What GAO RecommendsGAO recommends that the Chief Executive Officer of MCC
(1) ensure that economic analyses of compact proposals better reflect
country conditions and involve country participation and (2) improve
monitoring and evaluation by obtaining more reliable baseline data,
ensuring a clear linkage to economic analyses, developing criteria for
establishing and adjusting targets, and ensuring timely development of
evaluation designs. MCC generally agreed with GAO's recommendations; State
commented that some of GAO's findings reflect transitory problems.
MCC undertook a wide range of activities in its due diligence,
includingfive key aspects of the Madagascar, Cape Verde, and Honduras
proposals: (1) countries' consultation with local groups in developing
compact proposals, (2) projects' coherence with compact goals, (3)
environmental and social impacts, (4) institutional and financial
sustainability, and (5) impact on economic growth and poverty reduction.
MCC based its assessments on an evolving set of criteria: early, general
guidance to the countries followed by later, more specific guidance. MCC's
analyses of the projects' economic impact were limited in that some of the
assumptions and data used may not reflect country conditions. As a result,
the projects selected on the basis of the analyses may not achieve compact
goals. In the two countries we visited, Madagascar and Cape Verde, MCC
conducted the analyses with limited country participation, which resulted
in countries' having little understanding of the process.
MCC and the three countries have made progress in establishing compact
country structures for oversight and management, procurement, fiscal
accountability, and monitoring and evaluation, although some of these
structures are not yet complete. The oversight structures allow for
country management with MCC review, but some organizations were not fully
staffed for months after the compacts entered into force. Madagascar and
Cape Verde have implemented fiscal accountability structures for
MCC-funded projects, and established procurement structures with effective
characteristics; however, these structures are still largely untested and
some are still under development. Finally, MCC and the countries have
established monitoring and evaluation frameworks to track and account for
program results. However, limitations in the baseline data collected,
linkage to economic analyses, methods of addressing uncertainty associated
with program results, and the timely design of randomized controlled
trials may constrain MCC's ability to monitor and evaluate program
results.
Status of MCC Compact Development and Appropriations
a$22.4 million of the committed amount has been disbursed to begin compact
implementation.
*** End of document. ***