    U.S. International Broadcasting: Management of Middle East
    Broadcasting Services Could Be Improved (04-AUG-06, GAO-06-762).

    The Broadcasting Board of Governors' (BBG) broadcasting services,
    Radio Sawa, and the Alhurra satellite television
    networks--collectively known as the Middle East Broadcasting
    Networks, Inc. (MBN)--currently aim to reach Arabic speakers in
    19 countries and areas throughout the Middle East. Annual
    spending for current activities amounts to about $78 million. GAO
    reviewed MBN's (1) strategic planning to address competition in
    the Middle Eastern media market, (2) implementation of internal
    control, (3) procedures MBN has developed to ensure compliance
    with its journalistic standards, and (4) performance indicators
    and whether targets have been met.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-06-762
        ACCNO:   A58042
      TITLE:     U.S. International Broadcasting: Management of Middle
    East Broadcasting Services Could Be Improved
         DATE:   08/04/2006
      SUBJECT:   Broadcasting standards
    	     Competition
    	     Internal controls
    	     International broadcasting
    	     International relations
    	     Mass media
    	     Public diplomacy
    	     Radio
    	     Radio broadcasting
    	     Television
    	     Performance measures
    	     Employee training
    	     Alhurra Television
    	     Middle East
    	     Radio Sawa

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-06-762

         * Report to the Chairman, Subcommittee on National Security, Emerging
           Threats and International Relations, Committee on Government Reform,
           House of Representatives
              * August 2006
         * U.S. INTERNATIONAL BROADCASTING
              * Management of Middle East Broadcasting Services Could Be Improved
         * Contents
              * Results in Brief
              * Background
                   * Development of MBN
                   * MBN Funding
                   * MBN's Mission and Scope of Operations
                   * Radio Sawa's Operations
                   * Alhurra's Operations
              * Lack of Long-term, Comprehensive Planning Could Limit MBN's
                Ability to Respond to Market Challenges
                   * Radio Sawa and Alhurra Face Significant Competitive
                     Challenges
                   * MBN Has Taken Some Steps to Address Its Competitive
                     Challenges
                   * MBN Has Conducted Planning Exercises but Lacks a Long-term
                     Strategic Plan
              * MBN Has Made Progress in Developing Internal Control, but
                Additional Steps Could Improve Its Control Structure
                   * MBN Faced Initial Challenges in Developing Its Internal
                     Control Structure
                   * MBN Has Made Progress in Developing Its System of Internal
                     Control, but Some Control Elements Could Be Improved
                        * MBN's Internal Control Board Has Not Met to Establish
                          Protocols to Develop and Monitor Its Internal Control
                          Structure
                        * MBN Has Not Completed an Internal Control Plan
                        * MBN Has Not Established a Comprehensive Process to
                          Analyze Risk
                        * MBN Has Not Developed a Comprehensive Training Program
                          on Internal Controls
              * MBN Has Developed Editorial Procedures to Support Its
                Journalistic Standards, but Could Improve Monitoring and Some
                Quality Control Measures
                   * MBN Has Developed Journalistic Standards and Implemented a
                     Wide Variety of Pre- and Postbroadcast Editorial Procedures
                   * MBN Could More Fully Develop Some Quality Control Measures
                     and Lacks Regular Editorial Training
                   * MBN Lacks Comprehensive, Regular Program Reviews to
                     Determine Compliance with Mission and Journalistic Standards
                        * Annual Program Reviews Have Not Been Conducted for
                          MBN's Services
                        * Radio Sawa's Program Review Was Limited by Its Emphasis
                          on Audience Input
                        * Radio Sawa's Program Quality Score Is Inconsistent with
                          Other BBG Entities
                        * MBN Staff Had Limited Awareness of the Radio Sawa
                          Program Review and Follow-up
              * BBG Has Developed Several Performance Indicators, Although
                Weaknesses in Audience Surveys Call into Question Whether MBN
                Audience Size and Credibility Targets Have Been Met
                   * MBN Performance Indicators and Targets Have Been
                     Established, but Some Are Not Fully Developed
                   * BBG Should Increase Steps to Ensure Reliability and
                     Transparency of Its Performance Estimates
                        * Measurement and Research Methods Should Allow for
                          Confidence in Reliability of Performance Information
                        * BBG Has Not Taken Sufficient Steps to Ensure
                          Reliability and Transparency of Its Performance
                          Estimates for MBN
              * Conclusions
              * Recommendations for Executive Action
              * Agency Comments and Our Evaluation
         * Scope and Methodology
              * Review of MBN Internal Controls
              * Review of MBN's Editorial Procedures
              * Review of MBN Audience Survey Methods and Performance Indicator
                Estimates
         * Radio Sawa and Alhurra Programming
         * Radio Sawa Program Review
         * CIBAR Standards
         * Comments from the Broadcasting Board of Governors
              * GAO Comments
         * GAO Contact and Staff Acknowledgments

    Report to the Chairman, Subcommittee on National Security, Emerging
    Threats and International Relations, Committee on Government Reform, House
    of Representatives

    August 2006

    U.S. INTERNATIONAL BROADCASTING

    Management of Middle East Broadcasting Services Could Be Improved

    Contents

    Tables

    Figures

    August 4, 2006Letter

    The Honorable Christopher Shays Chairman Subcommittee on International
    Security, Emerging Threats     and International Relations Committee on
    Government Reform House of Representatives

    Dear Mr. Chairman:

    The President's 2006 National Security Strategy stipulates that public
    diplomacy, which includes U.S. international broadcasting, is an important
    instrument for engaging and informing foreign audiences, creating and
    maintaining a dialogue with Muslim leaders, and confronting misinformation
    about the United States. Under the direction of the Broadcasting Board of
    Governors (BBG), an independent agency, the mission of all U.S.
    international broadcasting services is to promote freedom and democracy
    and enhance understanding through objective journalism. The BBG's priority
    broadcasting services, Radio Sawa and the Alhurra satellite television
    networks1-both operated since 2005 by the nonprofit grantee Middle East
    Broadcasting Networks, Inc. (MBN)-currently aim to reach Arabic speakers
    in 19 countries and areas throughout the Middle East. Spending for MBN's
    broadcasting services has increased in recent years and currently amounts
    to about $78 million annually. In fiscal year 2005, MBN received
    additional funding to support a new Alhurra satellite television
    initiative to reach Arabic speakers in Europe. MBN has recently requested
    funding to enhance its radio and television programming content.

    This report assesses (1) the extent to which MBN has used strategic
    planning to address its competitive challenges in the Middle Eastern media
    market, (2) the progress MBN has made in developing financial and
    administrative controls to ensure that the organization operates
    efficiently and complies with its grant agreement, (3) the procedures MBN
    has developed to ensure compliance with its journalistic standards, and
    (4) the extent to which BBG has developed performance indicators for MBN's
    efforts and met its primary performance targets.

    To accomplish our objectives, we reviewed documentation and spoke with
    officials from the BBG and its broadcasting entities, including the
    grantees (MBN, Radio Free Europe/Radio Liberty (RFE/RL), Radio Free Asia
    (RFA)), the Voice of America (VOA), and the International Broadcasting
    Bureau. We also spoke with several outside experts, including
    representatives from the Department of State, National Public Radio, the
    InterMedia research organization, Zogby International, and foreign
    international broadcasters such as the British Broadcasting Corporation
    (BBC) and Deutsche Welle. In addition, we visited MBN's offices in Amman,
    Jordan, and Dubai, United Arab Emirates (UAE). We held telephone
    interviews with MBN's current affairs contractor in Beirut, Lebanon, and
    corresponded with the head of the MBN office in Baghdad, Iraq. We
    conducted our evaluation from August 2005 through May 2006 in accordance
    with generally accepted government auditing standards. (Appendix I
    provides a more detailed description of our scope and methodology.)

    Results in Brief

    MBN faces a number of challenges in carrying out its mission of
    broadcasting in the Middle Eastern media market and has taken some steps
    to address them. Although MBN has participated in BBG's long-term planning
    exercises, it has not yet established a formal long-term strategic plan. A
    formal documented strategic plan could help MBN meet challenges in its
    competitive marketplace. MBN's challenges include the fact that its
    Alhurra television station is currently competing in a market with over
    140 other stations and also faces operational and programmatic competitive
    disparities, such as having fewer overseas bureaus than its primary
    competitors. In addition, its Radio Sawa service lacks FM radio coverage
    in certain markets. MBN has begun a number of efforts and proposed
    initiatives to address some of its challenges, such as proposals for
    Alhurra to increase its hours of news coverage and current affairs
    programming and for Radio Sawa to increase the amount of local content it
    broadcasts to better compete with local stations. However, MBN lacks a
    shared vision of operations for Radio Sawa and Alhurra as well as detailed
    implementation strategies and related resource requirements for its
    proposed goals. According to GAO guidance,2 organizations should make
    decisions in the context of a strategic plan. The absence of a long-term
    plan and comprehensive strategic approach for MBN's operations limits
    opportunities for strategically addressing impediments to the
    organization's achieving its goals and objectives and identifying
    opportunities for more efficient operations.

    MBN has developed several financial and administrative controls to manage
    and safeguard its financial resources. For example, in the past year, MBN
    has hired additional financial staff, acquired a new financial information
    system, developed administrative and financial policies and procedures,
    provided some training to staff, and received an unqualified opinion on
    its fiscal year 2005 financial statements. However, additional steps could
    be taken to strengthen MBN's system of internal control,3 including (1)
    establishing an internal control board to formally develop controls and
    coordinate audits, (2) preparing an internal control plan, (3) conducting
    a risk assessment to address potential risks to its operation, and (4)
    developing a comprehensive training program for its staff. These steps
    could help MBN more effectively manage its operations and improve its
    internal control structure.

    MBN has procedures in place to help ensure its programming meets its
    journalistic standards and is consistently accurate, objective, balanced,
    and comprehensive. However, MBN has not fully implemented some editorial
    procedures, or a comprehensive, standardized program review process of its
    broadcast operations. To comply with its standards, MBN has put in place a
    variety of pre- and postbroadcast procedures, such as editorial meetings
    and postbroadcast discussions that enable producers and editors to review
    and provide input to Radio Sawa and Alhurra broadcasts on a daily basis.
    However, MBN could more fully develop some quality control measures, such
    as using listener and viewer feedback to improve program quality. In
    addition, although the BBG calls for its broadcasting services to undergo
    a standardized annual program review, MBN's Radio Sawa has had only one
    program review, and Alhurra has not had a program review to date.
    Furthermore, unlike reviews of other broadcasting services and contrary to
    BBG guidance, the Radio Sawa program review was less comprehensive in that
    it relied upon several audience panels but did not include input from
    in-house analysts or external expert listeners. The lack of annual,
    comprehensive program reviews hampers BBG's ability to assure its
    audience, Congress, and the BBG that its services are complying with its
    journalistic standards and mission.

    The BBG has developed performance indicators and targets4 related to
    audience size and program credibility for MBN's Radio Sawa and Alhurra
    broadcasting services. However, it is not clear whether the Radio Sawa and
    Alhurra performance targets have actually been met because of weaknesses
    in MBN's methodology and documentation. To develop estimates for audience
    size and credibility, the BBG uses results from audience research
    conducted overseas. Documentation we reviewed indicated that the BBG
    extensively uses nonprobability audience survey results that cannot be
    reliably projected to represent a broader population in the region. While
    it is not always possible to expect the BBG to use probability samples in
    countries where there are security risks, the BBG did not take certain
    steps that could have explained and increased the reliability of its
    estimates, such as fully documenting its research methods, measuring the
    level of uncertainty surrounding its estimates, disclosing significant
    limitations, limiting the scope of its projections to areas actually
    covered by its surveys, and developing and consistently implementing
    policies and procedures for verifying data.

    We are making several recommendations to the chairman of the BBG. First,
    we recommend that the BBG take steps to improve the efficiency of MBN and
    its ability to address challenges by ensuring that MBN develops a
    comprehensive long-term strategic plan. In addition, to improve controls
    and performance monitoring, we recommend that the BBG require MBN to
    implement additional elements of internal control, establish a regular
    mechanism for undertaking annual program reviews, implement performance
    indicators consistent with other BBG entities, and improve the
    transparency and documentation used for audience research to allow for
    assessments of confidence in data used for performance indicators.

    We received written comments on a draft of this report from the BBG. In
    general, the BBG concurred with our recommendations and said that MBN has
    made significant progress in the 2 years of its operation in establishing
    a sound journalistic organization with financial and administrative
    controls. However, the BBG raised a number of concerns about the report's
    criticisms of the audience research conducted by the BBG and its
    contractors.

    Background

    Middle East Broadcasting Networks, Inc. (MBN), which includes the services
    Radio Sawa and Alhurra, is an independent nonprofit grantee overseen by
    the Broadcasting Board of Governors (BBG). The BBG is an independent,
    federal agency responsible for overseeing all U.S. government-sponsored,
    nonmilitary, international broadcasting programs. The BBG also manages the
    operations of the International Broadcasting Bureau, the Voice of America
    (VOA), and the Office of Cuba Broadcasting, which are all federal
    entities. It also provides funding and oversight to three independent
    grantees, MBN, Radio Free Europe/Radio Liberty (RFE/RL), and Radio Free
    Asia (RFA) (see figure 1).

    Figure 1: Organization of the BBG

    Development of MBN

    In March 2002, due to concerns about the effectiveness of its outreach to
    Arabic speakers in the Middle East, the BBG replaced VOA's Arabic radio
    service with Radio Sawa, a 24-hour, 7-day a week, Arabic language radio
    station. In April 2003, Congress, at the request of the administration,
    provided $26 million in the fiscal year 2003 Emergency Wartime
    Supplemental Appropriations Act5 to establish a nonprofit corporation, the
    Middle East Television Network (MTN), as a grantee of the BBG to launch
    and operate Alhurra TV. Ten months later, on February 14, 2004, Alhurra,
    MTN's Arabic-language satellite television station, was launched and
    initially started broadcasting 14 hours a day, expanding to 24 hours a day
    2 months later. In November 2003, the fiscal year 2004 Emergency
    Supplemental Act for Defense and for Reconstruction of Iraq and
    Afghanistan6 included $40 million to establish a second 24-hour channel,
    Alhurra-Iraq. On April 27, 2004, Alhurra-Iraq was launched. In 2005, MTN
    was renamed the Middle East Broadcasting Networks, Inc. (MBN), and Radio
    Sawa transferred to MBN. See figure 2 for a timeline of these events and
    figure 3 for details on the current organization of MBN.

    Figure 2: Timeline Showing Creation and Key Dates of MBN

    Figure 3: Organization of MBN

    MBN Funding

    Congress appropriated more than $274 million to fund Radio Sawa, Alhurra,
    and Alhurra-Iraq from fiscal year 2002 through fiscal year 2006.7 Each
    year MBN's funding level has increased to support additional 24-hour
    television streams and Radio Sawa's 24-hour radio programming. MBN's grant
    in fiscal year 2006 is $78.7 million to support Alhurra, Alhurra-Iraq, and
    Radio Sawa, as well as the launch of Alhurra-Europe. See figure 4 for a
    breakdown of funds by fiscal year.

    Figure 4: Appropriations to Radio Sawa, Alhurra, Alhurra-Iraq, and Alhurra
    Europe, Fiscal Years 2002-2006

    MBN's Mission and Scope of Operations

    The Alhurra networks and Radio Sawa are BBG's priority broadcasting
    services designed to support the BBG's antiterror broadcasting initiatives
    in the Middle East and counter media campaigns used by terrorists by
    providing accurate reporting and analysis of the news and by explaining
    U.S. policies. Although MBN and its Alhurra broadcasting services postdate
    the BBG's current 2002-2007 strategic plan, Radio Sawa, in particular, was
    singled out as an opportunity in the plan to target Arabic-speaking youth
    and provide them with news that is objective, comprehensive, fresh, and
    relevant and to provide a forum for reasoned discussion of "hot button"
    issues and U.S. policies. MBN's current mission statement is to broadcast
    factual, timely, and relevant news and information about the Middle East,
    the United States, and the world to people of all ages in order to advance
    the long-term U.S. interests of promoting freedom and democracy and
    enhancing understanding in the Middle East. Radio Sawa and Alhurra aim to
    be among the sources that audiences turn to in the Middle East for news
    and information, to increase the standards of other broadcasters in the
    region, and to offer distinctive and provocative programming unavailable
    on other stations.

    MBN's target audience includes 19 Arabic-speaking countries and
    territories8 in North Africa, the Near East, and the Gulf region, which
    are home to approximately 250 million people. In addition to its
    headquarters in Springfield, Virginia, MBN has several overseas offices,
    including a production center in Dubai that broadcasts Radio Sawa's live
    newscasts during 8 hours each day and produces some opinion features for
    Radio Sawa and current affairs programming for Alhurra (see figure 5).9

    Figure 5: Map of MBN Target Countries and Locations of Overseas Offices

    Radio Sawa's Operations

    Radio Sawa's broadcasts are designed to reach a target audience of 15- to
    29-year olds in the Middle East with Western and Arab popular music, news
    broadcasts, and specialized programming. Radio Sawa broadcasts 24 hours of
    programming every day through a combination of FM, medium wave (AM),
    digital audio satellite, and Internet transmission resources. See figure 6
    for a map of Radio Sawa's regional reach.

    Figure 6: Radio Sawa's FM and AM Broadcast Range

    Radio Sawa has developed seven distinct programming streams, including (1)
    Iraq, (2) Jordan and the West Bank, (3) the Gulf, (4) Egypt and the
    Levant, (5) Morocco, (6) Sudan and Yemen, and (7) Lebanon. All of the
    streams generally feature the same major newscasts, current affairs, and
    policy features; however, the Iraq program differs slightly, and the
    streams all offer differentiated music programs.10 Radio Sawa's streams
    broadcast between 31 and 35 hours of news each week. (See appendix II for
    more on Radio Sawa's programming.)

    Alhurra's Operations

    The Alhurra satellite television station is designed to reach a broad
    audience in the Middle East by providing news, current affairs, and
    entertainment programming 24 hours a day, 7 days a week. Alhurra-Iraq is
    designed to provide Iraqi citizens with daily newscasts and talk shows
    that specifically address issues in Iraq. Both Alhurra networks broadcast
    between 36 and 43 hours of news and news updates a week.11 Alhurra
    broadcasts on the Arabsat and Nilesat satellites, which currently allow it
    to cover the entire Middle East. Alhurra-Iraq also broadcasts through
    these satellites and a combination of terrestrial transmitters made
    available in Iraq.12 Alhurra also expects to start broadcasting to Europe
    on August 1, 2006.13 (See appendix II for more information on Alhurra and
    Alhurra-Iraq programming.)

    Lack of Long-term, Comprehensive Planning Could Limit MBN's Ability to
    Respond to Market Challenges

    MBN faces a variety of challenges to broadcasting in the Middle East,
    including operating in a competitive satellite television broadcast
    market, operational and programming competitive disparities, and lack of
    coverage for Radio Sawa in certain FM markets. MBN has conducted some
    planning efforts and, by using market research and internal assessments of
    its competitors, has undertaken or proposed some initiatives to address
    many of these challenges, such as increasing its hours of news coverage
    and current affairs programming for Alhurra and increasing the amount of
    local content Radio Sawa broadcasts. However, MBN has not developed a
    long-term strategic plan that fully addresses its operational and
    competitive challenges.

    Radio Sawa and Alhurra Face Significant Competitive Challenges

    MBN faces several significant competitive challenges. These include the
    competitive Middle East satellite television market; operational and
    programming competitive disparities, such as Alhurra's lack of news
    bureaus compared with its competitors; and lack of coverage in certain FM
    radio markets.

    MBN operates in the competitive Middle Eastern satellite television market
    that has over 140 channels. Pan-Arab satellite television stations-in
    particular, the news stations Al Jazeera and Al Arabiya--are currently the
    primary competitors to Alhurra. According to the BBG's research firm
    Intermedia, Al Jazeera is currently the top international broadcaster as a
    source of news and information for audiences in many countries throughout
    the Middle East. Moreover, Alhurra will face new competition from the
    BBC's entry to the Middle Eastern satellite television market in 2007. BBC
    officials have indicated that the new station's overall approach in the
    region will be multimedia in focus, taking advantage of the BBC's more
    than 60 years experience of broadcasting on the radio to the region as
    well as its award-winning Arabic-language news Web site. For Radio Sawa,
    the primary competitive challenge comes from existing local radio stations
    in its broadcast range and the BBC World Service in Arabic, as well as
    from the generally increasing competitiveness of the Middle East radio
    market.

    Alhurra also faces operational and programmatic competitive disparities,14
    since both Al Jazeera and Al Arabiya are estimated to receive significant,
    although unknown, levels of funding from their respective supporting
    Qatari and Saudi financiers--allowing them to develop large networks of
    correspondents and bureaus throughout the Middle East and other parts of
    the globe. Al Jazeera, in particular, has bureaus in over 30 locations
    across 6 continents, which enable it to respond to breaking news events on
    a timely basis. Alhurra and Radio Sawa, by comparison, only have overseas
    bureaus in Baghdad, Dubai, and Amman. In addition, the BBC has a large
    network of correspondents and bureaus around the globe and, unlike
    Alhurra, has a vast in-house library of desirable BBC-produced content,
    including documentaries and current affairs programming, which can be
    readily translated into Arabic. The BBC also has favorable licensing and
    co-production arrangements with many companies.

    One of Radio Sawa's other primary challenges is its lack of broadcast
    coverage in certain countries in the Middle East region. For example,
    Radio Sawa does not have any broadcasting coverage in Tunisia, Libya, and
    Algeria. Moreover, it has faced difficulties expanding its transmission to
    include FM coverage in some countries, such as Egypt, Saudi Arabia, Syria,
    Yemen, and Oman. MBN is attempting to negotiate transmission agreements
    with several of these countries, but still faces significant challenges to
    finalizing agreements.

    MBN Has Taken Some Steps to Address Its Competitive Challenges

    Since its inception, MBN has conducted some planning exercises to address
    its competitive challenges. These have included developing a "2006 Goals
    and Strategies" overview document to guide operations for the current
    fiscal year, establishing a 2006 annual performance plan as part of the
    Office of Management and Budget's Program Assessment Rating Tool process,
    and participating in the development of the BBG's new long-term strategic
    plan covering fiscal years 2008-2012. In addition, MBN conducts ongoing
    assessments of its competitors and uses various types of market research
    to  gain information about its audience and media usage patterns in the
    Middle East. It has used this information to make adjustments to its
    programming within its current budget, and also to develop proposals for
    obtaining additional funding for new efforts. BBG and MBN officials have
    explained that they use audience surveys, audience monitoring panels,
    focus groups, in-depth interviews, and Arab television and music station
    monitoring to inform MBN's current efforts and planning, whenever
    possible.

    For example:

    o As a result of market research performed in August 2004, MBN officials
    identified television viewing patterns and made changes to Alhurra program
    schedules, such as by offering programming appealing to women (e.g.,
    current affairs and health and fitness programming) during the daytime.

    o Through a review of the current competition in the market, MBN officials
    decided that it was important to increase the number of debate programs
    they broadcast on Alhurra. As a result, MBN created a series of "town hall
    meetings" that allowed journalists and experts to discuss issues of
    regional interest with interaction from a live audience.

    o As a result of audience monitoring panels, MBN officials made changes to
    Radio Sawa's program schedule by adding new features on subjects such as
    social and cultural issues. MBN officials also learned of the importance
    of efforts to localize those features, and made changes to tailor programs
    to the interests of audiences of the various Radio Sawa streams.

    MBN has also developed program enhancement proposals for Alhurra and Radio
    Sawa as part of the BBG's language review and budget request processes.
    Several proposals were included in the President's fiscal year 2007 budget
    request. For example, the budget request includes a proposal to increase
    Alhurra's newsroom hours to increase on-the-spot and breaking news
    coverage.15 In addition, in fiscal year 2005, the administration requested
    additional funds for providing satellite Alhurra broadcasts to Europe.
    According to planning documents, many of the Alhurra proposals were
    designed to reinforce one another with the goal of improving Alhurra's
    credibility, as well as building audience size and increasing viewing time
    of those who already tune in.16 MBN also developed one proposal for
    enhancing Radio Sawa's operations that calls for increasing the amount of
    localized news content offered on five regional streams,17 which officials
    say would allow the station to more effectively compete with local
    stations in its broadcast range. MBN's president said that, given the
    increasing level of competitiveness in radio broadcasting in the Middle
    East and expressed audience interest in news about their home country,
    creating more localized content on Radio Sawa streams is important.

    MBN Has Conducted Planning Exercises but Lacks a Long-term Strategic Plan

    Strategic planning is a good management practice for all organizations.
    Although MBN has conducted some planning exercises, it lacks a long-term
    strategic plan and a strategic approach that outlines (1) a shared vision
    of operations for Alhurra and Radio Sawa, (2) detailed implementation
    strategies to achieve measurable outcomes for its goals, and (3) the
    competitive challenges it faces and how it plans to address its key
    challenges to broadcasting in the Middle East.

    Strategic planning, including the development of a strategic plan, is a
    good management practice for all organizations. Additionally, risk
    assessment is an integral part of strategic planning. According to GAO
    guidance,18 organizations should make management decisions in the context
    of a strategic plan, with clearly articulated goals and objectives that
    identify resource issues and internal and external threats, or challenges,
    that could impede the organization from efficiently and effectively
    accomplishing its objectives. Additionally, Office of Management and
    Budget (OMB) guidance suggests that strategic plans contain, among other
    things, a statement of the organization's long-term goals and objectives;
    define approaches or strategies to achieve goals and objectives; and
    identify the various resources needed and the key factors, risks, or
    challenges that could significantly affect the achievement of the
    strategic goals.

    MBN has yet to create its own long-term strategic plan. MBN's president
    stated that funding uncertainties and other more pressing organizational
    needs--such as the development of financial and administrative policies
    and procedures--have delayed the development of MBN's strategic plan and
    related planning policies. In addition, he commented that MBN did not
    emphasize planning in its early stages because it was focusing on making
    its networks broadcast-ready. BBG officials said another reason for the
    delays in planning is that the BBG and MBN are still learning about the
    market, especially for Alhurra, and are taking a close look at the results
    of audience surveys, focus groups, and in-depth interviews to determine
    the best direction for these initiatives. MBN has stated that, to date, it
    has primarily used the BBG strategic plan for organizational guidance.
    Nevertheless, BBG officials said the BBG also has the expectation that all
    broadcasting entities will develop their own strategic plans, particularly
    to guide funding decisions.

    In the absence of a strategic plan of its own, MBN lacks a comprehensive,
    strategic approach that fully outlines (1) a shared vision of operations
    for Alhurra and Radio Sawa, (2) detailed implementation strategies to
    achieve measurable outcomes related to its goals, and (3) the competitive
    challenges it faces and how it plans to address them.

    First, MBN does not have a comprehensive strategic vision for the
    integration of Radio Sawa and Alhurra operations in the organization. For
    example, although MBN's most recent annual performance plan contains a
    goal to "integrate news operations for more effective television and radio
    news-gathering," none of MBN's current plans outline specific, shared
    objectives for Radio Sawa and Alhurra. MBN officials told us that several
    steps toward integration of Alhurra and Sawa have occurred to date, such
    as sharing financial and administrative support staff. However, Radio Sawa
    and overseas bureau staff we talked with said that cooperation between
    Alhurra and Radio Sawa is limited, the identities of the stations are
    separate, and the two stations work largely independently of one another.
    Radio Sawa staff noted several areas for further increasing cooperation,
    including more sharing of interviews, sound bites, field correspondents,
    Web site stories, and copy editors. One staff member in the Baghdad bureau
    said that Radio Sawa and Alhurra operations in Iraq are completely
    independent, including separate offices, and only the financial activities
    of both offices are supervised by the same person. In addition to gaining
    more efficiency in operations, a vision for further integration of Radio
    Sawa and Alhurra may help MBN more effectively identify opportunities to
    address its challenges from increasing competition in the Middle East.

    Second, MBN has not yet developed detailed implementation and resource
    strategies needed for successful implementation of its goals. For example,
    with regard to MBN's initiative to localize content on Radio Sawa, we were
    not able to identify a plan directing what types of local news and
    features will be considered on the various streams, to what degree
    existing program schedules might be affected, and how required resources
    might be divided among the various streams. Additionally, although MBN's
    most recent annual performance plan states a goal of "expanding overseas
    production of news coverage for radio and television," neither that
    document nor any other plan MBN identified provides details or direction
    for the overseas production of news for Alhurra in existing overseas
    offices. MBN officials have stated that uncertainties in the future
    commitment of resources to Alhurra have affected MBN's ability to, for
    example, plan for and use existing overseas offices for Alhurra news.
    Further, BBG officials have said MBN is still learning about the Middle
    Eastern media market. However, given MBN's internal enhancement requests
    to the BBG to increase the number of news bureaus in the region, among
    others, it should clarify, for example, what implementation steps are
    necessary to maximize the use of existing overseas offices.19

    Third, MBN has not yet comprehensively outlined its challenges or
    developed a strategic approach for how it plans to address its key
    challenges to broadcasting in the Middle East. While MBN is planning to
    expand its broadcast operations into Europe, it has not clearly identified
    how its broadcasts will meet competitive challenges in the Middle East. As
    an example, MBN has not indicated how it will address the implications of
    the upcoming BBC Arabic-language television initiative. The BBC could gain
    a significant audience that potentially would interfere with Alhurra's
    market share, credibility, and use as a source of alternative
    information.20

    MBN Has Made Progress in Developing Internal Control, but Additional Steps
    Could Improve Its Control Structure

    MBN was initially limited in developing its internal control structure
    because it was focused on quickly starting up its broadcasting operations.
    In response to an external review of its financial operations by Grant
    Thornton LLP in May 2004, MBN strengthened several of its controls, after
    which it received an unqualified opinion on its Fiscal Year 2005 Single
    Audit.21 However, MBN has not fully implemented several of the Grant
    Thornton review's key recommendations related to its control
    environment,22 including (1) establishing an internal control board to
    formally develop its controls and coordinate audits, (2) preparing an
    internal control plan, (3) conducting a risk assessment to address
    potential risks to its operation, and (4) developing a training program
    for its staff.

    MBN Faced Initial Challenges in Developing Its Internal Control Structure

    Internal control refers to the policies and procedures that help ensure
    the proper management and application of an organization's assets. Clear,
    strong controls can provide some assurance that management problems are
    unlikely to occur or will be addressed if they do occur. MBN's internal
    control is governed by several OMB circulars cited in its grant
    agreement.23 The Comptroller General's Standards for Internal Control in
    the Federal Government24  also provides guidance that is available to MBN.

    MBN faced some initial challenges in establishing its internal control
    structure. According to MBN documents and officials, MBN management
    initially focused on establishing broadcasting operations rather than the
    development of internal control policies and procedures, because the
    organization had only several months to plan the launch of its 24-hour a
    day Alhurra television network. As a result, MBN's internal control lagged
    behind. The MBN chief financial officer (CFO) told us that problems
    encountered by MBN in hiring and retaining staff added to the delay in
    developing internal controls.

    Due to concern over the slow development of MBN's internal control
    structure, the BBG commissioned a review by Grant Thornton LLP accountants
    and management consultants, which was completed in spring 2004, to assess
    MBN's internal control and make recommendations for improvement. The
    report by Grant Thornton LLP cited numerous findings, such as inadequate
    financial policies and procedures, understaffing, and inadequate training,
    that impeded MBN's ability to successfully mitigate risks.25 The report,
    however, also noted that MBN's controls were improving.

    MBN Has Made Progress in Developing Its System of Internal Control, but
    Some Control Elements Could Be Improved

    Grant Thornton's May 2004 report on MBN's system of internal control made
    recommendations that covered staffing, MBN's financial system, training,
    administrative policies and procedures, developing a decision support
    structure, improving logs and records, and MBN's control environment. MBN
    accepted the review's recommendations and agreed to implement them,
    according to MBN officials, and the BBG concurred with these
    recommendations. Our analysis shows that MBN subsequently responded to
    many of the recommendations, including hiring additional financial
    department staff, developing financial and administrative policies and
    procedures, and completing an annual single audit (see table 1). For
    example, although MBN provided us with copies of its Fiscal Year 2003 and
    Fiscal Year 2004 Single Audits well past the deadlines for completing
    those audits,26 MBN's Fiscal Year 2005 Single Audit was completed on time,
    provided an unqualified opinion, and showed marked improvement over
    previous years.27

    Table 1: Actions taken by MBN to Address Grant Thornton's Internal Control
    Recommendations

              Recommendation                   Status of implementation
    Hire additional staff for the Completed: New vice president for finance
    financial department          and administration hired in August 2004.
                                     According to MBN, the organization now has
                                     16 financial staff, compared to the 6 who
                                     were working there when the assessment was
                                     conducted
    Acquire suitable financial    Completed: MBN acquired a new enterprise
    information system            resource planning system (Deltek-Costpoint)
                                     in October 2004 that has enhanced its
                                     ability to manage its finance department.
    Enhance and expand MBN's      Completed: Developed policies and procedures
    financial and administrative  that lay out roles and responsibilities.
    policies and procedures
    Improve procurement and       Completed: MBN's CFO told us that MBN
    travel logs and records       improved its procurement logs and records.
                                     In addition, MBN has hired a travel analyst
                                     who keeps a monthly log of travel orders and
                                     vouchers.
    Develop a system for          Completed: The finance department provides
    providing timely and focused  MBN high-level management with a detailed
    financial information to      financial package on a monthly basis..
    senior management
    Undertake and complete an     Completed: Completed  in late May 2006.
    annual audit as required by   MBN's Fiscal Year 2003 and Fiscal Year 2004
    OMB Circular No. A-133        Single Audits under Circular No. A-133 were
                                     not released within established deadlines.
                                     MBN's Fiscal Year 2005 Single Audit was
                                     completed on time.
    Establish an internal control Partially Addressed: MBN's internal control
    board                         board is not yet operational.
    Prepare an internal control   Not Completed: Not done as of May 2006.
    plan
    Conduct an MBN-wide internal  Partially Addressed: Preliminary risk
    control risk assessment       assessment did not identify objectives,
                                     identify risks, analyze the possible effects
                                     of the risks, or propose a strategy to
                                     mitigate them. This document merely contains
                                     a list of completed actions addressing the
                                     issues raised in the Grant Thornton LLP
                                     report.
    Provide training on MBN's     Partially Addressed: MBN has provided some
    internal control processes    training on internal control, but has not
                                     yet developed a regular training schedule
    Manage training by            and training is not managed by an internal
    establishing an internal      control board.
    control board

    Source: GAO analysis of information provided by Grant Thornton LLP and
    MBN.

    Some control elements could be improved in order to better implement best
    management practices based on OMB circulars and GAO internal control
    standards. For example, although MBN is establishing an internal control
    board, the board has not met to establish protocols and outline its
    responsibilities. The organization also has not developed an internal
    control plan. Furthermore, MBN has not established a comprehensive process
    to analyze risks the organization faces from internal and external
    sources. Finally, MBN has provided some training on internal control but
    has not yet developed a regular structured training program for its
    staff.28

    MBN's Internal Control Board Has Not Met to Establish Protocols to Develop
    and Monitor Its Internal Control Structure

    The Grant Thornton review recommended that MBN establish an internal
    control board of key managers and officers to determine the internal
    control risks facing MBN, work towards decreasing these risks, and oversee
    MBN's efforts to employ strong controls. Moreover, according to GAO
    guidance,29 organizations should have an audit committee or senior
    management council-similar to an internal control board-that reviews the
    internal audit work and coordinates closely with external auditors.

    According to the MBN president, MBN is establishing an internal control
    board consisting of three members, including, as recommended, the MBN
    president, general counsel, and CFO. However, the board has not yet
    formally met to establish protocols and outline responsibilities. In
    addition, according to MBN's general counsel, MBN has not appointed a
    member from the BBG, as was recommended, to serve on the board. MBN's
    executive committee, which examines issues affecting MBN and reports back
    to the BBG, has provided some support on management and administrative
    issues, such as approving the construction and expansion of MBN's new
    facility and providing guidance on hiring high-level staff.30 However, the
    committee has not fulfilled the role of an internal control board as
    previously described. Instead, MBN's CFO has largely taken on the sole
    responsibility of establishing and overseeing MBN's controls, reviewing
    audits, and coordinating with external auditors. The MBN president told us
    that they have not convened an internal control board because the
    organization is too new and, therefore, is focused on developing policies
    and procedures rather than mechanisms to review them. MBN is planning to
    develop an internal audit function, implemented by an external firm, to
    provide assurance to MBN management that the organization is operating
    appropriately.

    MBN Has Not Completed an Internal Control Plan

    According to the Grant Thornton review, MBN should develop an internal
    control plan to ensure that effective controls are established and
    monitored regularly. Such a plan should identify the roles and
    responsibilities of all individuals whose work affects internal control,
    lay out specific control areas, cover risk assessment and mitigation
    planning, and include monitoring and remediation procedures. MBN officials
    told us in January 2006 that they were in the process of developing such a
    plan and have developed internal control guidelines, but as of the end of
    May 2006, they had not provided us with a finalized plan.

    MBN Has Not Established a Comprehensive Process to Analyze Risk

    The Grant Thornton review called on MBN to conduct a broad risk
    assessment, led by its internal control board, to evaluate and mitigate
    potential obstacles to efficiently and effectively achieving its
    operational objectives. According to Grant Thornton LLP, failure to
    conduct an MBN-wide risk assessment could result in the loss of resources
    and could decrease confidence of the grantor and of Congress, which could
    ultimately compromise MBN's achievement of its mission. According to GAO
    guidance, risks should be identified during both short- and long-term
    forecasting and as part of strategic planning. Moreover, after conducting
    a risk assessment, organizations need to develop internal control
    activities to manage or mitigate the risks that have been identified. For
    example, Radio Free Asia identified avian influenza and signal jamming
    from China as two threats to their operations. As part of their risk
    assessment, they considered how to address and overcome these issues, such
    as by broadcasting from alternate locations.

    In February 2005, MBN prepared an initial risk assessment that identified
    a list of actions taken to address the issues raised in the Grant Thornton
    LLP report. However, the document did not identify the organization's
    objectives or the risks it faces, nor did it analyze the possible effects
    of the risks or propose a strategy to mitigate them, as recommended by GAO
    guidance. MBN officials told us that they assess risk on an ongoing,
    biweekly basis. However, in taking a short-term approach to analyzing
    risk, MBN lacks a comprehensive basis from which to establish a strong
    internal control structure. Some risks identified by MBN include threats
    to the security of its staff and bureaus in the field, particularly in
    Iraq, and the risk of a terrorist attack on its facilities in the United
    States. There are some risks that MBN has not identified or addressed. For
    example, PriceWaterhouseCoopers, MBN's external auditor, noted that MBN
    has not adequately addressed its risks related to information security.
    Doing so would reduce the risk of security incidents and unauthorized
    system activity, according to the auditor. PriceWaterhouseCoopers also
    found that MBN's lack of a business continuity plan or an adequate
    disaster recovery plan could result in slower recovery in the case of such
    an event, as well as significant loss of revenue, inability to meet
    customer needs and third party obligations, and potential noncompliance
    with legal requirements. The internal audit function planned by MBN may at
    some point take on the function of assessing risk, but this body is not
    yet operational.

    MBN Has Not Developed a Comprehensive Training Program on Internal
    Controls

    MBN has provided some training on internal controls but has not yet
    developed a regular structured training program for its staff of about
    240, as recommended by the Grant Thornton LLP review and by GAO leading
    practices.31 MBN's CFO and controller attended a seminar on grants
    management in October 2004 and subsequently shared the information with
    the 16 in-house financial staff. MBN also provided internal control
    compliance training to its managers in December 2004, and the organization
    regularly provides training to staff at its business manager meetings.
    However, other U.S. broadcasting entities, such as Radio Free Europe/Radio
    Liberty (RFE/RL) and Radio Free Asia (RFA), have more organized, ongoing
    training programs on internal control. The MBN CFO concurred that there is
    a great need for internal training. According to him, the underdeveloped
    training situation is due to a lack of resources, including a lack of
    funds specifically designated for training and limited time to plan or
    implement training. This lack of structured, recurrent internal control
    training can cause problems if staff are unfamiliar with an organization's
    business processes and controls, and can lead to the inefficient or
    improper use of resources.

    MBN Has Developed Editorial Procedures to Support Its Journalistic
    Standards, but Could Improve Monitoring and Some Quality Control Measures

    MBN has established journalistic standards, as well as procedures to help
    ensure that the organization's broadcasts comply with these standards.
    However, it has not fully developed some quality control measures, such as
    the use of listener and viewer feedback. Additionally, the BBG has not
    held regular comprehensive program reviews for MBN, thereby making it
    difficult for MBN to assure its audience, Congress, the administration,
    and the BBG that its controls are working and that it is broadcasting
    quality programming.

    MBN Has Developed Journalistic Standards and Implemented a Wide Variety of
    Pre- and Postbroadcast Editorial Procedures

    The International Broadcasting Act of 199432 calls for U.S. international
    broadcasting to be conducted in accordance with the highest professional
    standards of broadcast journalism, including the production of news that
    is consistently reliable, authoritative, accurate, and objective. The act
    also calls for U.S. international broadcasting to present a balanced and
    comprehensive projection of U.S. thoughts and institutions, as well as
    clear and effective presentation of U.S. government policies and
    responsible discussion of those policies. MBN's mission statement, which
    partly draws upon the principles and standards contained in the U.S.
    International Broadcasting Act of 1994, calls for MBN to broadcast
    factual, timely, and relevant news and information that promotes freedom
    and democracy.

    MBN has developed journalistic standards, including a code of ethics, as
    part of its effort to ensure that its news broadcasts are consistently
    accurate, authoritative, objective, balanced, and comprehensive. MBN's
    general counsel said that the code was also established to ensure that MBN
    fully complies with its mission and the U.S. International Broadcasting
    Act of 1994. MBN's journalistic standards were drafted by MBN management
    using input from professional journalistic organizations and another
    grantee. According to BBG officials, MBN's standards appear to be as good
    as those of other U.S. international broadcasters. In our analysis, we
    found that the standards cover areas similar to the codes of other
    broadcasters--such as RFE/RL, RFA, Voice of America (VOA), and National
    Public Radio--focusing on accuracy, impartiality, establishing context,
    clearly distinguishing analysis from reporting, using a tone of moderation
    and respect, avoiding advocacy, and promoting ethical conduct. The
    standards also include guidelines for conducting interviews, as well as
    editing and production requirements. For example, according to MBN
    officials, MBN strives to present opposing views accurately and achieve a
    balance among the guests on its current affairs shows. According to MBN
    officials, when broadcasting about the war in Iraq, they try to ensure
    that the program incorporates both pro-war and antiwar views.

    To help ensure that staff comply with MBN's journalistic standards, Radio
    Sawa and Alhurra have established a number of pre- and postbroadcast
    procedures, which are roughly similar to those of other U.S. international
    broadcasting entities. Examples of MBN editorial procedures include daily
    editorial meetings, the use of two or more sources to support a news item,
    checks by editors and producers to determine whether news stories are
    properly written and accurate, a headquarters-level review of all
    materials produced in MBN's overseas offices before broadcast,33 and
    postbroadcast discussions. In addition, we observed that MBN employs an
    experienced journalist from the Arab world to review all of Alhurra's and
    Alhurra-Iraq's weekday newscasts for technical and stylistic errors, a
    control not implemented by other U.S. broadcasting entities. The
    journalist watches the newscasts just before they are aired, then provides
    MBN management with an evaluation of the newscast's quality. MBN provided
    us with records of these evaluations, which assess whether the newscasts
    are presentable, balanced, and free of technical errors. In some cases,
    technical mistakes can be caught before the piece is aired. However, while
    MBN management follows up on critiques of individual journalists, they do
    not systematically review and assess the journalist's evaluations.

    MBN officials told us that the organization places a high value on
    journalistic controls, particularly due to the volatility of certain areas
    in the Middle East and the impact news reports can have. In addition,
    MBN's controls can serve as an assurance to its audiences and others that
    they are broadcasting quality programming. Since Radio Sawa and Alhurra
    were established, MBN has not had to retract a single story or apologize
    for any error, according to MBN officials.

    MBN Could More Fully Develop Some Quality Control Measures and Lacks
    Regular Editorial Training

    There are several areas in which MBN could more fully develop some quality
    control measures for its programming. These areas include using listener
    and viewer feedback to improve program quality, making better use of
    weekly compilations, and ensuring its style guide is distributed to all
    staff. Unlike other U.S. international broadcasters, MBN typically only
    partially utilizes the following measures:

    o Although MBN collects feedback from its listeners and viewers, it does
    not rely extensively on this feedback as a program quality control.

    o Although MBN produces weekly compilations that are distributed to
    interested parties summarizing what was broadcast on Radio Sawa and
    Alhurra that week, it does not use these compilations to do any formal,
    long-term analysis of errors and programming.

    o MBN has developed a style guide to provide critical guidance on the use
    of sensitive terms and to help staff avoid grammatical mistakes. However,
    Radio Sawa staff did not receive the guide until early 2006, and MBN has
    not distributed it to all of its overseas offices, inhibiting its use by
    some staff.

    In addition, MBN has not provided regular training to its journalists and
    producers. MBN told us that it does not provide regular training to help
    its editorial staff maintain and increase their professional competence.
    MBN's level of training for its editorial staff is also not on a par with
    the training offered by other grantees. For example, RFE/RL and RFA both
    have extensive training programs for their employees at their headquarters
    and in their bureaus in different countries, according to RFE/RL and RFA
    officials. While MBN provides some initial technical training to its
    journalists, an informal mentoring program, and sporadic training
    overseas, according to MBN officials, the organization does not have an
    ongoing training program to educate journalists throughout the
    organization about editorial and ethical issues they might encounter on
    the job. According to the MBN CFO, the organization does not have a
    well-established training program because the network is relatively new
    and lacks training resources. MBN officials have also noted that their
    most important controls are the editors themselves, and that MBN tries to
    hire experienced staff. However, regardless of experience, staff can and
    do make mistakes, and MBN's lack of regular training increases the risk
    that correspondents and editors will make mistakes.

    MBN Lacks Comprehensive, Regular Program Reviews to Determine Compliance
    with Mission and Journalistic Standards

    The BBG's main mechanism to determine whether its broadcasting services
    comply with its mission and journalistic standards is a regular program
    review, which is designed to improve programming and ensure quality
    control. However, only one review has been conducted for Radio Sawa, and
    none have been conducted for Alhurra. The Radio Sawa review was more
    limited in scope than program reviews conducted by other entities. In
    addition, Radio Sawa's program quality score is inconsistent with other
    BBG entities, and without a program review to develop a program quality
    score for Alhurra, the BBG will not be able to measure the contribution of
    these efforts to the goals of the organization, or be able to ensure that
    the quality of Alhurra's broadcasts conforms to applicable standards.
    Finally, many MBN staff were not extensively involved in the Radio Sawa
    review.

    Annual Program Reviews Have Not Been Conducted for MBN's Services

    According to written guidance from the International Broadcasting Bureau,
    which coordinates and supports all VOA program reviews, program review is
    an annual process by which an institution judges itself and solicits the
    judgment of others to make improvements and fulfill its mission with
    regard to U.S. national interests. The process enables the broadcasting
    entity to better connect its mission to the market where it is
    broadcasting and assess whether its editorial procedures are functioning
    effectively, while also allowing the BBG to fulfill its requirements from
    OMB that it conduct regular program evaluations to capture a program's
    impact over time. In addition, the Senate Committee on Foreign Relations
    Report on the Fiscal Year 2003 Foreign Relations Authorization Act called
    for significant resources to be dedicated to postbroadcast analysis of
    Radio Sawa programming to ensure that broadcasts are consistent with U.S.
    interests and values and with the standards in the U.S. International
    Broadcasting Act. Program review typically includes a study of the target
    area's media environment, analyses by internal and external reviewers,
    background quantitative research, reports on marketing and transmission,
    and target area profiles compiled by the broadcaster itself. Reviewers
    rate programs

    based on criteria34 for content--such as accuracy, timeliness,
    objectivity, relevance, and quality of analysis and interviews--and for
    presentation, such as pace and liveliness, presentation style, sound
    quality, and host interaction. These inputs are then discussed at a
    meeting that includes the program review coordinators and the management
    and staff of the entity being reviewed. Following the main program review
    meeting, key participants develop an action plan, and 3 months later the
    group meets again to determine to what extent the action plan has been
    carried out.

    Although the BBG calls for program reviews to be conducted annually, MBN
    has not complied with this guidance. Neither the BBG nor MBN has a regular
    mechanism in place to systematically review MBN programs. In December
    2004, the BBG convened a program review meeting for Radio Sawa, which had
    begun operating in March 2002. However, there has been no Radio Sawa
    review since then, while Alhurra, which has been operating for more than 2
    years, has had no program review at all. The BBG is planning to initiate a
    program review of Alhurra by the end of this year, but has not set a firm
    date.

    Radio Sawa's Program Review Was Limited by Its Emphasis on Audience Input

    BBG officials have stated that program quality should be sampled and
    assessed by both internal and external evaluators.35 This is designed to
    produce a balanced and robust review.36 In 2000, in response to our report
    that found a lack of consistency in how program quality scores were
    developed, the BBG stated that it intended to harmonize and standardize

    program reviews across broadcasting entities.37 BBG guidance now calls for
    all U.S. international broadcasting entities to be evaluated using the
    same standards, definitions, and scoring methods.

    However, when conducting the Radio Sawa program review, the BBG relied
    only on audience monitoring panels to assess program quality and did not
    utilize internal analysts or external control listeners, as is common
    practice among other U.S. international broadcasters. Using only audience
    monitoring panels gives the audience more weight in the review results
    and, in turn, more potential to influence the strategic direction of the
    organization. In addition, since the audience tends to be unfamiliar with
    a broadcasting service's journalistic standards and editorial procedures,
    having input only from monitoring panels makes it more difficult for a
    service to provide reasonable assurance that its editorial procedures are
    working and that it is broadcasting quality programming. Also, external
    control listeners are specifically tasked with examining the programming
    in light of the service's mission, something audience monitoring panels
    are not asked to do. As a result, Radio Sawa's program review strongly
    emphasized the audience's perspective and therefore provided a less
    thorough evaluation of Radio Sawa's mission and standards. (For more
    information on Radio Sawa's program review, see appendix III.)

    Radio Sawa's Program Quality Score Is Inconsistent with Other BBG Entities

    Furthermore, MBN's program score for Radio Sawa is not comparable with
    other BBG broadcaster scores for program quality--the percentage of a
    station's language services judged on both content and presentation
    criteria as being of good-or-better quality. The BBG says that to measure
    Radio Sawa's program quality, it has developed standardized criteria
    applicable to different media and methods of delivery, while minimizing
    subjective judgments on content and presentation. Although the criteria
    used to measure Radio Sawa's program quality are similar to those used by
    other broadcasting entities, the BBG did not use as many inputs when
    calculating the program quality scores for Radio Sawa, leading to a less
    robust result (see table 2).

    Table 2: Inputs to Program Quality Scores for BBG Broadcasting Entities

    Broadcaster      Program review input  Percentage of total program quality
                                                                            score
    MBN (Radio Sawa) Monitoring panels                                     100
    RFE/RL           Internal analysis                                      45
                        Monitoring panels                                      35
                        External experts                                       20
    RFA              Internal analysis                                      45
                        Monitoring panels                                      35
                        External experts                                       20
    VOA              Internal analysis                                      45
                        Monitoring panels                                      35
                        External experts                                       20

    Sources: BBG, International Broadcasting Bureau, RFE/RL, RFA.

    Without a consistent process for broadcaster program reviews, the BBG is
    limited in its ability to assess and compare broadcaster performance. In
    addition, without a program review to develop a program quality score for
    Alhurra, the BBG will not be able to measure the contribution of these
    efforts to the goals of the organization, or be able to ensure that the
    quality of Alhurra's broadcasts conforms to applicable standards.
    Currently, the BBG is considering how it will conduct the future program
    reviews for Radio Sawa, but officials could not yet provide specifics
    about the approach BBG will use.

    MBN Staff Had Limited Awareness of the Radio Sawa Program Review and
    Follow-up

    MBN staff in the United States and overseas do not appear to have much
    knowledge of the Radio Sawa review or its follow-up. Program review can be
    a learning experience for staff, who are usually encouraged to sit in on
    their language service's program review meeting, and the process can also
    bring people together who normally do not interact, which can help
    generate ideas and improvements. However, according to MBN management,
    most Radio Sawa staff did not attend the Radio Sawa program review. In
    addition, there was also not much awareness about the program review in
    MBN's overseas offices.

    While Radio Sawa staff lacked awareness about their review, some staff who
    had worked at VOA in the past spoke positively of the program review
    process in general. Other broadcasters routinely involve staff in their
    program reviews. For example, RFA requires all staff from the service that
    is being reviewed to be present at the program review meeting. According
    to an RFA official, program review is one of the few times that all of the
    key players for a service are in one place, and it therefore presents a
    good opportunity for communication. In addition, a BBC official noted that
    program review can be a good way to expose staff to an organization's
    values. Increasing staff involvement in program review could therefore
    increase chances for communication throughout all levels of the
    organization, as well as provide a forum for discussing potential
    programming improvements.

    BBG Has Developed Several Performance Indicators, Although Weaknesses in
    Audience Surveys Call into Question Whether MBN Audience Size and
    Credibility Targets Have Been Met

    The BBG has established several standard performance indicators and
    targets for MBN programs, including measures of audience size and
    credibility; however, it has not implemented some performance indicators
    fully, including a program quality measure. Additionally, we were unable
    to determine the accuracy of MBN's audience size and program credibility
    estimates due to weaknesses in MBN's methodology and documentation.
    Therefore, it is not clear whether the Radio Sawa and Alhurra performance
    targets have been met.

    MBN Performance Indicators and Targets Have Been Established, but Some Are
    Not Fully Developed

    The Government Performance and Results Act of 1993 (GPRA) requires that
    all government agencies establish performance indicators, or measures,
    that provide a meaningful reading of how well the organization is
    progressing towards its goals. The BBG has developed a standard set of
    performance indicators for its broadcasting entities, which it says are a
    best effort to measure its level of effectiveness now and where its
    performance is targeted to be in the future. The BBG established common
    indicators for its entities to allow it to better assess overall progress
    for the organization. Although the BBG has made progress in establishing
    these standard performance indicators, as well as targets for MBN's
    programs, it

    has not formally established or implemented all of them for Radio Sawa and
    Alhurra.38

    According to BBG, the three most important standard indicators for its
    entities, referred to as primary performance indicators, are (1) audience
    size, or the overall weekly audience of a station; (2) credibility, which
    represents the percentage of viewers in a target area that consider the
    station's news programs somewhat or very reliable; and (3) program
    quality, or the percentage of a station's programs judged on standard
    criteria as being of good-or-better quality. These three indicators are
    tied to the current BBG mission and strategic plan for U.S. international
    broadcasting.39 In addition to its standard primary performance
    indicators, the BBG has a number of "secondary" measures that provide
    management with additional information for gauging cost- effectiveness,
    marketing and promotion activities, and transmission efforts, including
    the number of transmitters and affiliates, cost per listener, signal
    strength (radio only), and awareness.

    The BBG has established and implemented two primary indicators-audience
    size and credibility--for both Radio Sawa and Alhurra.40 The BBG has also
    established performance targets for Radio Sawa and Alhurra for these two
    indicators. For example, MBN's reported fiscal year 2005 audience size
    performance target for Radio Sawa was 18 million listeners and for Alhurra
    was 12.8 million viewers.41 However, the BBG has not yet established
    Alhurra's program quality indicator, or consistently implemented this
    measure for Radio Sawa. Without a measure of program quality for Alhurra
    and Radio Sawa, the BBG will not be able to consistently assess MBN's
    performance against that of other grantees, or fully assess MBN's
    contribution to the overall goals of the BBG organization. The BBG has
    established all of its standard secondary performance indicators for MBN's
    services except for audience awareness.42 See table 3 for a list of the
    performance indicators implemented for Alhurra and Radio Sawa.

    Table 3: BBG's Standard Performance Indicators and MBN's Status

     Indicator      BBG         Definition       Source of     Current     Current
        type    performance                    data used to    Alhurra      Sawa
                 indicator                       calculate   performance performance
                    name                          measure     indicator   indicator
     Primary    Audience     Total number of   This figure   Yes         Yes
                size (also   all adults        is calculated
     indicators called       listening or      by the office
                overall      viewing a         of research
                weekly       service's         and is an
                audience)    programming at    estimate
                             least once a week derived from
                             in target areas   audience
                                               surveys and
                                               estimates of
                                               national
                                               population.
                Credibility  Percentage of     This figure   Yes         Yes
                             total estimated   is calculated
                             weekly            by the office
                             viewers/listeners of research
                             in target areas   and is an
                             that determined   estimate
                             the reliability   derived from
                             of news and       the results
                             information of    of audience
                             the station to be surveys and
                             somewhat or very  estimates of
                             reliable          overall
                                               weekly
                                               audience.
                Program      Percentage of a   This figure   No          Yes
                quality      station's         is calculated
                             language services by the office
                             whose programming of strategic
                             is assessed as    planning and
                             being of          performance
                             good-or-better    management.
                             quality           For Radio
                                               Sawa, this
                                               figure is
                                               determined by
                                               scoring from
                                               a monitoring
                                               panel of
                                               regular
                                               listeners
                                               using
                                               predefined
                                               criteria for
                                               content and
                                               presentation.
     Secondary  Awareness    Percentage of     This figure   No          No
     indicators              adults in the     is calculated
                             target area that  by the office
                             have prompted and of research
                             unprompted        and is an
                             awareness of the  estimate
                             station           derived from
                                               the results
                                               of audience
                                               surveys and
                                               estimates of
                                               overall
                                               weekly
                                               audience.
                Affiliations Total number of   This figure   Yes         Yes
                and          affiliates that   is calculated
                transmitters regularly         by the office
                             rebroadcast       of marketing
                             identified        and program
                             programs on       placement
                             competitive media using
                             and number of     affiliates
                             owned and         data and the
                             operated local    office of
                             transmitters      engineering
                                               using
                                               transmitter
                                               data
                Signal       Quality of radio  This figure   N/A         Yes
                strength     signal monitored  is calculated
                             and rated on a    by the office
                             scale of nil to   of
                             excellent         engineering
                                               using data
                                               sampled
                                               periodically
                                               from overseas
                                               monitoring
                                               stations.
                Cost per     Budget divided by This figure   Yes         Yes
                listener     weekly audience   is calculated
                                               by BBG's
                                               office of
                                               strategic
                                               planning and
                                               performance
                                               management
                                               using budget
                                               data and
                                               audience size
                                               figures.

    Source: GAO and BBG.

    BBG Should Increase Steps to Ensure Reliability and Transparency of Its
    Performance Estimates

    We and others have noted that agencies' performance indicators and data
    should provide a reliable means to assess progress. However, we were
    unable to determine the accuracy of MBN's reported audience size and
    program credibility estimates due to weaknesses in MBN's methodology and
    documentation. Therefore, it is not clear whether the Radio Sawa and
    Alhurra performance targets have been met. While BBG has taken several
    important steps to ensure the validity and reliability of its performance
    measurement approach, it has primarily used a methodology that cannot be
    reliably projected to the broader population. Although it is difficult to
    conduct probability sampling in many locations in the Middle East, the BBG
    has not taken steps to explain and increase the reliability of MBN's
    performance information, such as by maintaining more detailed
    documentation to support its estimates, reporting significant data
    limitations, limiting the scope of its projections to areas actually
    covered by its surveys, and developing BBG policies and procedures for
    verifying performance data.

    Measurement and Research Methods Should Allow for Confidence in
    Reliability of Performance Information

    We have previously reported that performance indicators should provide a
    reliable way to assess progress.43 In particular, agencies should
    implement quality control procedures to mitigate errors that can occur at
    various points in the collection, maintenance, processing, and reporting
    of performance data and can impact its reliability. In addition, agencies
    should select sampling methods that ensure representative samples, where
    possible. For example, probability surveys are designed to ensure each
    person in the population has a measurable chance of being selected for the
    survey, enabling the results to be reliably projected to the larger
    population with known levels of precision. Additionally, agency
    performance reporting should provide sources, disclose limitations, and
    discuss the implications of them. Explaining the limitations of
    performance information--as well as actions taken to compensate for
    low-quality data-can provide context for understanding and assessing
    agencies' performance and the costs and challenges that agencies face in
    gathering, processing, and analyzing needed data.44

    The U.S. International Broadcasting Act of 1994 requires the BBG's use of
    audience research to guide its decisions about its language services. For
    the past 5 years, the BBG has contracted with Intermedia to serve as the
    primary research contractor for the BBG and its broadcasting entities,
    including MBN. Intermedia works with subcontractors, partners, or both to
    gather audience research survey data from citizens in various locations
    overseas. This survey information is used by the BBG to develop estimates
    for MBN and other entities' audience size and credibility performance
    indictors. For example, for fiscal year 2005, the BBG estimated that Radio
    Sawa and Alhurra had each achieved an audience size of about 21.6 million
    people. Table 4 shows the breakdown of individual survey data reported by
    the BBG.

    Table 4: BBG's Reported Audience Size and Credibility Estimates for Radio
    Sawa and Alhurra, Fiscal Year 2005

    Country          Year      Sawa  Alhurra estimated        Sawa     Alhurra
    surveyed     surveyed estimated      audience size credibility credibility
                              audience                           rate        rate
                                  size
    Algeria          2004       N/A          2,466,888         N/A         N/A
    Bahrain          2005   127,008            132,814          78          86
    Djibouti         2003   151,620                N/A         N/A         N/A
    Egypt            2005 4,791,552          2,019,968          82          92
    Iraq             2005 8,336,320          6,860,435          63          73
    Jordan           2005   965,216            729,120          75          68
    Kuwait           2005   299,739            249,612          82          72
    Lebanon          2005    68,400            738,720          68          79
    Morocco          2005 4,072,455          2,679,367          64          70
    Qatar            2004   324,786                N/A          79         N/A
    Saudi Arabia     2005   734,928          2,204,784          72          73
    Syria            2004 1,468,800          2,661,120          76          60
    Tunisia          2005       N/A            676,800         N/A          75
    UAE              2005   265,842            138,510         79a          82
    Regionwide                 21.6       21.6 million          69          73
                              milliond
    fiscal year
    2005
    estimateb,c

    Source: BBG.

    aThe BBG originally reported this figure as 55 percent, but later
    determined that the correct figure was 79 percent.

    bTo develop the regional estimate for audience size, listening/viewing
    rates are gathered from the national surveys and are, in most cases,
    multiplied by each country's respective national population estimate to
    obtain a total audience size estimate for each country. For some
    countries--such as Bahrain, Kuwait, Qatar, UAE and Saudi Arabia--the
    results are projected to nationals and Arab expatriate populations only.
    These estimates are developed for all countries surveyed and then added to
    develop a regional audience figure, which is the service's overall weekly
    audience estimate.

    cTo develop the regional estimate for credibility, percentage credibility
    scores are gathered from national surveys and multiplied by that country's
    audience size estimate to obtain a total number of listeners or viewers
    for a given country that view the station's news and information as very
    or somewhat reliable. These estimates are developed for all countries
    surveyed and then used to create a weighted average of credibility.

    dThe BBG reported in its fiscal year 2007 budget request that its audience
    size for Alhurra was 21.5 million.

    BBG Has Not Taken Sufficient Steps to Ensure Reliability and Transparency
    of Its Performance Estimates for MBN

    Although the BBG has taken several important steps to enhance the validity
    and reliability of its audience survey designs, several factors call into
    question the accuracy of the data used for BBG's audience size and
    credibility performance indicators for Radio Sawa and Alhurra. First, we
    observed that, in 12 out of 14 cases, the BBG used nonprobability surveys,
    which cannot be reliably projected to the broader population, to develop
    its regional estimates for audience size and credibility. While it is
    difficult to conduct probability surveys in hostile environments, such as
    those in Iraq and Saudi Arabia, the BBG did not take certain steps that
    could have increased the accuracy of its estimates and explained their
    limitations, thereby increasing confidence in the data. Therefore, we are
    unable to determine whether MBN actually met its performance targets for
    fiscal year 2005.

    BBG has taken several important steps to enhance the validity and
    reliability of its audience survey designs. For example, the BBG's
    questionnaires are reviewed by multiple parties, and its contractors
    extensively pretest the questionnaires in the field, conduct pilot
    studies, and use throwaway surveys45 for training new subcontractors. In
    addition, the BBG's contractors exercise quality control when collecting
    data and receiving it from the field, including by conducting preliminary
    electronic testing of the data, among other things.

    However, in 12 of 14 cases, we found the country-level estimates used for
    generating Radio Sawa and Alhurra's fiscal year 2005 performance
    indicators of audience size and credibility were not based on probability
    survey46 results. Many of  the surveys conducted used judgment sampling, a
    form of nonprobability sampling, instead.47 BBG officials told us that its
    use of nonprobability surveys for certain countries is due to either the
    cultural, political, or security situation in those countries, which
    limits the selection of individuals or the geographical areas that can be
    surveyed.48 In addition, the International Broadcasting Bureau director of
    research stated that in many developing countries, existing map and
    population data is not adequate to support pure probability-based
    sampling. The Conference of International Broadcasters' Audience Research
    Services, called CIBAR, whose standards are specified as a source of
    guidance for BBG research contractors, requires that audience measurement
    use samples based on the principles of random probability and that other
    sampling methods should only be used in cases where, for reasons of
    practicality or cost, proper random samples cannot be used. (See appendix
    IV for more on the CIBAR standards.)

    We recognize that many agencies face challenges in collecting credible
    performance data and that, due to security risks and political
    considerations in many Middle Eastern countries, it is not always possible
    to expect BBG to use random samples. However, the BBG did not take certain
    steps that could have explained and increased the reliability of its
    estimates, such as fully documenting its research methods, measuring the
    level of uncertainty surrounding its estimates, disclosing significant
    limitations, limiting the scope of its projections to areas actually
    covered by its surveys, and developing and implementing procedures for
    verifying data.

    First, the BBG and its research contractors were unable to provide us with
    certain documentation commonly required by international broadcasting
    research standards. CIBAR requires that, in all measurement research, the
    sampling methods used and other technical aspects of the survey be both
    fully and accurately described in the project documentation and open to
    independent scrutiny. We asked the BBG and its contractors to provide us
    with detailed documentation-including clear information on sampling plans
    and related assumptions, response rates, and adjustments applied to the
    data to reflect the effects of the survey design-for all 14 of its Middle
    Eastern country surveys used to develop its fiscal year 2005 performance
    indicator estimates. For two cases, the BBG was unable to provide us with
    any survey documentation,49 and for all but one case, the BBG and its
    contractors were unable to provide us all the detailed information we

    requested.50 BBG officials acknowledged the lack of complete documentation
    in contractor technical reports, and said it was due in part to their
    failure to follow up with the contractors to obtain the details, aswell as
    to the contractors' general practice of not generating such detailed
    documentation.

    Second, the BBG has not sufficiently measured the level of uncertainty
    surrounding MBN's performance estimates. CIBAR requires that in all
    measurement research, technical aspects of the survey, including margins
    of error and confidence levels where appropriate, be both fully and
    accurately described in the project documentation. The BBG has not been
    able to measure sampling errors51 for its surveys, in part because it has
    not required contractors to document the information that is needed to
    calculate the sampling errors accurately. Moreover, officials said that it
    is not customary for their contractors to maintain this information.
    Therefore, the BBG only has a rough idea of what the margins of error52
    might be for its surveys, further limiting confidence in the reliability
    of its current performance information. One research official told us that
    he believed that the overall margins of error for Radio Sawa and Alhurra's
    audience size and credibility estimates are large, but said that currently
    the agency cannot accurately calculate them.53

    Third, the BBG lacks transparency in reporting data sources and
    significant limitations affecting MBN's audience size and credibility
    performance information. In reporting performance data, agencies should
    provide data sources, disclose limitations, and discuss the implications
    of them. CIBAR standards also recommend that proper care be exercised when
    reporting estimates, to ensure that the type of audience covered by the
    estimate is clearly stated and that, at all stages in the calculation and
    extrapolation process, sources, assumptions, and methods be fully
    documented and available for independent scrutiny. However, the BBG has
    not sufficiently explained the specific methods used for generating
    estimates for its performance indicators, such as the number and names of
    the countries surveyed, methods of sampling used, sources of the
    population data, and basic procedures used to create the estimates.
    Moreover, the BBG has reported only two limitations for MBN performance
    indicators to date: (1) that credibility ratings are highly dependent on
    volatile political factors; and (2) that, depending on political, social,
    and media conditions, measurement of audience size may either be easily
    attained or impossible. However, the BBG has not explained many
    significant limitations, and their resulting implications, on the data
    reliability of the performance information for MBN. The largest
    identifiable limitations not reported are that many of the BBG surveys are
    not based on probability sampling, cover only part of the country, have
    very low response rates, or have high substitution rates; therefore, those
    results cannot be reliably projected to be representative of the larger
    population.54 In the case of the survey in Morocco, we calculated that the
    survey only covered 35 percent of the general population and had a 48
    percent substitution rate, but the results were projected to represent a
    broader population. In addition, we calculated low response rates for a
    number of MBN's surveys; in the case of the survey of Egypt, the response
    rate was about 19 percent. BBG research staff explained that their
    stakeholders to date, including the BBG, have not required such a level of
    detail in reporting, and specifically have not required margins of error.
    However, a discussion on data limitations in performance reporting can
    help decision makers determine their level of confidence in the agency's
    ability to report on its performance goals and indicators and identify
    actions needed to improve its ability to measure performance.

    Fourth, we found the BBG has not taken sufficient steps to avoid
    projections to areas outside the population surveyed. For example, the
    elimination of projections to the rural population of Morocco, when only
    urban areas were surveyed, or to those governorates of Jordan not
    contained in the survey, would have resulted in estimates of audience size
    and credibility with fewer limitations for those countries. In addition,
    BBG project documentation does not clearly describe what steps have been
    taken to restrict the scope of MBN's surveys to provinces or areas that
    have sufficient map or population data. Probability sampling projected to
    those areas only would result in more reliable estimates of audience size
    and credibility.

    Fifth, we found the lack of verification procedures inhibits assessments
    of BBG's data quality. Data reliability is increased by the use of
    verification and validation procedures, such as checking performance data
    for significant errors by formal evaluation or audit. However, BBG has not
    fully implemented such procedures or formally documented policies and
    procedures governing its research. BBG officials said they have
    implemented some forms of verification, such as a research director
    review, at various stages of performance data analysis. However, it is not
    clear that the BBG always thoroughly verifies performance indicator
    estimates and calculations used for reporting. For example, we identified
    some errors in internal calculation spreadsheets for performance
    estimates, in official external reporting, and in an informal presentation
    on MBN performance. BBG officials have acknowledged that they need to
    develop policies and procedures and implement more rigorous verification
    of performance data. They said that, as a result of our review, they have
    taken some steps to expand their verification procedures and will document
    those and other research procedures in the BBG's research contract and
    manual of operations.

    Conclusions

    MBN was established to broadcast accurate and relevant news and
    information to the Middle East in order to advance freedom, democracy, and
    long-term U.S. interests in the region. MBN's programs are challenged by
    existing television competitors, such as Al Jazeera and Al Arabiya, and
    planned initiatives such as the BBC's planned Arabic-language television
    network; by numerous local radio stations across the region; and by
    limited opportunities for increasing coverage of radio transmission. It
    has attempted to address these challenges through some planning efforts,
    such as by developing proposals to increase its news time for Alhurra and
    expand its number of news bureaus in the Middle East. However, MBN has not
    developed a strategic plan or taken a detailed strategic approach to
    addressing certain issues, including identifying opportunities for
    additional gains that could be made from further integrating Radio Sawa
    and Alhurra or fully utilizing MBN's overseas offices. Developing a
    strategic plan that establishes specific objectives, provides an overall
    shared vision or framework for decision-making, and comprehensively
    addresses competitive challenges could enable MBN to identify efficiencies
    or opportunities to address its competitors more effectively and increase
    congressional confidence in its operations.

    Although MBN has developed a number of financial and administrative
    controls, it could take additional steps to ensure its system of internal
    control is fully implemented. MBN's planned internal control board needs
    to establish protocols to oversee and monitor its internal control
    structure and to ensure timely completion of MBN's financial audits.
    Additionally, MBN should develop an internal control plan, conduct a
    comprehensive risk assessment, and develop an organized training program
    for its staff.

    Further, MBN has developed journalistic standards for its broadcast
    operations and has put in place a number of editorial procedures. However,
    the network has not conducted annual program reviews called for in BBG
    guidance. These reviews are a key mechanism for improving programming and
    ensuring quality control.

    Finally, the BBG has established performance indicators and targets for
    MBN services related to measures of audience size and program credibility.
    For example, in fiscal year 2005, the BBG reported that Radio Sawa and
    Alhurra had each achieved radio and television audiences of about 21.6
    million. However, limitations in the survey methods and documentation used
    for developing these estimates raise questions about the accuracy of MBN's
    performance estimates. In addition, the BBG has not put in place policies
    and procedures for verifying the accuracy of its performance information.
    These weaknesses in methodology and documentation inhibit an accurate
    assessment of whether the Radio Sawa and Alhurra performance targets have
    been met. The accuracy of the MBN estimates could be improved by more
    rigorous reliance on probability sampling, wherever possible, and
    avoidance of projections to areas not contained within the survey. At the
    same time, insistence on proper fieldwork documentation from contractors,
    full disclosure of survey methods and limitations, and greater
    transparency in the development of aggregate audience estimates would
    further enhance confidence in MBN performance estimates.

    Recommendations for Executive Action

    To improve efforts to monitor performance and the efficiency and
    effectiveness of the broadcasting activities of the Middle East
    Broadcasting Networks, Inc., we recommend that the chairman of the BBG do
    the following:

    o Require that MBN develop a long-term strategic plan, which incorporates
    a shared vision for Alhurra and Radio Sawa operations and details specific
    measurable objectives and implementation strategies for the successful
    implementation of the goals in the plan.

    o Require that MBN implement the remaining recommendations from the Grant
    Thornton LLP report and require that its internal control board meet on a
    regular basis to coordinate MBN's single audits and oversee MBN's ongoing
    efforts to use sound internal control procedures.

    o Develop a process for analyzing risk as part of strategic planning that
    identifies approaches to mitigate the potential obstacles to efficiently
    and effectively achieving MBN's operational objectives.

    o Require MBN to develop a comprehensive training program covering both
    internal controls and editorial procedures to meet the continuing needs of
    all employees.

    o Initiate a schedule of annual program reviews for Radio Sawa and Alhurra
    to regularly ensure that the quality of Alhurra's broadcasts conforms to
    applicable standards.

    o Implement program quality performance indicators for MBN's broadcast
    services, consistent with other BBG entities, to assess and compare their
    performance and measure the contribution of these efforts to the goals of
    the overall organization.

    o Require research contractors to improve the methods used in audience
    research to allow for probability sampling and document the sample
    selection so that survey sampling errors can be calculated, where
    possible.

    o Identify and report significant methodological limitations and the
    implications of them for performance indicators, including, where
    applicable, sampling errors, margins of error, or confidence intervals.

    o Develop, document, and report policies and procedures for verification
    and analysis of performance indicator estimates.

    Agency Comments and Our Evaluation

    The Broadcasting Board of Governors (BBG) provided written comments on a
    draft of this report. The BBG's comments, along with our response to
    specific points, are reprinted in appendix V. The BBG also provided
    technical comments, which we incorporated where appropriate.

    In general, the BBG concurred with all of our recommendations and said it
    looked forward to implementing them. The BBG said that MBN has made
    significant progress in the 2 years of its operations in establishing a
    sound journalistic organization with financial and administrative
    controls. However, the BBG raised a number of concerns about the report's
    criticisms of the audience research conducted by the BBG and its
    contractors. Specifically, the BBG said that we did not fully understand
    the difficulty in surveying audiences in Middle Eastern countries and that
    the research practices used by the BBG and its contractors follow industry
    standards for commercial and media research.

    Our report examines the reliability of the BBG's fiscal year 2005
    performance information in order to determine whether or not MBN's
    performance targets have been met. We acknowledge, in our report, that BBG
    has taken positive steps to enhance the validity and reliability of its
    audience survey designs. We also acknowledge that there are challenges to
    conducting audience research in the Middle East, and that there are
    tradeoffs between cost and data reliability when conducting research. In
    conducting our data reliability assessment, as referenced in the draft, we
    largely used the international audience research guidelines published by
    the Conference of International Broadcasters' Audience Research Services
    (CIBAR), which are specified as a source of guidance for BBG research
    contractors. For example, CIBAR standards recommend that the proper care
    be exercised when reporting estimates to ensure that the type of audience
    covered by the estimate is clearly stated, and that, at all stages in the
    calculation and extrapolation process, sources, assumptions, and methods
    are fully documented and available for independent scrutiny. We were
    unable to determine the accuracy of MBN's fiscal year 2005 audience size
    and program credibility estimates due to weaknesses in MBN's methodology
    and documentation. As noted in the report, in several instances the BBG
    and its contractors departed from CIBAR research standards. In particular,
    the BBG did not take certain steps that could have explained and increased
    the reliability of its estimates, such as by fully documenting its
    research methods, measuring the level of uncertainty surrounding its
    estimates, disclosing significant limitations, limiting the scope of its
    projections to areas actually covered by its surveys, and developing and
    consistently implementing policies and procedures for verifying data. For
    these reasons, it is not clear whether the Radio Sawa and Alhurra
    performance targets have been met.

    As agreed with your office, unless you publicly announce the contents of
    this report earlier, we plan no further distribution of it until 30 days
    from the date of this letter. We will then send copies to others who are
    interested and make copies available to others who request them.

    If you or your staff have any questions about this report, please contact
    me on (202) 512-4128. Contact points for our Offices of Congressional
    Relations and Public Affairs may be found on the last page of this report.
    Key contributors are listed in appendix VI.

    Sincerely yours,

    Jess T. Ford Director, International Affairs and Trade

    Appendix I:  Scope and Methodology

    To accomplish our objectives, we reviewed documentation and spoke with
    officials from the Department of State; the Broadcasting Board of
    Governors (BBG) and its grantees; including the Middle East Broadcasting
    Networks; Inc. (MBN); Radio Free Europe/Radio Liberty (RFE/RL); Radio Free
    Asia (RFA); the Voice of America (VOA); and the International Broadcasting
    Bureau. We also spoke with several outside experts, including
    representatives from National Public Radio, the InterMedia research
    organization, Zogby International, and foreign international broadcasters
    such as the British Broadcasting Corporation (BBC) and Deutsche Welle.

    In addition to our audit work in the Washington, D.C., area, we visited
    MBN's offices in Amman, Jordan, and Dubai, United Arab Emirates. We held
    telephone interviews with MBN's current affairs contractor in Beirut,
    Lebanon, and sent questions to the MBN office in Baghdad, Iraq.

    Because our work was focused on reviewing the BBG's MBN
    initiatives--Alhurra and Radio Sawa-we did not include RFE/RL
    Arabic-language media activities in Iraq in the scope of our work.

    Review of MBN Internal Controls

    To address our objective of assessing MBN's internal controls, we used
    criteria contained in MBN's grant agreement, OMB Circulars No. A-110 and
    A-133, GAO published standards on internal controls, and a report on MBN's
    internal controls completed by Grant Thornton LLP accountants and
    management consultants. We also met with other broadcasters, including
    RFE/RL, RFA, and National Public Radio, to obtain an understanding of
    their internal controls and to make comparisons with MBN. We held
    discussions with MBN officials and reviewed documentation provided by them
    to determine whether the organization was complying with appropriate
    criteria. We spoke with PriceWaterhouseCoopers officials responsible for
    the 2003, 2004, and 2005 single audits to obtain their perspectives on the
    status of problems cited in the Grant Thornton LLP report and to learn
    about any new issues arising from their audit work. We did not test MBN's
    controls, since doing so would have been beyond the scope of our review.

    Review of MBN's Editorial Procedures

    To address our objective of assessing the procedures MBN has developed to
    ensure it complies with its journalistic standards, we spoke with other
    broadcasters, including RFE/RL, RFA, VOA, and National Public Radio, to
    compare their editorial standards and procedures with those of MBN. We
    also met with the BBG general counsel, as well as MBN producers, editors,
    and journalists. To better understand the program review process, we
    attended several VOA program reviews. We also held detailed discussions
    and obtained documentation on program review from the BBG officials who
    conducted Radio Sawa's review, the International Broadcasting Bureau
    office of performance review, and the research directors of RFA and
    RFE/RL. We did not test MBN's editorial procedures.

    Review of MBN Audience Survey Methods and Performance Indicator Estimates

    To address our objective of determining the extent to which the BBG met or
    exceeded its fiscal year 2005 performance MBN targets, we reviewed the
    reliability of BBG audience research that was used to generate the
    estimates of credibility and audience size contained in the BBG's Fiscal
    Year 2007 Budget Request, which was the BBG's most up-to-date source of
    officially-reported GPRA performance information for MBN for fiscal year
    2005. We did not review the inputs used to generate performance
    information for fiscal years other than 2005, nor did we review the
    reliability of the data used to develop other MBN performance indictors
    reported in the 2007 Budget Request.

    To conduct our assessment of the reliability of BBG estimates of
    credibility and audience size for fiscal year 2005, we reviewed available
    documentation provided as of May 2005 against applicable standards and
    common professional research practices, including Conference of
    International Broadcasters' Audience Research Services (CIBAR) guidelines
    and the professional standards of the American Association for Public
    Opinion Research. We reviewed relevant surveys, including technical
    reports and printouts of the survey results, as well as spreadsheets used
    to calculate the performance indicator estimates. We also conducted a
    series of interviews and corresponded extensively with the BBG's
    coordinator for performance planning and research, the head of the
    International Broadcasting Bureau office of research, and officials from
    Intermedia to discuss the survey methods and how performance indicator
    estimates were generated. In addition, we requested information from the
    BBG's research subcontractors, including AC Nielsen and D3 Systems, to
    obtain details on the survey sampling plans and related assumptions,
    response rates, and adjustments applied to the data to reflect the effects
    of the survey design. As a result of our review, we determined that MBN's
    reported audience size and credibility estimates are not statistically
    reliable and are rather rough estimates of performance.

    Appendix II:  Radio Sawa and Alhurra Programming

    Radio Sawa currently has seven programming streams,1 with the Iraq
    programming stream airing more news and features than the other streams.
    We reviewed one week of programming and found that the Iraq stream aired
    about 50 hours of news and features a week, while five other streams aired
    about 40 hours a week (see table 5).

    Table 5: Radio Sawa's Weekly News and Features Programming

    Programs            Jordan    Iraq   Persian    Morocco    Egypt    Sudan
                                            Gulf
    Total news          31 hrs 2  34 hrs 31 hrs 2   31 hrs 2   31 hrs 2 31 hrs
    programming time    min       48 min min        min        min      2 min
    Total features      8 hrs     14 hrs 8 hrs 22.5 7 hrs 57.5 8 hrs    7 hrs
    programminga time   22.5 min  47 min min        min        29.5 min 57.5
                                                                           min
    Total time for news 39 hrs 25 49 hrs 39 hrs 25  39 hrs     39 hrs   39 hrs
    and features        min       36 min min                   32 min

    Source: GAO analysis of sample MBN programming schedule.

    Note: The sample programming schedule was provided for the week of October
    19, 2005. The exact breakdown of programming may differ from week to week.

    aFeatures programming includes the following Radio Sawa shows: Ask the
    World Now, Sawa Chat, Call Us Answers, Call Us Questions, Health News,
    What's New, Free Zone, Challenges, To the Point, A Poet's Story, Window on
    the Iraqi Constitution, Law and the Citizen, and Heart of the Matter.

    Alhurra's programming focuses on news and information, including hourly
    news updates, daily hour-long newscasts, and current affairs talk shows
    (see table 6). In addition, Alhurra broadcasts current affairs shows on
    subjects including health and fitness, entertainment, sports, and science
    and technology. Alhurra also airs documentaries on a diverse range of
    topics designed to appeal to a broad audience. In a sample week, Alhurra's
    acquired programming, which mainly includes documentaries, accounted for
    the largest number of hours aired, about 35 percent of the total weekly
    programming hours--or approximately 60 hours in a week. News and news
    updates amounted to 22 percent of weekly programming hours.

    Table 6: Alhurra's Weekly Programming Schedule

    Programming News    Talk  Current News    Economic Acquired     Change in
    time                shows affairs updates news     programminga programmingb
                                                 updates
    Per week    26      29    32 hrs, 10 hrs, 36 min   58 hrs, 20   9 hours
                   hours   hrs,  50 min  21 min           min
                           40
                           min
    Total       166
                   hours
                   and 47
                   minutes

    Source: GAO analysis of sample MBN programming schedule for the week of
    February 19-25, 2006.

    Note: The breakdown of shows may differ from week to week.

    aAcquired programming includes documentaries, historical programs,
    profiles, and travel and adventure programs.

    bChange in programming is a deviation from the normal schedule, including
    broadcasts of National Basketball Association games.

    In contrast, Alhurra-Iraq's talk shows accounted for the largest number of
    hours aired--- about 46 hours, or 27 percent of total programming time
    (see table 7).

    Table 7: Alhurra-Iraq's Weekly Programming Schedule

    Programming News  Talk  Current News    Economic   Acquired    Change in
    time              shows affairs updates news       programming programming
                                               updates
    Per week    34    45    31 hrs, 8 hrs,  36 min     42 hrs      5 hours
                   hours hrs,  20 min  24 min
                         40
                         min
    Total       167
                   hours

    Source: GAO analysis of sample MBN programming schedule for the week of
    February 19-25, 2006.

    Note: The breakdown of shows may differ from week to week.

    Appendix III:  Radio Sawa Program Review

    The BBG initiated the Radio Sawa review because MBN did not have the
    in-house capacity to do so, according to a BBG official. Moreover, both
    the BBG and MBN wanted to ensure that the review was conducted by an
    entity other than MBN to make it more independent.

    To plan and conduct this review, BBG officials reported that they held
    preliminary discussions with the International Broadcasting Bureau's
    office of performance review, hired an outside expert to serve as a
    consultant to the project, and contracted with the InterMedia research
    organization to convene panels of regular listeners in Iraq, Morocco,
    Egypt, and Jordan. According to representatives of InterMedia, they
    recruited a large number of panelists because the BBG wanted more input
    than usual. Panelists listened to and evaluated 6 days worth of
    programming, responded to a questionnaire that evaluated all of Radio
    Sawa's broadcast criteria, answered open-ended questions, provided
    detailed feedback, and made recommendations about the programming.
    Overall, the results of Radio Sawa's monitoring panels were generally
    positive, with an overall program quality score of 2.9 out of 4, which is
    considered "good" by the BBG.1 To follow up on the program review, MBN is
    attempting to implement action points drafted by the BBG after the review,
    such as reducing abrupt transitions between music and news and localizing
    Radio Sawa's streams.

    The BBG told us it placed great emphasis on the audience's response,
    noting that if there had been any problems, they would have heard about
    them from the listeners. The BBG's research contractor concurred that it
    is important to obtain the audience's perspective, since the audience can
    judge programming in the context of the alternatives available to them in
    the local media market. According to a BBG official, it was decided that
    information obtained from the audience monitoring panel would capture the
    issues that would normally be covered by the internal analysts in the
    content and production analyses.

    However, even a detailed, well-thought-out questionnaire does not replace
    the different perspectives internal and external control reviewers bring
    to the review, according to a grantee official. According to an
    International Broadcasting Bureau official, even though the three
    components of a program review-audience panels and internal and external
    reviewers-use nearly the same criteria, they provide different insights
    into the program's quality, direction, and context. According to several
    media experts,2 while audience panelists can evaluate news and information
    from their own perspective, they are less qualified to evaluate a
    service's mission or judge the extent to which a show complies with
    journalistic standards. In particular, the absence of an internal review
    meant that Radio Sawa's programming was not evaluated by people familiar
    with BBG and MBN standards and controls. Moreover, the external control
    listener's evaluation is intended to give the service a sense of what the
    panel's responses would be like if the environment were freer and more
    open, information that a monitoring panel cannot provide.

    The BBG told us that it did not believe an internal evaluation was
    necessary because the English translations of each of the panel reports
    enabled them to examine and assess MBN's programming. The BBG also told us
    that it did not use external control listeners in the review for several
    reasons: (1) Radio Sawa's focus on youth and popular music would make it
    difficult for a typical older control listener to evaluate Sawa's
    programs; (2) the BBG lacked resources to use control listeners for the
    multiple countries in Radio Sawa's review; (3) it would have been hard to
    find impartial listeners; and (4) the four in-country panels ensured
    enough diversity of opinion so that there was less need for a control
    reviewer. However, according to an International Broadcasting Bureau
    official, the VOA is able to successfully evaluate youth shows using its
    regular internal reviewers. Moreover, it may be possible to use younger
    control listeners, or to use the older ones selectively. For example,
    RFE/RL uses a control listener to evaluate the news of Radio Farda. In
    addition, the challenge of finding impartial control listeners is not
    unique to the Middle East, but is potentially present to some degree for
    every language service.

    Appendix IV:  CIBAR Standards

    The Conference of International Broadcasters' Audience Research Services
    (CIBAR) guidelines were developed by an international group of
    broadcasters - including the BBG's International Broadcasting Bureau -- to
    encourage, among other things, the appropriate use of audience and market
    data for decision-making within international broadcasting organizations,
    the establishment and maintenance of standards in international audience
    research, and the efficient and responsible use and application within
    member organizations of all forms of audience feedback. By providing a
    context for international audience research and a clear statement of the
    minimum standards required, the guidelines attempt to address the
    challenges faced by international broadcasters, including the tension
    between the needs of proper audience measurement and survey practice and
    the conditions and expertise in many of the countries where the research
    is carried out.

    The third edition of the guidelines, published in 2001, covers the nature
    of international audience research measures; survey design, sampling, and
    fieldwork; and data management and reporting.1 Specific guidelines
    relevant to this report include the following:

    4.1: Samples and method:

    o The basis for audience measurement should be samples based on the
    principles of random probability.

    o The principles of random selection should be applied at all stages of
    the sampling process, from initial sampling point to selection of
    individuals.

    o Quota samples should be used only in cases where, for reasons of
    practicality, cost, or both, proper random samples cannot be used. When
    quota samples are used, this should be clearly labeled in the reporting
    and documentation of the data.

    o It should be a requirement of all measurement research that the sampling
    methods used and other technical aspects of the survey, including margins
    of error and confidence levels where appropriate, be both fully and
    accurately described in the project documentation, and open to independent
    scrutiny.

    4.2: Survey coverage:

    o Where certain groups are disproportionately sampled and weighting
    techniques are applied at the analysis stage to correct for this, project
    documentation should give full details of the weights applied.

    5.2: Data reporting:

    o The research agency should provide the following information to a
    contracting broadcaster:

    o Background information - client for whom the study was conducted;
    purpose of the study; names of subcontractors and consultants performing
    any substantial part of the work.

    o Intended and achieved sample -- universe covered; statistics used (e.g.
    census data; size, nature and geographical distribution of the sample;
    sampling method and weighting methods used; response rates and possible
    bias due to non-response).

    o Data collection - method of collection; field staff; briefing and field
    quality control; method of recruiting respondents; and fieldwork dates.

    o Presentation of results - relevant factual findings obtained; bases of
    percentages (weighted and unweighted); margins of error; and questionnaire
    and other documents used.

    o Proper care be exercised when reporting such estimates, to ensure that
    the type of audience covered by the estimate is clearly stated.

    o At all stages in the calculation and extrapolation process, it is vital
    that sources, assumptions, and methods be fully documented and available
    for independent scrutiny.

    o All reporting of worldwide and regional audiences should be accompanied
    by a technical appendix giving details of the sources, assumptions, and
    measures used.

    Appendix V:  Comments from the Broadcasting Board of Governors

    The following are GAO's comments on the Broadcasting Board of Governor's
    (BBG) letter dated July 20, 2006.

    GAO Comments

    1.GAO acknowledges in the report that MBN has participated in long-range
    planning in coordination with the BBG's update of its 2008-2012 strategic
    plan, and that MBN has developed a 2006 annual performance plan as part of
    the OMB Program Assessment Rating Tool process. However, contrary to the
    BBG's comments, the fiscal year 2006 performance plan document for MBN
    that we obtained did not contain all of the elements of a stand-alone
    strategic plan. In addition, we were told by the former executive director
    of the BBG, who is currently the president of MBN, that the Radio Sawa
    strategic plan drafted in 2002 by the BBG-which the BBG stated in its
    comments contributed to its considerable long-term planning for MBN--was
    never adopted by the BBG. We were also told by MBN's former president that
    he had never received a copy of the draft Radio Sawa strategic plan.

    2.The BBG stated that MBN's Single Audit evaluates the effectiveness of
    its internal control program. However, while MBN's Single Audits address
    the fair presentation of the organization's financial statements, its
    compliance with certain laws and regulations, and the organization's
    internal control as it relates to financial reporting, MBN's 2005 Single
    Audit as prepared by PriceWaterhouseCoopers does not-contrary to what the
    BBG stated in its comments--offer an opinion on the overall effectiveness
    of MBN's internal control. Thus, the unqualified opinion that MBN received
    on its fiscal year 2005 Single Audit does not provide a broad assessment
    or opinion of MBN's internal control system. For the purposes of our
    report, it was therefore necessary to consider other assessments of MBN's
    financial and administrative controls, including the Grant Thornton
    report.

    3.MBN's grant agreement states that MBN should make every reasonable
    effort to achieve the purpose of the grant in accordance with OMB
    Circulars A-110, A-122, and A-133. Our report did not attempt to make an
    assessment of MBN's compliance with its grant agreement, but rather
    focused on the progress MBN has made in developing its system of internal
    control and the ways in which MBN can continue to strengthen this system.
    To that end, we drew upon the work of Grant Thornton LLP, which assessed
    the status of MBN's controls against relevant OMB Circulars and GAO
    guidance on internal control.

    4.While the Grant Thornton recommendations are not obligatory, they are
    based on best management practices from OMB and GAO. According to
    officials, both the BBG and MBN accepted the results of the report, which
    provides detailed insight into MBN's financial operations, when it was
    completed in May 2004. Adopting the report's remaining recommendations
    would help MBN in its efforts to build a fully mature internal financial
    management operation.

    5.The BBG stated that it believes its standards yield data of sufficient
    reliability to allow it to estimate MBN's audiences and credibility.
    However, the BBG offers no evidence for this. BBG's estimates for MBN are
    based on judgment and not statistics, and its current methods do not and
    cannot estimate the error in its estimates. As noted in the report, in
    several instances the BBG and its contractors departed from CIBAR research
    standards. For example, we analyzed 12 of the 14 country surveys conducted
    by BBG contractors for 2005. In the BBG's fiscal year 2007 budget request,
    the BBG reported the results of each country survey and an overall
    estimate for audience size of 21.5 for Alhurra and 21.6 million for Radio
    Sawa. However, our analysis of the 12 surveys identified a number of
    methodological weaknesses, including BBG's failure to fully document
    research methods, measure the level of uncertainty surrounding its
    estimates, disclose significant limitations, limit the scope of its
    projections to areas actually covered by its surveys, and develop and
    consistently implement policies and procedures for verifying data. These
    limitations were not reported in BBG's audience survey estimates and
    prevent us from concluding that the estimates are accurate and reliable.

    6.We acknowledge in the report that there are challenges to conducting
    audience research in the Middle East, and that there are tradeoffs between
    cost and data reliability when conducting research.

    7.As we noted in the report, it is difficult to conduct probability
    sampling in many locations in the Middle East. But, it is not impossible.
    Organizations, including the United States Census Bureau, have
    collaborated on probability surveys in the Middle East, including in Saudi
    Arabia and Jordan. The major problem of the BBG estimates is their lack of
    transparency, and the lack of an explanation for the methodology behind
    BBG's estimates.

    8.In conducting our data reliability assessment, as referenced in the
    report, we largely used the international audience research guidelines
    published by CIBAR. The BBG participated in drafting these standards,
    which are specified as a source of guidance for BBG research contractors.
    CIBAR standards recommend that proper care be exercised when reporting
    estimates to ensure that the type of audience covered by the estimate is
    clearly stated, and that, at all stages in the calculation and
    extrapolation process, sources, assumptions, and methods are fully
    documented and available for independent scrutiny. As noted in the report,
    in several instances the BBG and its contractors departed from CIBAR
    research standards, leading to weaknesses in MBN's methodology and
    documentation. As a result, we were unable to determine the accuracy of
    MBN's fiscal year 2005 audience size and program credibility estimates.

    9.The BBG stated that it believes the level of methodological detail it
    has provided in public documents is comparable with that typically used by
    other research organizations for studies in the Islamic world; therefore,
    it believes it is consistent with industry practice. However, the Reports
    Consolidation Act of 20001 requires federal agencies, such as the BBG, to
    assess the completeness and reliability of the performance data in their
    performance reports and to discuss any material inadequacies in the
    completeness and reliability of their performance data, as well as actions
    to address the inadequacies. In performance reporting such as the BBG
    fiscal year 2007 budget request, the BBG reported the results of its
    research for MBN as 21.5 million viewers for Alhurra and 21.6 million for
    Radio Sawa. However, the BBG did not sufficiently explain the specific
    methods used for generating estimates for its performance indicators, such
    as the number and names of the countries surveyed (including the sizes of
    its samples), methods of sampling used, sources of the population data,
    and basic procedures used to create the estimates. Moreover, the BBG
    reported only two limitations for MBN performance indicators to date: (1)
    that credibility ratings are highly dependent on volatile political
    factors; and (2) that, depending on political, social, and media
    conditions, measurement of audience size may either be easily attained or
    impossible. Significant limitations not reported are that many of the BBG
    surveys are not based on probability sampling, cover only part of the
    country, have very low response rates, or have high substitution rates.
    Therefore, the BBG does not provide a full level of confidence in the
    credibility of its performance data for MBN. In contrast, the Department
    of Transportation, has a separate compendiums available online that
    provides source and accuracy statements, which provide detail on the
    methods used to collect performance data, sources of variation and bias in
    the data, and methods used to verify and validate the data.

    10.The BBG cannot calculate the sampling error for the BBG estimates of
    audience size and credibility because the probability of selection is not
    known. Although increased sample size will generally decrease sampling
    error, it is impossible to accurately estimate the sampling error of the
    BBG surveys because they are nonprobability surveys. As the BBG has
    stated, its use of a simple random sampling formula when calculating
    sampling errors for its surveys underestimates the sampling error. This
    formula is not appropriate for the sample designs used by MBN.

    11.Although our recommendation asks the BBG to require research
    contractors to improve the methods used in audience research to allow for
    probability sampling and document the sample selection so that survey
    sampling errors can be calculated, where possible, it is the
    responsibility of the BBG to explain and justify the need to conduct
    nonprobability samples.

    Appendix VI:  GAO Contact and Staff Acknowledgments

    GAO Contact

    Jess Ford, (202) 512-4268

    Staff Acknowledgments

    Diana Glod, Melissa Pickworth, Eve Weisberg, Dorian Herring, and Joe
    Carney made key contributions to this report. Chanetta Reed, Jay Smale,
    Karen O'Conor, and Jackie Nowicki provided technical assistance.

    (320376)

    www.gao.gov/cgi-bin/getrpt?GAO-06-762.

    To view the full product, including the scope

    and methodology, click on the link above.

    For more information, contact Jess T. Ford at (202) 512-4128 or
    fordj@gao.gov.

    Highlights of GAO-06-762, a report to the Chairman, Subcommittee on
    National Security, Emerging Threats and International Relations, Committee
    on Government Reform, House of Representatives

    August 2006

    U.S. INTERNATIONAL BROADCASTING

    Management of Middle East Broadcasting Services Could Be Improved

    The Broadcasting Board of Governors' (BBG) broadcasting services, Radio
    Sawa, and the Alhurra satellite television networks-collectively known as
    the Middle East Broadcasting Networks, Inc. (MBN)-currently aim to reach
    Arabic speakers in 19 countries and areas throughout the Middle East.
    Annual spending for current activities amounts to about $78 million.

    GAO reviewed MBN's (1) strategic planning to address competition in the
    Middle Eastern media market, (2) implementation of internal control, (3)
    procedures MBN has developed to ensure compliance with its journalistic
    standards, and (4) performance indicators and whether targets have been
    met.

    What GAO Recommends

    GAO is making several recommendations to the chairman of the BBG to
    improve the efficiency and operations of MBN. These include fully
    implementing strategic planning, internal control, and regular program
    reviews, as well as addressing weaknesses in the conduct and reporting of
    its audience research.

    The BBG generally concurred with our recommendations, although it
    disagreed with some of our analysis.

    MBN faces a number of competitive challenges in carrying out its mission
    of broadcasting in the Middle Eastern media market and has taken some
    steps to address them. However, MBN lacks a comprehensive, long-term
    strategic plan. As MBN emerges from its start-up mode and faces future
    challenges, a long-term strategic plan will be important.

    While MBN has developed financial and administrative controls to manage
    and safeguard its financial resources, it could take additional steps to
    strengthen its system of internal control. For example, the MBN has not
    (1) convened a meeting of its internal control board to formally develop
    its controls and coordinate audits, (2) completed an internal control
    plan, (3) completed a risk assessment to address potential risks to its
    operation, or (4) developed a comprehensive training program for its
    staff.

    MBN has procedures in place to help ensure its programming meets its
    journalistic standards. However, MBN lacks regular editorial training and
    has not fully implemented a comprehensive, regular program review process
    to determine whether its programming complies with those standards or with
    MBN's mission. While the BBG calls for its broadcasters to undergo an
    annual program review, Radio Sawa has only held one such review, and
    Alhurra has not completed one to date.

    The BBG has developed several performance indicators and targets for MBN's
    Radio Sawa and Alhurra services, including measures of audience size and
    program credibility. However, it is not clear whether the Radio Sawa and
    Alhurra performance targets have been met because of weaknesses in MBN's
    survey methodology and documentation. The BBG did not take certain steps
    that could have explained and increased the reliability of its estimates,
    such as by fully documenting its research and estimation methods,
    measuring the level of uncertainty surrounding its estimates, disclosing
    significant limitations, and consistently implementing policies and
    procedures for verifying data.

    Alhurra's Broadcast Center in Springfield, Virginia
    *** End of document. ***