    Vocational Rehabilitation: Improved Information and Practices May
    Enhance State Agency Earnings Outcomes for SSA Beneficiaries
    (23-MAY-07, GAO-07-521).

    State vocational rehabilitation (VR) agencies, under the
    Department of Education (Education), play a crucial role in
    helping individuals with disabilities prepare for and obtain
    employment, including individuals receiving disability benefits
    from the Social Security Administration (SSA). In a prior report
    (GAO-05-865), GAO found that state VR agencies varied in the
    rates of employment achieved for SSA beneficiaries. To help
    understand this variation, this report analyzed SSA and Education
    data and surveyed state agencies to determine the extent to which
    (1) agencies varied in earnings outcomes over time; (2)
    differences in state economic conditions, client demographic
    traits, and agency strategies could account for agency
    performance; and (3) Education's data could be used to identify
    factors that account for differences in individual earnings
    outcomes.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-07-521
        ACCNO:   A69847
      TITLE:     Vocational Rehabilitation: Improved Information and
    Practices May Enhance State Agency Earnings Outcomes for SSA
    Beneficiaries
         DATE:   05/23/2007
      SUBJECT:   Aid for the disabled
    	     Beneficiaries
    	     Data collection
    	     Data integrity
    	     Economic analysis
    	     Employment assistance programs
    	     Employment of the disabled
    	     Federal social security programs
    	     Persons with disabilities
    	     Program evaluation
    	     State-administered programs
    	     Vocational rehabilitation
    	     Disability Insurance Program
    	     Supplemental Security Income Program
    	     Vocational Rehabilitation Program

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Product.                                                 **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-07-521

         * [1]Results in Brief
         * [2]Background

              * [3]Challenges Facing the Social Security Disability Program
              * [4]Description of Education's Vocational Rehabilitation Program

         * [5]State VR Agencies Consistently Showed Very Different Rates o

              * [6]Proportion with Earnings, Earnings Levels, and Departures fr
              * [7]Trends Were Similar over Time and by Agency Type

         * [8]State Economic Conditions and SSA Beneficiary Characteristic

              * [9]Differences in Agency Outcomes Were Largely Due to a State's
              * [10]Demographic Characteristics and the Types of Disabilities of

                   * [11]Demographic Differences
                   * [12]Differences in Types of Disabilities
                   * [13]Proportion of SSA Beneficiaries Served

         * [14]A Few Agency Practices Appeared to Yield Better Earnings Out

              * [15]Agencies with State-Certified Counselors or Strong Relations
              * [16]Agency Expenditures on Training Yield Positive Outcomes
              * [17]Effect of Using In-house Benefits Counselors is Unclear
              * [18]VR Officials in Some Agencies Credited Other Practices with

         * [19]Limitations in Education's Data May Have Hampered Analyses o
         * [20]Conclusions
         * [21]Recommendations for Executive Action
         * [22]Agency Comments and Our Evaluation
         * [23]Section 1: Data Used, Information Sources, and Data Reliabil

              * [24]Education and SSA Beneficiary Data
              * [25]VR Agency Administrative Data
              * [26]State Economic and Demographic Data
              * [27]VR Agency Survey Data

         * [28]Section 2: Study Population and Descriptive Analyses

              * [29]Study Population
              * [30]Computation of Dependent Variables
              * [31]Departures from the Disability Rolls
              * [32]Descriptive Analyses

         * [33]Section 3: Econometric Analyses
         * [34]Section 4: Limitations of our Analyses
         * [35]GAO Comments
         * [36]GAO Comments
         * [37]GAO Contact
         * [38]Acknowledgments
         * [39]GAO's Mission
         * [40]Obtaining Copies of GAO Reports and Testimony

              * [41]Order by Mail or Phone

         * [42]To Report Fraud, Waste, and Abuse in Federal Programs
         * [43]Congressional Relations
         * [44]Public Affairs

    United States Government Accountability Office

    GAO

    May 2007

    Report to Congressional Requesters

    VOCATIONAL REHABILITATION

    Improved Information and Practices May Enhance State Agency Earnings
    Outcomes for SSA Beneficiaries

    GAO-07-521

    Contents

    Letter 1

    Results in Brief 3
    Background 6
    State VR Agencies Consistently Showed Very Different Rates of Success for
    SSA Beneficiaries Who Completed VR Programs 11
    State Economic Conditions and SSA Beneficiary Characteristics Account for
    Much of the Difference in State VR Agency Success Rates 19
    A Few Agency Practices Appeared to Yield Better Earnings Outcomes, while
    the Results of Other Practices Were Inconclusive 24
    Limitations in Education's Data May Have Hampered Analyses of Individual
    Earnings Outcomes 27
    Conclusions 29
    Recommendations for Executive Action 30
    Agency Comments and Our Evaluation 30
    Appendix I Scope and Methodology 33
    Section 1: Data Used, Information Sources, and Data Reliability 33
    Section 2: Study Population and Descriptive Analyses 44
    Section 3: Econometric Analyses 46
    Section 4: Limitations of our Analyses 52
    Appendix II Comments from the Department of Education 55
    Appendix III Comments from the Social Security Administration 64
    Appendix IV GAO Contacts and Staff Acknowledgments 73
    Related GAO Products 74

    Tables

    Table 1: Explanatory Variables from the TRF Subfile 35
    Table 2: Explanatory Variables from Education's RSA-2 Data 36
    Table 3: State Economic and Demographic Explanatory Variables and Their
    Sources 38
    Table 4: Explanatory Variables from the VR Agency Survey Data 40
    Table 5: Dependent Variables Used in the Analyses 45
    Table 6: Coefficients for Multivariate Models Estimating the Effects of
    State and Agency Characteristics on Three VR Outcomes, and the Proportion
    of Variance Explained (R-Squared) by Each Model 50

    Figures

    Figure 1: Distribution of State VR Agencies by Percentage of SSA
    Beneficiaries with Earnings during the Year after VR 12
    Figure 2: Distribution of State VR Agency Average Annual Earnings for SSA
    Beneficiaries with Earnings during the Year after VR 13
    Figure 3: Distribution of State VR Agencies by Percentage of SSA
    Beneficiaries Leaving the Rolls 14
    Figure 4: Range across State VR Agencies of the Percentage of SSA
    Beneficiaries with Earnings during the Year after VR by Year 15
    Figure 5: Range of State VR Agency Average Earnings for SSA Beneficiaries
    by Year 16
    Figure 6: Range across State VR Agencies of the Percentage of SSA
    Beneficiaries with Earnings during the Year after VR by Agency Type 17
    Figure 7: Range of State VR Agency Average Earnings for SSA Beneficiaries
    by Agency Type 18
    Figure 8: Range of State VR Agency Average Rates of SSA Beneficiaries
    Leaving the Rolls by Agency Type 19

    Abbreviations

    CPI-U Consumer Price Index for All Urban Consumers
    CSPD Comprehensive System of Personnel Development
    DI Disability Insurance
    GSP gross state product
    IPE individual plan of employment
    MEF Master Earnings File
    OLS ordinary least squares
    SSA Social Security Administration
    SSI Supplemental Security Income
    TRF Ticket Research File
    VR vocational rehabilitation
    WIA Workforce Investment Act

    This is a work of the U.S. government and is not subject to copyright
    protection in the United States. It may be reproduced and distributed in
    its entirety without further permission from GAO. However, because this
    work may contain copyrighted images or other material, permission from the
    copyright holder may be necessary if you wish to reproduce this material
    separately.

    United States Government Accountability Office

    Washington, DC 20548

    May 23, 2007 May 23, 2007

    The Honorable Charles B. Rangel
    Chairman
    The Honorable Jim McCrery
    Ranking Minority Member
    Committee on Ways and Means
    House of Representatives

    The Honorable Michael R. McNulty
    Chairman
    The Honorable Sam Johnson
    Ranking Minority Member
    Subcommittee on Social Security
    Committee on Ways and Means
    House of Representatives

    The Honorable Sander M. Levin
    House of Representatives

    State vocational rehabilitation (VR) agencies, under the auspices of the
    Department of Education (Education), play a crucial role in helping
    individuals with disabilities prepare for and obtain employment. In fiscal
    year 2005, state VR agencies received $2.6 billion to provide people with
    disabilities a variety of supports such as job counseling and placement,
    diagnosis and treatment of impairments, vocational training, and
    postsecondary education. The VR program serves about 1.2 million people
    each year, and over a quarter of those who complete VR are beneficiaries
    of the Disability Insurance (DI) or Supplemental Security Income (SSI)
    programs administered by the Social Security Administration (SSA). This
    proportion has increased steadily since 2002. As our society ages, the
    number of SSA disability beneficiaries is expected to grow, along with the
    cost of providing SSA disability benefits, and it will be increasingly
    important to manage this growth by optimizing the ability of VR programs
    to help and encourage SSA beneficiaries to participate in the workforce.
    State vocational rehabilitation (VR) agencies, under the auspices of the
    Department of Education (Education), play a crucial role in helping
    individuals with disabilities prepare for and obtain employment. In fiscal
    year 2005, state VR agencies received $2.6 billion to provide people with
    disabilities a variety of supports such as job counseling and placement,
    diagnosis and treatment of impairments, vocational training, and
    postsecondary education. The VR program serves about 1.2 million people
    each year, and over a quarter of those who complete VR are beneficiaries
    of the Disability Insurance (DI) or Supplemental Security Income (SSI)
    programs administered by the Social Security Administration (SSA). This
    proportion has increased steadily since 2002. As our society ages, the
    number of SSA disability beneficiaries is expected to grow, along with the
    cost of providing SSA disability benefits, and it will be increasingly
    important to manage this growth by optimizing the ability of VR programs
    to help and encourage SSA beneficiaries to participate in the workforce.

    In 2005, GAO reported that state VR agencies varied substantially in terms
    of the employment rates they achieved for their clients,^1 particularly
    for SSA beneficiaries who, according to research, attain lower employment
    and earnings outcomes than other VR clients.^2 Depending on the state
    agency, as many as 68 percent and as few as 9 percent of SSA beneficiaries
    exited VR with employment. In addition, GAO found that Education's
    management of the VR program was lacking in several respects and
    recommended that Education revise its performance measures to account for
    economic differences between states, make better use of incentives for
    state VR agencies to meet performance goals, and create a means for
    disseminating best practices among state VR agencies. Education agreed
    with these recommendations but has yet to implement them.

    As a follow-up to our 2005 report, you asked us to determine what may
    account for the wide variations in state VR agency outcomes with respect
    to SSA beneficiaries. Therefore, we examined the extent to which (1)
    differences in VR agency outcomes for SSA beneficiaries continued over
    several years and across different outcome measures, (2) differences in VR
    agency outcomes were explained by state economies and demographic traits
    of the clientele served, (3) differences in VR agency outcomes were
    explained by specific policies and strategies of the VR agencies, and (4)
    Education's data allowed for an analysis of factors that account for
    differences in individual-level (as opposed to agency-level) outcomes.

    To perform our work, we used several data sources: (1) a newly available
    longitudinal dataset that includes administrative data from Education and
    SSA on SSA beneficiaries who completed the VR program between 2001 and
    2003, ^3 (2) original survey data collected by GAO from 78 of the 80 state
    VR agencies, (3) data from Education on yearly spending information by
    service category for each VR agency, and (4) data from the Census Bureau,
    Bureau of Labor Statistics, and other data sources regarding state
    demographic and economic characteristics. We conducted reliability
    assessments of these data and found them to be sufficiently reliable for
    our analyses.

    ^1GAO, Vocational Rehabilitation: Better Measures and Monitoring Could
    Improve the Performance of the VR Program, [45]GAO-05-865 (Washington,
    D.C.: September 2005).

    ^2David C. Stapleton and William A. Erickson, "Characteristics or
    Incentives: Why Do Employment Outcomes for the SSA Beneficiary Clients of
    VR Agencies Differ, on Average, from Those of Other Clients?"
    (Rehabilitation Research and Training Center for Economic Research on
    Employment Policy for Persons with Disabilities, Cornell University,
    Ithaca, New York, Oct. 2004).

    ^3The longitudinal dataset from SSA and Education contains information on
    beneficiaries for a longer time horizon (i.e., 1998 through 2004).
    However, we focused on the cohorts completing VR between 2001 and 2003
    because, at the time of our analysis, data were not available on earnings
    after 2004. Further, we excluded earlier cohort years due to limitations
    associated with collecting survey data from VR agencies prior to 2000. See
    appendix I for more information on our data.

    We took several steps to analyze these data. To answer our questions, we
    analyzed outcomes by state agency using three different earnings outcomes:
    (1) the percentage of beneficiaries with earnings during the year after
    VR, (2) the average beneficiary's annual earnings level during the year
    after VR, and (3) the percentage of beneficiaries that left the disability
    rolls by the close of 2005.^4 For objective one, we conducted descriptive
    statistical analyses of the data. For objectives two, three, and four, we
    conducted econometric analyses that controlled for a variety of
    explanatory factors.^5 We also identified and interviewed academic and
    agency experts in an effort to determine what variables to include in our
    models. As is the case with most statistical analyses, our work was
    limited by certain factors, such as the unavailability of certain
    information and the inability to control for unobservable characteristics
    and those that are not quantifiable. Our results only describe earnings
    outcomes of SSA beneficiaries included in our study and cannot be
    generalized beyond that population. We conducted our review from December
    2005 through April 2007 in accordance with generally accepted government
    auditing standards. See appendix I for a more detailed description of our
    scope and methods.

    Results in Brief

    When we analyzed state agency outcomes for SSA beneficiaries who completed
    VR between 2001 and 2003, we found that differences in agency outcomes
    continued over several years and across several outcome measures--i.e.,
    rates of beneficiaries with earnings, earnings levels, and departures from
    the disability rolls. The proportion of beneficiaries with earnings during
    the year after their completion of the VR program ranged from as little as
    0 percent in one state agency to as high as 75 percent in another.
    Similarly, average annual earnings levels among those SSA beneficiaries
    with earnings varied across state agencies from $1,500 to nearly $17,000
    in the year following VR. Additionally, the proportion of SSA
    beneficiaries who left the disability rolls varied greatly among agencies,
    with departure rates ranging anywhere from 0 to 20 percent.

    ^4For the purposes of our study, leaving the rolls is defined as the
    termination of cash disability benefits due to work.

    ^5We conducted our analyses using multivariate regression analysis.

    After controlling for certain economic, demographic, and agency factors,
    we found that state economic conditions and the characteristics of
    agencies' clientele accounted for much of the differences in average
    earnings outcomes across state agencies. Specifically, state unemployment
    rates and state per capita income levels accounted for a substantial
    portion--as much as one-third--of the differences between state agencies'
    VR outcomes for SSA beneficiaries. For example, significantly fewer SSA
    beneficiaries had earnings during the year after VR in those states with
    higher unemployment rates and lower per capita incomes. Despite the
    significant effect that state economies have on state agency outcomes,
    Education currently does not consider such factors when analyzing state
    agency outcomes and assessing their performance. Variations in the
    demographic profile of SSA client populations also accounted for some of
    the differences in earnings outcomes among agencies. For example, state VR
    agencies serving a higher percentage of women beneficiaries had
    significantly fewer SSA clients with earnings during the year after VR.

    We also found, after controlling for the same factors, that a few agency
    practices helped explain differences in state agency outcomes for SSA
    beneficiaries--and some were associated with positive outcomes. For
    example, agencies with a higher proportion of state-certified VR
    counselors--a certification now mandated by Education--had more SSA
    beneficiaries exiting the VR program with earnings. Further, agencies with
    closer ties to the business community also achieved higher average annual
    earnings for SSA beneficiaries and higher rates of departures from the
    disability rolls. Currently, Education promotes ties to the business
    community through an employer network. Our findings also show that
    agencies that received a greater degree of support and cooperation from
    other public programs or that spent a greater proportion of their service
    expenditures on training of VR clients had higher average annual earnings
    for SSA beneficiaries completing VR.

    We were unable to account for differences in individual beneficiary
    outcomes, which might further explain differences in state agency
    outcomes, in part because of limitations in Education's data. Our
    statistical models were able to explain a greater percentage of the
    differences in earnings outcomes when we analyzed state agency earnings
    outcomes compared to individual earnings outcomes (i.e., as much as 77
    percent compared to 8 percent). With so little variation explained by our
    analyses of individual-level outcomes, we decided not to report our
    individual-level analyses. Education's data lack information that we
    believe is critical to assessing earnings outcomes, and this may have
    hindered our ability to explain the variation in individual earnings
    outcomes. Specifically, although Education collects extensive client-level
    data, it does not systematically collect data that research has linked to
    work outcomes, such as detailed information on the severity of the
    client's disability--data that some state agencies independently collect
    for program purposes. Knowing the severity of a disability can indicate
    whether a person is physically or mentally limited in his or her ability
    to perform work, a fact that may influence the person's earnings outcomes.
    Further, other key data are self-reported and may not be verified by state
    agencies.

    We are recommending that Education consider the implications of the
    results of our analyses in its management of the VR program. Specifically,
    Education should further promote certain agency practices that we found
    show an effect on state agency outcomes and reassess the client-level data
    it collects through its state agencies. We also continue to believe that,
    as we recommended in our 2005 report, Education should consider economic
    factors, such as unemployment rates, when evaluating state agency
    performance.

    We received written comments on a draft of this report from Education and
    SSA. While Education generally agreed with the substance of our
    recommendations, it disagreed on when economic conditions and state
    demographics should be considered in assessing performance. Instead of
    using this information to help set performance measures, the department
    said that it takes these factors into account when it monitors agency
    performance results and believes that its approach is more effective. We
    continue to believe that incorporating this contextual information in
    assessing performance measures is essential to provide the state agencies
    with a more accurate picture of their relative performance. Although
    Education stated that it was open to our recommendation on improving data
    quality, it suggested that validating self-reported information would be a
    potential burden to state agencies and suggested other approaches, such as
    conducting periodic studies. Our recommendation that Education explore
    cost-effective ways to validate self-reported data was based on the
    experience of some VR agencies that have obtained data successfully from
    official sources and not relied solely on self-reported information.

    SSA stated that our report has methodological flaws that introduced
    aggregation bias and false correlations, and suggested that we should have
    focused on individual-level analysis or reported the results of both
    individual and aggregate-level analysis. We used aggregated data--a widely
    used means of analysis--because our primary objective was to understand
    better the wide variation in outcomes for state VR agencies that serve SSA
    beneficiaries rather than the outcomes for individuals. We used
    appropriate statistical techniques to ensure against bias and false
    correlations. Both Education and SSA provided additional comments, which
    we have addressed or incorporated, as appropriate. Education's and SSA's
    comments are reprinted in appendixes II and III respectively, along with
    our detailed responses.

    Background

    Challenges Facing the Social Security Disability Program

    In 2005, the Social Security Administration provided income support to
    more than 10 million working age people with disabilities. This income
    support is provided in the form of monthly cash benefits under two
    programs administered by the Social Security Administration--the
    Disability Insurance program and the Supplemental Security Income program.
    Some individuals, known as concurrent beneficiaries, qualify for both
    programs. The federal government's cost of providing these benefits was
    almost $101 billion in 2005.

    Over the last decade, the number of disability beneficiaries has
    increased, as has the cost of both the SSI and DI programs. This growth,
    in part, prompted GAO in 2003 to designate modernizing federal disability
    programs as a high-risk area--one that requires attention and
    transformation to ensure that programs function in the most economical,
    efficient, and effective manner possible. GAO's work found that federal
    disability programs were not well positioned to provide meaningful and
    timely support for Americans with disabilities. For example, despite
    advances in technology and the growing expectations that people with
    disabilities can and want to work, SSA's disability programs remain
    grounded in an outmoded approach that equates disability with incapacity
    to work. In 1999, GAO testified that even relatively small improvements in
    return-to-work outcomes offer the potential for significant savings in
    program outlays. GAO estimated that if an additional 1 percent of working
    age SSA disability beneficiaries were to leave the disability rolls as a
    result of returning to work, lifetime cash benefits would be reduced by an
    estimated $3 billion.

    SSA has had a long-standing relationship with Education's VR program,
    whereby SSA may refer beneficiaries to the VR program for assistance in
    achieving employment and economic independence.^6 As part of this
    relationship, SSA reimburses VR state agencies for the cost of providing
    services to beneficiaries who meet SSA's criteria for successful
    rehabilitation (i.e., earnings at the substantial gainful activity level
    for a continuous 9-month period). To further motivate beneficiaries to
    seek VR assistance and expand the network of VR providers, Congress
    enacted legislation in 1999 that created SSA's Ticket to Work (Ticket)
    Program.^7 Under the Ticket program, beneficiaries receive a document,
    known as a ticket, which can be used to obtain VR and employment services
    from an approved provider such as a state VR agency. Thus far, only a
    small fraction of SSA beneficiaries have used the Ticket program to obtain
    VR services. Administered by SSA, this program was intended to (1)
    increase the number of beneficiaries participating in VR by removing
    disincentives to work, and (2) expand the availability of VR services to
    include private VR providers. To date private VR providers have not
    participated heavily in the Ticket program, with over 90 percent of SSA
    beneficiaries participating in the Ticket program still receiving services
    from state VR agencies.

    Despite programs such as Ticket, SSA beneficiaries who wish to participate
    in the workforce still face multiple challenges. As we have previously
    reported, some SSA beneficiaries will not be able to return to work
    because of the severity of their disability.^8 But those who do return to
    work may face other obstacles that potentially deter or prevent them from
    leaving the disability rolls, such as (1) the need for continued health
    care, (2) lack of access to assistive technologies that could enhance
    their work potential, and (3) transportation difficulties.

    ^6Individuals may be referred from SSA to state VR agencies by state
    disability determination services (DDS), which are funded by SSA to render
    the initial decision on whether an individual qualifies for DI or SSI
    benefits, and thus are in a good position to consider whether the
    individual is an appropriate candidate for VR.

    ^7Ticket to Work and Work Incentives Improvement Act of 1999, Pub. L. No.
    106-170 (1999). The Ticket to Work Program was implemented in three
    phases, beginning in February 2002. Under the Ticket program, VR agencies
    and other providers can opt for one of two different reimbursement
    methods, one based on a successful outcome, the other based on
    successfully reaching milestones. State VR agencies can also continue to
    be reimbursed under the traditional cost reimbursement program if the
    beneficiary does not utilize his or her ticket to obtain services.

    ^8 GAO, Social Security: Disability Programs Lag in Promoting Return to
    Work, [46]GAO/HEHS-97-46 (Washington, D.C.: March 1997).

    Description of Education's Vocational Rehabilitation Program

    The Vocational Rehabilitation Program is the primary federal government
    program helping individuals with disabilities to prepare for and obtain
    employment. Authorized by Title I of the Rehabilitation Act of 1973, the
    VR program is administered by the Rehabilitation Services Administration,
    a division of the Department of Education, in partnership with the states.
    The Rehabilitation Act contains the general provisions states should
    follow in providing VR services. Each state and territory designates a
    single VR agency to administer the VR program--except where state law
    authorizes a separate agency to administer VR services for blind
    individuals. Twenty-four states have two separate agencies, one that
    exclusively serves blind and visually impaired individuals (known as blind
    agencies) and another that serves individuals who are not blind or
    visually impaired (known as general agencies). Twenty-six states, the
    District of Columbia, and five territories have a single combined agency
    that serves both blind and visually impaired individuals and individuals
    with other types of impairments (known as combined agencies). In total,
    there are 80 state VR agencies. ^9

    Although Education provides the majority of the funding for state VR
    agencies, state agencies have significant latitude in the administration
    of VR programs. Within the framework of legal requirements, state agencies
    have adopted different policies and approaches to achieve earnings
    outcomes for their clients. For example, although all state VR agencies
    are required to have their VR counselors meet Comprehensive System of
    Personnel Development (CSPD) standards, states have the ability to define
    the CSPD certification standard for their VR counselors. Specifically,
    under the CSPD states can establish certification standards for VR
    counselors based on the degree standards of the highest licensing,
    certification, or registration requirement in the state, or based on the
    degree standards of the national certification. For example, if an agency
    bases its certification standard on the national standard, VR counselors
    are required to have a master's degree in vocational counseling or another
    closely related field, hold a certificate indicating they meet the
    national requirement, or take certain graduate-level courses. Regardless
    of the individual state's definition of the certification standard,
    research has shown that VR agencies are concerned about meeting their
    needs for state-certified counselors because many experienced VR
    counselors may retire in the coming years, and a limited supply of
    qualified VR counselors are entering the labor market.^10

    9In this report, when we refer to state VR agencies, we are including
    agencies in the states and territories.

    VR agencies also vary in their locations within state government and their
    operations. Some are housed in state departments of labor or education,
    while others are free-standing agencies or commissions. Similarly, while
    all VR agencies are partners in the state workforce investment system, as
    mandated in the Workforce Investment Act (WIA) of 1998, VRs vary in the
    degree to which they coordinate with other programs participating in this
    system. ^11 For example, some VRs have staff colocated at WIA one-stop
    career centers, while others do not.

    By law, each of the 80 VR agencies is required to submit specific
    information to Education regarding individuals that apply for, and are
    eligible to receive, VR services. Some of the required information
    includes (1) the types and costs of services the individuals received; (2)
    demographic factors, such as impairment type, gender, age, race, and
    ethnicity; and (3) income from work at the time of application to the VR
    program. Education also collects additional information such as (1) the
    weekly earnings and hours worked by employed individuals, (2) public
    support received,^12 (3) whether individuals sustained employment for at
    least 90 days after receiving services,^13 and (4) summary information on
    agency expenditures in a number of categories from each state VR agency.

    Education also monitors the performance of state VR agencies, and since
    2000, Education has used two standards for evaluating their performance.
    One assesses the agencies' performance in assisting individuals in
    obtaining, maintaining, or regaining high-quality employment. The second
    assesses the agencies' performance in ensuring that individuals from
    minority backgrounds have equal access to VR services. Education also
    publishes performance indicators that establish what constitutes minimum
    compliance with these performance standards. Six performance indicators
    were published for the employment standard, and one was published for the
    minority service standard. To have passing performance, state VR agencies
    must meet or exceed performance targets in four of the six categories for
    the first standard, and meet or exceed the performance target for the
    second standard.

    ^10Tsze Chan, Recruiting and Retaining Professional Staff in State VR
    Agencies: Some Preliminary Findings from the RSA Evaluation Study, a
    special report prepared at the request of the Department of Education,
    October 2003.

    ^11WIA requires states and localities to bring together a number of
    federally funded employment and training services into a single
    system--the one-stop system. Funded through different federal agencies,
    these programs are to provide services through a statewide network of
    one-stop career centers to adults, dislocated workers, and youth.

    ^12Public support refers to cash payments made by federal, state, or local
    governments for any reason, including an individual's disability, age,
    economic, retirement, and survivor status. This excludes any noncash
    support payments such as Medicaid, Medicare, food stamps, and rental
    subsidies.

    ^13Education tracks individuals in terms of seven types of case closures,
    which can be collapsed into four categories for individuals who (1) exited
    without employment, during the application phase; (2) exited without
    employment, with limited services; (3) exited without employment, after
    receiving services under an employment plan; and (4) exited with at least
    90 days of employment, after receiving services under an employment plan.

    In 2005, GAO reported that Education could improve performance of this
    decentralized program through better performance measures and
    monitoring.^14 Specifically, we recommended that Education account for
    additional factors such as the economies and demographics of the states'
    populations in its performance measures, or its performance targets, for
    individual state VR agencies to address these issues. We also noted that
    whatever system of performance measures Education chooses to use, without
    consequences or incentives to meet performance standards, state VR
    agencies will have little reason to achieve the targets Education has set
    for them. We recommended that Education consider developing new
    consequences for failure to meet required performance targets and
    incentives for encouraging good performance. While Education agreed with
    our recommendations, it is currently considering them as part of the
    development of its VR strategic performance plan, and has not adopted them
    to date.

    Earlier this year, GAO reported on national-level earnings outcomes for
    SSA beneficiaries who completed VR from 2000 to 2003.^15 Among other
    findings, this report estimated that as a result of work, some DI and
    concurrent beneficiaries saw a reduction in their DI benefits--for an
    overall annual average benefit reduction of $26.6 million in the year
    after completing VR compared to the year before VR. Further, we reported
    that 10 percent of SSA beneficiaries who exited VR in 2000 or 2001 were
    able to leave the disability rolls at some point. However, almost one
    quarter of those who left had returned by 2005 for at least 1 month.

    ^14 [47]GAO-05-865 , 39.

    ^15GAO, Vocational Rehabilitation: Earnings Increased for Many SSA
    Beneficiaries after Completing VR Services, but Few Earned Enough to Leave
    SSA's Disability Rolls, [48]GAO-07-332 (Washington, D.C.: March 2007).

    State VR Agencies Consistently Showed Very Different Rates of Success for SSA
    Beneficiaries Who Completed VR Programs

    Before controlling for factors that might explain differences in outcomes
    among state VR agencies, our analysis of state agency outcomes over a
    3-year period showed very different rates of success for SSA
    beneficiaries. This was the case in terms of the proportion of
    beneficiaries with earnings, earnings levels, and departures from the
    disability rolls. The wide range in average earnings outcomes among
    agencies was generally consistent from 2001 through 2003 and within each
    of the three types of agencies--referred to as blind, general, and
    combined agencies.

    Proportion with Earnings, Earnings Levels, and Departures from the Disability
    Rolls for SSA Beneficiaries Differed Substantially among State Agencies

    Between 2001 and 2003, VR agencies varied widely in terms of outcomes for
    SSA beneficiaries who completed their VR programs. While the agency
    average for beneficiary earnings was 50 percent, the proportion of
    beneficiaries with earnings during the year following VR varied
    substantially among agencies: from 0 to 75 percent. (See fig. 1.)

    Figure 1: Distribution of State VR Agencies by Percentage of SSA
    Beneficiaries with Earnings during the Year after VR

    Note: n = 234, average = 50 percent. The 234 observations result from 78
    VR agencies providing data for 3 years (2001 through 2003).

    Similarly, while the agency average for annual earnings levels for SSA
    beneficiaries who had earnings was $8,140, such earnings ranged by agency
    from about $1,500 to nearly $17,000. (See fig. 2.)

    Figure 2: Distribution of State VR Agency Average Annual Earnings for SSA
    Beneficiaries with Earnings during the Year after VR

    Note: n = 232, average = $8,140. The number in figure 2 differs from that
    in figure 1 because two agencies did not have any beneficiaries with
    reported earnings in fiscal year 2002. All earnings are in 2004 dollars.

    Agencies also differed in the proportion of SSA beneficiaries who had left
    the disability rolls by 2005, with departure rates ranging anywhere from 0
    to 20 percent. The average departure rate was 7 percent. (See fig. 3.)

    Figure 3: Distribution of State VR Agencies by Percentage of SSA
    Beneficiaries Leaving the Rolls

    Note: n = 234, average = 7 percent.

    Trends Were Similar over Time and by Agency Type

    In general, the range of earnings outcomes across agencies was similar
    over the 3 years we examined. While the average percentage of SSA
    beneficiaries with earnings during the year after VR declined slightly
    over this period from 53 percent in 2001 to 48 percent in 2003, the spread
    in the percentage of beneficiaries with earnings remained widely dispersed
    across agencies for all 3 years, as shown in figure 4.

    Figure 4: Range across State VR Agencies of the Percentage of SSA
    Beneficiaries with Earnings during the Year after VR by Year

    Likewise, the range of average earnings among agencies was similar for all
    3 years, as shown in figure 5.^16

    16See appendix I for an explanation of why we did not compare agencies'
    rates of SSA beneficiaries leaving the rolls over this period.

    Figure 5: Range of State VR Agency Average Earnings for SSA Beneficiaries
    by Year

    Note: Two agencies did not have any beneficiaries with reported earnings
    in fiscal year 2002. All earnings are in 2004 dollars.

    There were also wide differences in performance within the three types of
    agencies that serve different types of clientele--known as blind, general,
    and combined agencies. Specifically, among blind agencies, the percentage
    of SSA beneficiaries with earnings during the year after VR ranged from 23
    to 67 percent, with an average of 46 percent. Among general agencies, the
    percentage of SSA beneficiaries with earnings after VR varied from 37 to
    74 percent, with an average of 55 percent, and for combined agencies the
    percentage varied from 0 to 75 percent, with an average of 49 percent.
    (See fig. 6.)

    Figure 6: Range across State VR Agencies of the Percentage of SSA
    Beneficiaries with Earnings during the Year after VR by Agency Type

    Average annual SSA client earnings among blind agencies varied the
    most--from $4,582 to $16,805, with an average of $10,699 per year. SSA
    client earnings among the combined agencies varied anywhere from $1,528 to
    $10,889, with an average of $7,088 per year. General agencies showed the
    least variation in earnings among their SSA clients--from $4,654 to
    $9,424--but the lowest average ($6,867). (See fig. 7.)

    Figure 7: Range of State VR Agency Average Earnings for SSA Beneficiaries
    by Agency Type

    Note: Two combined agencies did not have any beneficiaries with reported
    earnings in fiscal year 2002. All earnings are in 2004 dollars.

    Finally, for rates of departure from the SSA disability rolls by 2005,
    blind agencies ranged from 0 to 16 percent, with an average of 6.7
    percent; general agencies varied from 4 to 15 percent, with an average of
    7.5 percent; and combined agencies varied from 0 to 20 percent, with an
    average of 7 percent. (See fig. 8.)

    Figure 8: Range of State VR Agency Average Rates of SSA Beneficiaries
    Leaving the Rolls by Agency Type

    State Economic Conditions and SSA Beneficiary Characteristics Account for Much
    of the Difference in State VR Agency Success Rates

    After controlling for a range of factors, we found that much of the
    differences in state VR agency success rates could be explained by state
    economic climates and the characteristics of the SSA beneficiary
    populations at the VR agencies. Specifically, among a range of possible
    factors we considered, the economic conditions of the state appeared to
    explain up to one-third of the differences between state agency outcomes
    for SSA beneficiaries.^17 Additionally, differences in the characteristics
    of the clientele accounted for some of the variation in performance among
    VR agencies.

    ^17All findings discussed in this section are statistically significant at
    the 0.05 level, unless otherwise noted.

    Differences in Agency Outcomes Were Largely Due to a State's Economic Conditions

    When we controlled for a variety of factors using multivariate analysis,
    we found that state economic conditions accounted for a substantial
    portion of the differences in VR outcomes across state agencies. Not
    surprisingly, we found that fewer SSA beneficiaries had earnings during
    the year after completing VR in states with high unemployment rates after
    controlling for other factors. Moreover, our analysis showed that for each
    1 percent increase in the unemployment rate, the percentage of SSA
    beneficiaries who had earnings during the year after completing VR
    decreased by over 2 percent.^18 Across agencies, unemployment rates ranged
    from 2.3 to 12.3 percent between 2001 and 2003, with an average of 4.7
    percent.

    We also found that after controlling for other factors, VR agencies in
    states with lower per capita incomes saw fewer SSA beneficiaries who had
    earnings, lower earnings levels, and fewer departures from the disability
    rolls in the year after VR. Across states, per capita incomes ranged from
    approximately $4,400 to $46,000 dollars, with an average of approximately
    $28,000. Together, state unemployment rates and per capita incomes
    explained over one-third of the differences between states agencies in the
    proportion of SSA beneficiaries that had earnings during the year after VR
    and the proportion that left the rolls. ^19

    Agency officials commented that difficult economic environments result in
    lower earnings outcomes because a state's economy has a direct impact on
    an agency's ability to find employment for individuals. Our findings are
    also consistent with past research that has found labor market conditions
    to be among the most influential determinants of agency performance.^20
    Education, however, does not currently consider state economic conditions
    when evaluating agency performance.^21 Although Education agreed with our
    prior recommendation to consider economic and demographic characteristics
    when evaluating agency performance, Education is currently considering it
    as part of the development of its VR strategic performance plan and has
    not yet adopted this recommendation.

    ^18Unless otherwise indicated, the effects being discussed in this and the
    next section are marginal effects (i.e., the effect of a 1 unit change in
    the explanatory variable on the dependent variable, holding other factors
    constant). See appendix I, section 3, for more details on our econometric
    analyses.

    ^19We also found that in states with larger populations, fewer SSA
    beneficiaries (1) had earnings during the year after completing VR and (2)
    left the disability rolls. A study conducted by RTI International noted
    that states with small populations reported having improved access to
    other agencies and better collaboration with state leaders due to closer
    work and personal relationships.

    ^20Michael D. Tashjian, et al., Study of Variables Related to State
    Vocational Rehabilitation Agency Performance Revised Draft Final Report, a
    special report prepared at the request of the Department of Education,
    October 2004.

    Demographic Characteristics and the Types of Disabilities of Clientele Also
    Accounted for Some of the Disparities in State Agency Performance

    After controlling for a variety of factors, certain characteristics of the
    clientele served by state agencies accounted for some of the state agency
    differences in earnings outcomes for SSA beneficiaries. Among the factors
    we examined the influence of were: demographic characteristics, types of
    disabilities, and the proportion of SSA beneficiaries served by each state
    agency.^22

      Demographic Differences

    Several clientele characteristics influenced state agency earnings
    outcomes.^23 In particular, after controlling for other factors, state
    agencies that served a higher proportion of women beneficiaries had fewer
    beneficiaries with earnings during the year after completing VR. According
    to our analysis, a 10 percent increase in the percentage of women served
    by a VR agency resulted in a 5 percent decrease in the percentage of SSA
    beneficiaries with earnings. Research shows that for the population of
    low-income adults with disabilities, women were found to have lower
    employment rates than men.^24

    Further, we found that after controlling for other factors, state agencies
    serving a larger percentage of SSA beneficiaries between 46 and 55 years
    old when they applied for the VR program saw fewer SSA beneficiaries leave
    the disability rolls.^25 For every 10 percent increase in the percentage
    of beneficiaries in this age group, the percentage of SSA beneficiaries
    leaving the rolls decreased by approximately 1 percent.

    ^21Although state economic and demographic conditions are not factored
    into performance measures and targets, Education considers these factors
    through its monitoring of state agencies. In addition, the statutory
    funding formula for VR agencies allocates relatively more funds to poorer
    states based on per capita income to help offset a lack of resources.

    ^22See appendix I for a detailed list of the factors we controlled for.

    ^23These clientele characteristics appeared to influence one or more of
    the earnings outcomes measured, but not necessarily all three.

    ^24David Wittenburg and Melissa Favreault, "Safety Net or Tangled Web? An
    Overview of Programs and Services for Adults with Disabilities"
    (Occasional Paper No. 68, the Urban Institute, Washington, D.C.: 2003).

      Differences in Types of Disabilities

    When we considered the influence of various types of medical impairments
    on earnings outcomes, we found that some state agency outcomes were
    related to the proportion of SSA beneficiaries who had mental or visual
    impairments. Average earnings and departures from the disability rolls for
    SSA beneficiaries were lower in agencies that served a larger percentage
    of individuals with mental impairments, after controlling for other
    factors. Specifically, our analysis indicated that a 10 percent increase
    in the proportion of the beneficiary population with a mental impairment
    resulted in a decrease of almost 1 percent in the proportion of SSA
    beneficiaries who left the rolls. Some SSA beneficiaries may not leave the
    disability rolls because, as research has shown, they fear a loss of their
    public benefits or health coverage.^26 This is particularly true for
    individuals with mental impairments.

    Agencies with a higher proportion of blind or visually impaired
    beneficiaries had fewer departures from the disability rolls after
    controlling for other factors. We found that an increase of 10 percent in
    the proportion of individuals with a visual impairment resulted in a
    decrease of 0.5 percent of beneficiaries leaving the rolls. Some SSA
    beneficiaries with visual impairments are classified as legally blind. As
    such, they are subject to a higher earnings threshold, in comparison to
    those that are not legally blind, before their benefits are reduced or
    ceased. Our analysis also showed that holding other factors equal, blind
    agencies--those serving only clientele with visual impairments--had fewer
    SSA beneficiaries with earnings during the year after completing VR than
    agencies that served a lower proportion of beneficiaries with visual
    impairments.^27

    25While other variables were significant at the 0.05 level, this variable
    was significant at the 0.10 level. See appendix I for more information.

    ^26Timothy Tremblay et al., "Effect of Benefits Counseling Services on
    Employment Outcomes for People with Psychiatric Disabilities," Psychiatric
    Services, vol. 57, no. 6 (2006).

    ^27Specifically, holding other factors constant, agencies known as
    combined or general agencies had more SSA beneficiaries with earnings
    during the year after VR than agencies known as blind agencies.

      Proportion of SSA Beneficiaries Served

    Differences in the proportion of SSA beneficiaries served by an agency
    also affected earnings outcomes for SSA beneficiaries. Specifically,
    agencies with a greater proportion of SSA beneficiaries had more
    beneficiaries with earnings during the year after VR, but saw lower
    earnings levels for their SSA beneficiaries, holding other factors
    constant. VR state agency officials and experts with whom we consulted
    were unable to provide an explanation for this result.^28

    We also found that after controlling for other factors, agencies with a
    higher proportion of SSA beneficiaries who were DI beneficiaries had lower
    average annual earnings among SSA beneficiaries and a lower percentage of
    beneficiaries leaving the rolls. The earnings result might be explained by
    differences in the work incentive rules between the two programs.
    Specifically, the work incentive rules are more favorable for SSI
    beneficiaries who want to increase their earnings while not incurring a
    net income penalty.^29 The lower rates of departures from the rolls among
    agencies with a greater proportion of DI beneficiaries might be due to the
    limited time frames of our study and the fact that DI beneficiaries are
    allowed to work for a longer period of time before their benefits are
    ceased.^30

    28In its comments on our report, Education suggested that VR agencies with
    high proportions of SSA beneficiaries may also have high levels of
    collaboration with other agencies because the long-term supports that may
    be required to live in the community necessitate cooperation with other
    public programs. The department noted that this may account for our
    findings because benefit eligibility may be necessary to receive certain
    supports from outside agencies.

    ^29See [49]GAO-07-332 for a more detailed description of the differing DI
    and SSI benefit structures.

    ^30See appendix I, section 4, for a detailed description of why, given the
    time frames of our study, the rates of departures from the rolls might be
    lower for DI beneficiaries.

    A Few Agency Practices Appeared to Yield Better Earnings Outcomes, while the
    Results of Other Practices Were Inconclusive

    When we analyzed outcomes at the agency level, a few agency practices
    appeared to yield some positive results, albeit in different ways.
    Specifically, after controlling for other factors, we found that state
    agencies with a higher proportion of state-certified VR counselors, or
    stronger relationships with businesses or other public agencies appeared
    to have better earnings outcomes. Further, agencies that devoted a greater
    proportion of their service expenditures to training of VR clients had
    higher average annual earnings for SSA beneficiaries completing VR,
    holding other factors equal. On the other hand, our multivariate analyses
    suggest that agencies using in-house benefits counselors saw fewer
    beneficiaries with earnings following VR, but these results may not be
    conclusive because the benefits counseling program has changed
    considerably since the time period of our study.

    Agencies with State-Certified Counselors or Strong Relationships with Businesses
    or Other Public Agencies Appeared to Have Better Earnings Outcomes

    State VR agencies that reported employing a higher percentage of
    counselors meeting the state certification standards had higher rates of
    beneficiaries with earnings among those beneficiaries who completed VR
    between 2001 and 2003, holding other factors constant. On average, 62
    percent of counselors at an agency met the states' certification
    requirements, but the range was from 0 to 100 percent. According to our
    analysis, for every 10 percent increase in the percentage of counselors
    meeting state requirements, the percentage of SSA beneficiaries with
    earnings during the year after VR increased by 0.5 percent. This appeared
    to be consistent with research indicating that more highly qualified VR
    counselors are more likely to achieve successful earnings outcomes.^31
    While the certification requirements vary by state, agency officials
    reported that counselors with master's degrees in vocational
    rehabilitation are more likely to be versed in the history of the VR
    program and the disability rights movement and are likely to be more
    attuned to the needs of their clients than those without specialized
    degrees.

    VR agencies that had stronger relationships with the business community
    had higher average earnings among SSA beneficiaries during the year after
    completing VR and higher rates of departures from the disability rolls,
    holding other factors equal. These were agencies that reported interacting
    with the business community more frequently by sponsoring job fairs,
    hosting breakfasts, attending business network meetings, meeting with
    local businesses, meeting with local chambers of commerce, and interacting
    with civic clubs. To support these practices, Education has helped
    establish the Vocational Rehabilitation Employer Business and Development
    Network, which aims to connect the business community to qualified workers
    with disabilities through the efforts of staff located at each of the VR
    agencies who specialize in business networking.^32 VR agency officials
    with whom we spoke said that through interaction with the business
    community, they could dispel myths about the employability of people with
    disabilities, and they could tailor services for their clients to the
    specific needs of different businesses.

    ^31Edna Mora Szymanski, "Relationship of Level of Rehabilitation Counselor
    Education to Rehabilitation Client Outcome in the Wisconsin Division of
    Vocational Rehabilitation," Rehabilitation Counseling Bulletin, vol. 35,
    no. 1 (1991).

    In addition to business outreach, our multivariate analysis indicated that
    agencies that reported receiving a greater degree of support and
    cooperation from more than one public program--such as from state social
    services, mental health, and education departments--also showed higher
    average earnings among SSA beneficiaries. One VR agency official commented
    that people with disabilities need multiple supports and services and
    therefore are more effectively served through partnerships between
    government programs.^33 Another VR official said that coordination with
    other programs facilitated the provision of a complete package of
    employment-related services. For example, VR might provide employment
    training to an individual, while the department of labor might provide
    transportation services to get the person to work. Although many agencies
    said they were successful in coordinating with other programs, some
    reported difficulties. For example, they cited barriers to coordinating
    with WIA one-stops such as inability to share credit for successful
    earnings outcomes, staff that are not trained to serve people with
    disabilities, and inaccessible equipment, particularly for those with
    visual or hearing impairments.

    ^32The Council of State Administrators of Vocational Rehabilitation is
    also developing a national VR-business network whose aim is to coordinate
    VR outreach efforts to businesses. In addition to these national-level
    efforts, state VR agencies also participate in state-level business
    networks. In Utah, for example, the VR agency participates in the Utah
    Business Employment Team, which serves as a business-to-business network
    for recognizing and promoting best practices in hiring, retaining, and
    marketing to people with disabilities .

    ^33Past GAO reports have highlighted the need for greater coordination
    among agencies delivering services to people with disabilities. See, for
    example, GAO, Federal Disability Assistance: Wide Array of Programs Needs
    to Be Examined in Light of 21st Century Challenges, [50]GAO-05-626
    (Washington, D.C.: June 2, 2005).

    Agency Expenditures on Training Yield Positive Outcomes

    Additionally, agencies with a greater proportion of their service
    expenditures spent on training of VR clients--including postsecondary
    education, job readiness and augmentative skills, and vocational and
    occupational training--had higher average annual earnings for SSA
    beneficiaries completing VR, holding other factors equal.^34 The average
    percentage of service expenditures devoted to training of VR clients was
    47 percent, but this ranged from 3 to 84 percent across agencies. Research
    has shown that the receipt of certain types of training services, such as
    business and vocational training, leads to positive earnings outcomes.^35

    Effect of Using In-house Benefits Counselors is Unclear

    Our analysis suggests that after controlling for other factors, agencies
    with in-house benefits counselors--counselors who advise VR clients on the
    impact of employment on their benefits--had lower rates of SSA
    beneficiaries with earnings during the year after completing VR than
    agencies without them. Over the years we studied, only 14 percent of state
    agencies reported using in-house benefits counselors. However, this was a
    period of transition for the benefits counseling program. There was wide
    variation in how this service was provided, and clients in states that did
    not have on-site benefits counselors may have received benefits counseling
    from outside the agency. According to one researcher, the benefits
    counseling program has become more standardized since that period. In
    fact, other empirical research shows that benefits counselors have had a
    positive effect on earnings.^36

    34The expenditures considered for this calculation do not include
    assessment, counseling, guidance, and placement services provided directly
    by VR personnel since these services are generally provided to all VR
    clients. The total expenditures in this calculation include those optional
    services that are provided to clients based on their specific needs.

    ^35Becky J. Hayward and Holly Schmidt Davis, Longitudinal Study of the
    Vocational Rehabilitation Services Program Final Report 2: VR Services and
    Outcomes, a special report prepared at the request of the Department of
    Education, 2003.

    ^36Other research finds a positive effect of benefits counseling on
    earnings among beneficiaries with psychiatric disabilities and clients in
    the state of Vermont. See Timothy Tremblay, et al., "Effect of Benefits
    Counseling Services on Employment Outcomes for People with Psychiatric
    Disabilities," Psychiatric Services, vol. 57, no. 6 (2006).

    VR Officials in Some Agencies Credited Other Practices with Yielding Results

    Some agency officials credited certain other practices with yielding
    positive results, but we were not able to corroborate their ideas with our
    statistical approach. For example, VR agency officials cited the following
    practices as being beneficial: (1) collaborative initiatives between the
    state VR agency and other state agencies aimed to help specific client
    populations, such as individuals with mental impairments or developmental
    disabilities; (2) unique applications of performance measures, such as
    measuring performance at the team level rather than the individual
    counselor level; and (3) improved use of computer information systems,
    such as real-time access to the status of individual employment targets.
    Although we were able to examine many state practices with our survey
    data, there were not enough agencies employing these practices for us to
    determine whether these practices led to improved earnings outcomes for
    SSA beneficiaries among state VR agencies.

    Limitations in Education's Data May Have Hampered Analyses of Individual
    Earnings Outcomes

    Although we were able to explain a large amount of the differences in
    earnings outcomes among state agencies, we could only explain a small
    amount of the differences in earnings outcomes among individual SSA
    beneficiaries. Specifically, while our models accounted for between 66 and
    77 percent of the variation in agency-level earnings outcomes, our models
    using the individual-level data had low explanatory power, accounting for
    only 8 percent of variation in earnings levels across individuals and
    rarely producing reliable predictions for achieving earnings or leaving
    the rolls. With so little variation explained in individual-level
    outcomes, we could not be confident that our individual-level analyses
    were sufficiently reliable to support conclusions. As a result, we chose
    not to report on these analyses. Other researchers told us they have
    experienced similar difficulties using Education's client database to
    account for individual differences in earnings outcomes among VR clients.

    Education's data lack information that we believe is critical to assessing
    earnings outcomes, and not having this information may have hindered our
    ability to explain differences in individual earnings outcomes.^37
    Specifically, Education does not collect certain information on VR clients
    that research has linked to work outcomes, such as detailed information on
    the severity of the disability and historical earnings data. Research
    indicates that both of these factors are, or could be, important to
    determining employment success for people with disabilities.^38 With
    regard to obtaining information on the severity of the client's
    disability, knowing the severity of the disability can indicate the extent
    to which a person is physically or mentally limited in the ability to
    perform work, a fact that may influence the person's earnings outcomes.
    While Education's client data include information indicating whether a
    disability is significant--which is defined by the Rehabilitation Act--the
    data do not include more detailed information on the severity of the
    disability, such as the number and extent of functional limitations.^39
    Additionally, Education does not collect information on a client's
    historical earnings, which may provide a broader understanding of the
    client's work experience and likelihood to return to work. States may be
    able to obtain earnings data from other official sources, such as other
    state and federal agencies.

    ^37We cannot say with certainty that our results were detrimentally
    affected by these limitations because we do not have data without these
    limitations with which to test our hypotheses.

    Another limitation with Education's data is the collection of
    self-reported information from the client that may not be validated by the
    VR agency. For example, one agency official said that clients are asked to
    report their earnings at the time of application--information that
    Education is legally required to collect--and that these data may not be
    accurate. Reliable information on a client's earnings at the time of
    application to VR is essential for evaluating the impact of the VR program
    on earnings. However, some clients may misreport their earnings. One
    researcher reported, for example, that VR clients sometimes report net as
    opposed to gross earnings. Instead of relying on self-reported
    information, agencies may be able to obtain or validate this information
    from official sources. Specifically, some state VR agencies have
    agreements with other state and federal agencies to obtain earnings data
    on their clients. For example, agency officials from one state told us
    that they match their data against earnings data from the Department of
    Labor, while another agency relies on data from their state's Employment
    Development Department. However, in some cases state agencies are required
    to pay for these data.

    ^38Mitchell P. LaPlante and H. Stephen Kaye, "The Employment and Health
    Status of Californians with Disabilities" (Institute of Health and Aging,
    University of California, San Francisco, June 2005).

    ^39In evaluating the significance of a disability, some state VR agencies
    already collect such information.

    Conclusions

    The federal-state vocational rehabilitation program is still the primary
    avenue for someone with a disability to prepare for and obtain employment.
    Given the growing size of the disability rolls and the potential savings
    associated with moving beneficiaries into the workforce, it is important
    to make the nation's VR program as effective as possible to help people
    with disabilities participate in the workforce.

    Our findings indicate that it will be difficult to maximize the
    effectiveness of the VR program with assessments of state agency
    performance that do not account for important factors, such as the
    economic health of the state. Such comparisons will be misleading. Without
    credible indicators, VR agencies do not have an accurate picture of their
    relative performance, and Education may continue its reluctance to use
    sanctions or incentives to encourage compliance. Our findings underscore
    the recommendation that we made in 2005 that Education consider economic
    factors in assessing the performance of state vocational rehabilitation
    agencies.

    Moreover, our study points to deficiencies in Education's data that may
    hinder more conclusive analyses of individual-level earnings outcomes.
    Without data on the severity of a client's disability or information on
    historical earnings, VR programs may not be able to conduct valuable
    analysis to explain differences in individual outcomes. With the growing
    emphasis on the role of VR in helping people with disabilities enter the
    workforce, the need for such analyses--and data that can be used to
    conduct them--is likely to increase.

    Despite the deficiencies in Education's data, our findings show that
    certain agency practices may improve VR success across the country and
    give weight to current efforts by Education to promote such practices. The
    fact that agencies with stronger ties to the business community have
    achieved higher earnings among their SSA beneficiaries suggests the
    importance of such practices, such as Education's initiative to promote
    business networks. Our findings also demonstrate the value of having VR
    counselors meet state certification standards and having agencies
    collaborate with more than one supportive public agency to help their
    clients. Our study also suggests that other practices, such as state
    agencies devoting more resources to targeted training services for VR
    clients, may have positive benefits.

    Recommendations for Executive Action

    To improve the effectiveness of Education's program evaluation efforts and
    ultimately the management of vocational rehabilitation programs, we
    recommend that the Secretary of Education:

               1. Further promote agency practices that show promise for helping
               more SSA disability beneficiaries participate in the workforce.
               Such a strategy should seek to increase:

                            o the percentage of VR staff who meet state standards
                            and certifications established under the CSPD,
                            o partnership or involvement with area business
                            communities, and
                            o collaboration with other agencies that provide
                            complementary services.

               2. Reassess Education's collection of VR client data through
               consultation with outside experts in vocational rehabilitation and
               the state agencies. In particular, it should:

                            o consider the importance of data elements that are
                            self-reported by the client and explore
                            cost-effective approaches for verifying these data,
                            and
                            o consider collecting additional data that may be
                            related to work outcomes, such as more detailed data
                            on the severity of the client's disability and past
                            earnings history, collaborating whenever possible
                            with other state and federal agencies to collect this
                            information.

               3. In a 2005 report, we recommended that Education revise its
               performance measures or adjust performance targets for individual
               state VR agencies to account for additional factors. These include
               the economic conditions of states, as well as the demographics of
               a state's population. We continue to believe that Education should
               adopt this recommendation, especially in light of our findings on
               the impact of state unemployment rates, per capita incomes, and
               demographic factors on earnings outcomes.

    Agency Comments and Our Evaluation

    We received written comments on a draft of this report from Education,
    which oversees the VR program, and SSA, from which we received data that
    were used to evaluate its Ticket to Work program. Education commended our
    use of multiple data sources and said that it opens up new analytical
    possibilities in evaluating how VR programs serve SSA beneficiaries,
    including identifying low-performing and high-performing VR programs.
    However, Education also questioned whether the statistical relationships
    we found can be applied to how it administers a state-operated formula
    grant program. We continue to believe our findings have important
    implications for improving what data are collected and how VR services are
    delivered. While Education generally agreed with the substance of our
    recommendations, it disagreed on when economic conditions and state
    demographics should be considered in assessing agency performance. Instead
    of using this information to help set performance measures, the department
    said that it takes these factors into account when it monitors agency
    performance results and believes that its approach is effective. We
    believe that incorporating this contextual information into assessing
    performance is essential to provide the state agencies with a more
    accurate picture of their relative performance. Although Education stated
    that it was open to our recommendation on improving data quality, it
    suggested that validating self-reported information would be a potential
    burden to state agencies and suggested other approaches, such as
    conducting periodic studies. Our recommendation that Education explore
    cost-effective ways to validate self-reported data was based on the
    experience of some VR agencies that have obtained data successfully from
    official sources and not relied solely on self-reported information. We
    made additional technical changes as appropriate based on Education's
    comments. See appendix II for a full reprinting of Education's comments
    and our detailed responses.

    SSA stated that our report has methodological flaws that introduced
    aggregation bias and false correlations, and suggested that we should have
    focused on individual-level analysis or reported the results of both
    individual and aggregate-level analyses. We used aggregated data--a widely
    used means of analysis--because our primary objective was to understand
    better the wide variation in outcomes for state VR agencies that serve SSA
    beneficiaries rather than the outcomes for individuals. Further, we used
    appropriate statistical techniques to ensure the lack of bias due to
    clustering of individual cases within agencies (see app. I for a more
    detailed discussion). Because we used aggregated data, we did not attempt
    to infer the effects of individual behavior or individual outcomes.
    Additionally, SSA had concerns about the implications of our analysis of
    state economic factors on agency-level outcomes. Our findings related to
    the influence of state economic characteristics were highly statistically
    significant as well as corroborated by previous research, and we believe
    these results have important implications for VR agency performance
    measures. SSA provided additional comments, which we addressed or
    incorporated, as appropriate. See appendix III for a full reprinting of
    SSA's comments as well as our detailed responses.

    Copies of this report are being sent to the Secretary of Education, the
    Commissioner of SSA, appropriate congressional committees, and other
    interested parties. The report is also available at no charge on GAO's Web
    site at http://www.gao.gov. If you have any questions about this report,
    please contact me at (202) 512-7215. Other major contributors to this
    report are listed in appendix IV.

    Denise M. Fantone
    Acting Director, Education, Workforce, and Income Security Issues

    Appendix I: Scope and Methodology

    To understand the variation in state agency outcomes for Social Security
    Administration (SSA) disability beneficiaries completing the vocational
    rehabilitation (VR) program, we conducted two sets of analyses. First, we
    used descriptive analyses to compare agency performance with three
    measures of earnings outcomes from 2001 to 2003. Second, using agency and
    survey data, we conducted econometric analyses of the three measures of
    earnings outcomes to determine how state and agency characteristics
    related to state agency performance.

    We developed our analyses in consultation with GAO methodologists, an
    expert consultant, and officials from SSA and the Department of Education
    (Education).^1 To choose the appropriate variables for our analyses, we
    reviewed pertinent literature and consulted with agency officials and
    academic experts.

    This appendix is organized in four sections: Section 1 describes the data
    that were used in our analyses and our efforts to ensure data reliability.
    Section 2 describes the study population, how the dependent variables used
    in the analyses were constructed, and the descriptive analyses of those
    variables. Section 3 describes the econometric analyses. Section 4
    explains the limitations of our analyses.

    Section 1: Data Used, Information Sources, and Data Reliability

    This section describes each of the datasets we analyzed, the variables
    from each dataset that were used in our analyses, and the steps that were
    taken to assess the reliability of each dataset.

    To conduct our analyses, we used several data sources: (1) a newly
    available longitudinal dataset that includes information from several SSA
    and Education administrative databases on all SSA disability beneficiaries
    who completed the VR program from 2001 through 2003; (2) data from
    Education on yearly spending information by service category for each
    state VR agency; (3) data from the Census Bureau, the Bureau of Labor
    Statistics, and other data sources regarding state demographic and
    economic characteristics; and (4) original survey data collected by GAO
    from state VR agencies. To perform our analyses, we used variables from
    each of the above datasets by merging, by agency and year, each of the
    datasets into one large data file.

    ^1We are especially grateful to Professor Herbert Smith--Professor of
    Sociology and Director, Population Studies Center at the University of
    Pennsylvania, and an expert in the area of statistical analysis--who
    provided valuable advice on our statistical methods.

    Education and SSA Beneficiary Data

    We obtained a newly available longitudinal dataset--a subfile of SSA's
    Ticket Research File (TRF)--which contains information from several SSA
    and Education administrative databases on all SSA disability beneficiaries
    who completed the federal-state VR program between 1998 and 2004.^2 SSA
    merged this dataset with its Master Earnings File (MEF), which contains
    information on each beneficiary's annual earnings from 1990 through 2004.
    The combined data provide information about each beneficiary's disability
    benefits, earnings, and VR participation.^3 See section 2 of this appendix
    for a description of how these data were used to create our dependent
    variables on earnings outcomes.

    We were interested in how earnings outcomes were affected by differences
    across agencies, including differences in characteristics of the
    individuals served by the different agencies. Table 1 shows information
    from the TRF subfile on characteristics of our study population that we
    included among our explanatory variables.^4

    2In 2003, SSA contracted with Mathematica Policy Research to conduct a
    full evaluation of the Ticket to Work Program. As part of this evaluation,
    Mathematica constructed the Ticket Research File, a compilation of
    longitudinal data from SSA. An extract of the TRF was merged with
    vocational rehabilitation data from the Department of Education's RSA-911
    database by an SSA official.

    ^3Education's data on VR closures were available from 1998 to 2004. Data
    from SSA's TRF database were available from 1994 to 2004, with MEF
    earnings data available from 1990 to 2004. Social Security's MEF data are
    annual earnings based on Internal Revenue Service W-2 tax filings. At the
    time we obtained this dataset from SSA, earnings data for 2005 were not
    available.

    ^4For the purposes of this study, the term "explanatory variable" is used
    to describe a variable that is used to predict the value of another
    variable, and the term "dependent variable" is used to describe a variable
    whose values are predicted by the explanatory variable.

    Table 1: Explanatory Variables from the TRF Subfile

    State agency demographic characteristics
    Percentage of beneficiaries between the ages of 18 and 25
    Percentage of beneficiaries between the ages of 26 and 35
    Percentage of beneficiaries between the ages of 36 and 45
    Percentage of beneficiaries between the ages of 46 and 55
    Percentage of beneficiaries between the ages of 56 and 64
    Percentage of female beneficiaries
    Percentage of white beneficiaries
    Percentage of African-American beneficiaries
    Percentage of Native-American beneficiaries
    Percentage of Asian and Pacific Islander beneficiaries
    Percentage of Hispanic beneficiaries
    Percentage of multiracial beneficiaries
    Stage agency medical characteristics
    Percentage of beneficiaries who are blind or have visual impairments
    Percentage of beneficiaries with sensory impairments
    Percentage of beneficiaries with physical impairments
    Percentage of beneficiaries with mental impairments
    State agency program participation
    Percentage of beneficiaries receiving Supplemental Security Income
    Percentage of beneficiaries receiving Disability Insurance
    Percentage of concurrent beneficiaries (receiving both SSI and DI)
    Proportion of SSA beneficiaries served by an agency^a

    Source: SSA and Education data.

    a To construct this variable, additional information was obtained from
    Education on the total number of clients completing the VR program.

    To determine the reliability of the TRF subfile, we

               o reviewed SSA and Education documentation regarding the planning
               for and construction of the TRF subfile,
               o conducted our own electronic data testing to assess the accuracy
               and completeness of the data used in our analyses, and
               o reviewed prior GAO reports and consulted with GAO staff
               knowledgeable about these datasets.

    On the basis of these steps, we determined that despite the limitations
    outlined in section 4, the data that were critical to our analyses were
    sufficiently reliable for our use.

    VR Agency Administrative Data

    To determine whether differences in agency size and expenditure patterns
    affected earnings outcomes, we obtained information on state VR agency
    expenditures for the years 2000 through 2002 from the RSA-2 data, an
    administrative dataset compiled by Education. The RSA-2 data contain
    aggregated agency expenditures for each of the 80 state VR agencies as
    reported in various categories, such as administration and different types
    of services. Table 2 shows the variables that were derived from the RSA-2
    data.

    Table 2: Explanatory Variables from Education's RSA-2 Data

    Agency structure
    Type of agency: (1) general, (2) blind, and (3) combined agencies
    Number of people receiving services (proxy for size)
    Total expenditures on services (proxy for size)
    Agency expenditures
    Percentage of all service expenditures spent on assessment
    Percentage of all service expenditures spent on diagnosis/treatment
    Percentage of all service expenditures spent on training services for VR
    clients
    Percentage of all service expenditures spent on maintenance
    Percentage of all service expenditures spent on transportation
    Percentage of all service expenditures spent on personal assistance
    services
    Percentage of all service expenditures spent on placement services
    Percentage of all service expenditures spent on post employment services
    Percentage of all service expenditures spent on other services
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on assessment^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on diagnosis/treatment^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on training services for VR
    clients^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on maintenance^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on transportation^a

    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on personal assistance
    services^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on placement^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on post employment services^a
    Percentage of total service expenditures (not including assessment,
    counseling, guidance, and placement) spent on other services^a
    Percentage of total expenditures spent on administration
    Percentage of total expenditures spent on services provided directly by VR
    personnel
    Percentage of total expenditures spent on purchased services
    Percentage of total expenditures spent on services purchased from public
    vendors
    Percentage of total expenditures spent on services purchased from private
    vendors
    Percentage of total expenditures spent on services to individuals with
    disabilities
    Percentage of total expenditures spent on services to groups with
    disabilities

    Source: Education data.

    ^aThese total expenditures include those optional services that are
    provided to clients based on their specific needs. They do not include
    assessment, counseling, guidance, and placement services provided directly
    by VR personnel since these services are generally provided to all VR
    clients.

    To determine the reliability of the RSA-2 data, we

               o reviewed relevant agency documentation and interviewed agency
               officials who were knowledgeable about the data, and
               o conducted our own electronic data testing to assess the accuracy
               and completeness of the data used in our analyses.

    On the basis of these steps, we determined that the data that were
    critical to our analyses were sufficiently reliable for our use.

    State Economic and Demographic Data

    We were interested in how differences in state characteristics affected
    earnings outcomes of SSA beneficiaries completing VR at different VR
    agencies. The state characteristics we considered included economic
    conditions (unemployment rates, per capita income, and gross state product
    growth rates), population characteristics (including size, density, and
    percentage living in rural areas and on Disability Insurance), and
    availability of the Medicaid Buy-in program. Data on state characteristics
    were downloaded from several sources, including federal agencies and
    research institutes. The research institutes from which we obtained data
    included Cornell University Institute for Policy Research and Mathematica
    Policy Research, Inc., both authorities in social science research. Table
    3 summarizes the state data that were collected and the sources for those
    data.

    Table 3: State Economic and Demographic Explanatory Variables and Their
    Sources

    Variable                        Data source
    Annual state unemployment rates Department of Labor, Bureau of Labor
                                       Statistics
    Gross state product (GSP)       Department of Commerce, Bureau of Economic
    growth rate                     Analysis
    Annual per capita income        Department of Commerce, Bureau of Economic
                                       Analysis
    Annual population               Department of Commerce, Census Bureau
    Population density              Department of Commerce, Census Bureau
    Percentage of rural population  Department of Commerce, Census Bureau
    Medicaid Buy-In participation   Cornell University Institute of Policy
                                       Research and Mathematica Policy Research,
                                       Inc. (primary sources)
    Ticket to Work program          Mathematica Policy Research, Inc.
    implementation

    Source: Various data sources listed in table.

    For each of these data sources we reviewed documentation related to the
    agency's or research organization's efforts to ensure the accuracy and
    integrity of their data. On the basis of these reviews, we concluded that
    the data were sufficiently reliable for the purposes of our review.

    VR Agency Survey Data

    We were also interested in how differences in the VR agencies themselves
    affected earnings outcomes. To obtain information about the policies,
    practices, and environment of each state VR agency, we conducted a
    detailed survey of all state agencies. The survey was intended to collect
    information that may be relevant to explaining earnings outcomes of SSA
    beneficiaries who exited the VR program between federal fiscal years 2001
    through 2003. Specifically, we collected information on the structure of
    the VR program, staffing and turnover rates, performance measures, service
    portfolios, and the extent of integration with outside partners such as
    other state and federal agencies and the business community.^5 In
    developing our survey, we identified relevant areas of inquiry by
    conducting a review of the literature on state VR agency performance and
    consulting with state agency officials and outside researchers.

    ^5Electronic copies of the survey are available upon request.

    For the final survey, we sent e-mail notifications asking state agency
    officials to complete either a Web-based version of the survey (which was
    accessible to those with visual impairments) or a Microsoft Word version
    of the survey by August 4, 2006. We closed the survey on August 22, 2006.
    We obtained survey responses from 78 of the 80 state VR agencies, for a
    response rate of 98 percent.

    Because this was not a sample survey, it has no sampling errors. However,
    the practical difficulties of conducting any survey may introduce errors,
    commonly referred to as nonsampling errors. For example, difficulties in
    interpreting a particular question or sources of information available to
    respondents can introduce unwanted variability into the survey results. We
    took steps in developing the questionnaire, collecting the data, and
    analyzing them to minimize such nonsampling error. For example, we
    pretested the content and format of our survey with officials from 17
    state agencies to determine if it was understandable and the information
    was feasible to collect, and we refined our survey as appropriate. When
    the data were analyzed, an independent analyst checked all computer
    programs. Since the data were collected with a Web-based and Word format
    survey, respondents entered their answers directly into the electronic
    questionnaire, thereby eliminating the need to key data into a database,
    minimizing another potential source of error.

    The variables that we analyzed from the survey data are presented in table
    4. These included the structure of the agency (stand-alone agencies,
    umbrella agencies with and without autonomy over staff and finances, and
    others), agency staffing, agency management, indicators of the existence
    of performance targets and incentives, specialized caseloads, case
    management systems and system components, and integration with outside
    partners and the business community. Since we had data on each of the
    earnings outcomes and most of the state and agency characteristics for
    each of the 3 years, we included in our analysis an indicator for year.

    Table 4: Explanatory Variables from the VR Agency Survey Data

    Agency structure
    Number of clients per VR counselor^a
    Number of counselors employed (proxy for agency size)^a
    Indicates whether the director had authority over developmental disability
    services
    Indicates whether the director had authority over independent living
    services
    Indicates whether the director had authority over disability determination
    services
    Indicates whether the director had authority over other programs or
    services
    Percentage of counselors who left VR agency (turnover)^a
    Percentage of counselors meeting comprehensive system of personnel
    development (CSPD) standards^a
    Percentage of senior managers who left VR agency (turnover)^a
    Length of time director has held his/her position (director tenure)^a
    Length of time director has been with the VR agency (director
    experience)^a
    Length of time the director has held his/her position as a percent of
    their time at the agency^a
    Indicates whether the agency operated under an order of selection
    Indicates whether the program had a wait list
    Length of wait list
    Indicates whether the program had a wait list and, if so, its length
    Performance targets/incentives
    Scale indicating number of reported specific and numerical targets
    including SSA reimbursements, individual plans for employment (IPE)
    initiated, client referrals, contacts with businesses, client
    satisfaction, and other client employment outcomes by year
    Indicates whether counselors had performance expectations with numerical
    targets based on successful VR employment outcomes (status 26 closures)
    Nature of performance expectations
    Indicates whether counselors had numerical targets in their performance
    expectations
    Average number of status 26 case closures required for satisfactory
    performance^a
    Indicates whether there were performance expectations that contained
    numerical targets for SSA reimbursements
    Indicates whether there were performance expectations that contained
    numerical targets for the number of IPEs initiated
    Indicates whether there were performance expectations that contained
    numerical targets for the number of client referrals
    Indicates whether there were performance expectations that contained
    numerical targets for the number of contacts made with businesses for job
    development
    Indicates whether there were performance expectations that contained
    numerical targets for client satisfaction rates
    Indicates whether there were performance expectations that contained
    numerical targets for any other outcomes
    Indicates whether there were monetary performance incentives to VR
    counselors
    Indicates how frequently a VR program reported on agencywide performance
    Specialized caseloads
    Indicates whether there were in-house benefits counselors
    Number of benefits counselors^a
    Indicates whether there were job development specialists
    Number of job development specialists^a
    Scale measuring the number of types of specialized caseloads covered,
    including transitioning high school students, mental health, developmental
    disabilities, traumatic brain/spinal cord injuries, hearing impairments,
    visual impairments (not counted for blind-serving agencies), or other
    groups
    Percentage of counselors with specialized caseloads serving transitioning
    high school students^a
    Percentage of counselors with specialized caseloads serving clients with
    mental health issues^a
    Percentage of counselors with specialized caseloads serving clients with
    developmental disabilities^a
    Percentage of counselors with specialized caseloads serving clients with
    traumatic brain/spinal cord injuries^a
    Percentage of counselors with specialized caseloads serving clients with
    hearing impairments^a
    Percentage of counselors with specialized caseloads serving clients with
    visual impairments^a
    Percentage of counselors with specialized caseloads serving any other
    group of clients^a
    Case management system
    Scale indicating the sophistication of the case management system
    including the ability of the case management system to collect Education
    data, collect fiscal data, generate IPEs, generate client letters, produce
    state-level management reports, and produce counselor-level management
    reports
    Indicates whether an agency used an automated case management system
    Indicates whether the automated case management system was new if an
    agency used one
    Indicates whether an agency used an automated case management system and
    if so, whether the system was new
    Indicates whether case management system could collect RSA 911 data
    Indicates whether case management system could collect fiscal data
    Indicates whether case management system could generate IPEs
    Indicates whether case management system could generate client letters
    Indicates whether case management system could generate state level
    management reports
    Indicates whether case management system could generate reports at VR
    counselor level
    Integration with outside partners
    Indicates whether any VR staff worked full-time or part-time at Workforce
    Investment Act (WIA) one-stops
    Total number of staff (both full- and part-time) that worked at a WIA site
    Indicates whether VR program purchased any services from public or private
    vendors
    Indicates how many purchased services had fee for service arrangements
    Indicates how many purchased services had contracts with outcome-based
    performance measures
    Indicates how many purchased services had vendor fees tied to meeting
    performance measures
    Indicates how many purchased services had renewal of their contracts tied
    to meeting performance measures
    Indicates how many purchased services were evaluated by VR to see whether
    performance measures were met at contract end
    Indicates how many purchased services were evaluated by VR by group or
    type of vendor
    Scale indicating the average support level received from different types
    of programs including WIA one-stops, social service departments, mental
    health departments, education systems, Medicaid program, Medicare program,
    substance abuse departments, and developmental disabilities programs
    Indicates the extent to which a VR program received support from the state
    WIA one-stop system
    Indicates the extent to which a VR program received support from state
    social services
    Indicates the extent to which a VR program received support from the state
    mental health department
    Indicates the extent to which a VR program received support from the state
    education system
    Indicates the extent to which a VR program received support from the state
    Medicaid program
    Indicates the extent to which a VR program received support from the state
    Medicare program
    Indicates the extent to which a VR program received support from the state
    substance abuse department
    Indicates the extent to which a VR program received support from the state
    development disabilities program
    Indicates the extent to which a VR program received support from another
    state program
    Integration with business community
    Scale indicating agency's level of integration with the business
    community, including the average frequency with which the agency sponsors
    job fairs, attends business network meetings, meets with local businesses,
    meets with chambers of commerce, interacts with civic clubs, and hosts
    employer breakfasts
    Frequency with which agency sponsored job fairs
    Frequency with which agency representatives attended job fairs
    Frequency with which agency representatives attended meetings of business
    networks
    Frequency with which agency met with local businesses
    Frequency with which agency met with local chambers of commerce
    Frequency with which agency representatives interacted with civic clubs
    Frequency with which agency hosted employer breakfasts
    Frequency with which agency representatives participated in other business
    outreach
    Agency structure 1--indicates whether agency is (1) part of an umbrella
    agency with autonomy over its own staff and finances, (2) part of an
    umbrella agency without autonomy over its own staff and finances, (3) a
    stand-alone agency, and (4) other type of agency
    Agency structure 2--indicates whether agency is part of an umbrella agency
    Agency structure 3--indicates whether agency is in an umbrella agency that
    was a part of (1) social services, (2) education, (3) labor (4) human
    services, (5) a stand-alone, or (6) other type of agency
    Agency staffing
    Percentage of service delivery sites staffed full-time^a
    Percentage of service delivery sites staffed part-time^a
    Percentage of service delivery sites shared with social services^a
    Percentage of service delivery sites shared with education^a
    Percentage of service delivery sites shared with labor^a
    Percentage of service delivery sites shared with human services^a
    Percentage of service delivery sites shared with other agencies^a
    Indicates whether the VR program experienced a hiring freeze in a given
    fiscal year
    Indicates whether the VR program experienced a large number of retirements
    in a given fiscal year
    Indicates whether the VR program experienced a large influx of new hires
    in a given fiscal year
    Indicates whether the VR program experienced downsizing through layoffs in
    a given fiscal year
    Indicates whether the VR program experienced unusual changes in staffing
    in a given fiscal year
    Indicates whether VR counselors were affiliated with a union in a given
    fiscal year
    Agency management

    Source: GAO survey data.

    a Indicates variables that were categorized.

    To determine whether the survey data were sufficiently reliable for our
    analysis, we collected and analyzed additional data. Specifically, we
    included questions in the survey that were designed to determine whether
    each state VR agency uses certain practices to monitor the quality of
    computer-processed data that were used to complete the survey.^6 From
    these questions, we developed a variable to indicate whether a particular
    agency might have unreliable data. To determine whether there was a
    relationship between agencies with data reliability issues and the
    earnings outcomes we were studying, we included this variable in our three
    models of earnings outcomes (described below).

    We found two issues associated with the survey data that are related to
    our findings. First, net of other effects, agencies that reported having a
    data reliability issue had significantly lower rates of SSA beneficiaries
    departing the disability rolls.^7 Although we suspect that data quality
    issues do not have a direct effect on the rates of SSA beneficiaries
    departing the rolls, poor data quality might be correlated with some other
    characteristic that we were not able to measure (e.g., agency efficiency),
    which may have an impact on the rate of departures from the rolls. Second,
    11 agencies did not report the percentage of CSPD-certified counselors (a
    variable that we found to be significantly related to the percentage of
    SSA beneficiaries with earnings during the year after completing VR) for
    at least 1 year. For these agencies, the percentage of counselors was
    imputed using the mean derived from agencies that did report. Statistical
    tests were conducted to ensure that the observations for which data were
    imputed did not have significantly different rates of having earnings than
    those for which the data were not missing.

    ^6Specifically, we inquired about whether (1) there were written
    procedures that define data elements or specify how the data for each data
    system were collected and if so, how well the procedures were followed;
    (2) anyone conducted routine internal reviews of the data to check for
    errors in completeness, accuracy, or reasonableness; (3) anyone
    independent of the organization conducted periodic monitoring or audits of
    the data to check for errors in completeness, accuracy, or reasonableness;
    and (4) there were any potential problems or limitations with the
    reliability of the data that were used to answer the survey questions.

    ^7This variable was significant at the 0.10 level.

    Section 2: Study Population and Descriptive Analyses

    Study Population

    In consultation with SSA officials and contractors as well as Education
    officials, we selected as our study population working age individuals who
    (1) were either receiving Disability Insurance (DI) only, Supplemental
    Security Income (SSI) only, or both DI and SSI benefits concurrently; and
    (2) exited VR after having completed VR services.^8 To use the most recent
    data available, we further refined this population to include those
    beneficiaries who

               o Began receiving VR services no earlier than 1995 and who
               completed VR after having received services in fiscal years 2001
               though 2003.
               o Had received a DI or SSI benefit payment at least once during
               the 3 months before application for VR services. Beneficiaries
               were defined as concurrent if they received both DI and SSI
               benefits for at least 1 month in the 3 months before VR
               application. We selected a 3-month window to account for the fact
               that many beneficiaries, SSI beneficiaries in particular,
               fluctuate in their receipt of benefits for any given month.

    We excluded from our study population those disability beneficiaries who

               o Completed VR after 2003, because we lacked at least 1 year of
               post-VR earnings data.
               o Applied for or started VR services, but did not complete VR.
               o Began receiving disability benefits after receiving VR services
               because these beneficiaries may have differed in certain important
               characteristics from those receiving benefits before VR
               participation.
               o Reached age 65 or died at any point in their VR participation or
               during the time frame of our study. We excluded the beneficiaries
               who died or reached age 65 because they would have left the
               disability rolls for reasons unrelated to employment. For example,
               beneficiaries who reach age 65 convert to SSA retirement benefits.

    ^8Our study population included disabled adult children and disabled
    widow(er)s, who may receive DI benefits based on their parents' or
    spouses' Social Security earnings record. While their benefits are paid
    from the Old-Age and Survivors Insurance Trust Fund, these individuals are
    disabled and are eligible for VR services.

    Computation of Dependent Variables

    Using the Ticket Research File (TRF) subfile combined with data from SSA's
    Master Earnings File (MEF), we computed three measures of earnings
    outcomes for the 2001 through 2003 exit cohorts for each state VR agency:
    (1) the percentage of beneficiaries who had earnings during the year after
    receiving VR services, (2) the average amount they earned,^9 and (3) the
    percentage that left the disability rolls by 2005. The data sources for
    our three earnings outcomes or dependent variables are shown in table 5.

    Table 5: Dependent Variables Used in the Analyses

                                                             Dataset from which
    Dependent variable                                    variable was derived
    Percentage of beneficiaries with earnings during the  MEF
    year after VR
    Average annual earnings for SSA beneficiaries among   MEF
    those with earnings during the year after exiting VR
    Percentage of beneficiaries that left the rolls by    TRF subfile
    2005

    Source: SSA data.

    To adjust for inflation, all of our earnings figures were computed in 2004
    dollars using the Consumer Price Index for All Urban Consumers (CPI-U).
    The CPI-U, maintained by the Bureau of Labor Statistics, represents
    changes in prices of all goods and services purchased for consumption by
    urban households. The CPI-U can be used to adjust for the effects of
    inflation, so that comparisons can be made from one year to the next using
    standardized dollars. We standardized the value of average annual earnings
    to 2004 dollars because this was the most recent year for which earnings
    data were available at the time of our analysis.

    Departures from the Disability Rolls

    To determine whether disability beneficiaries left the rolls before 2005,
    we used data from the TRF subfile that indicated the month in which a
    beneficiary left the rolls because of work. We included all beneficiaries
    who left the rolls after their VR application date. Concurrent
    beneficiaries were considered to have left the rolls only if they stopped
    receiving benefits from both programs.

    ^9To determine a beneficiary's earnings in the year after VR, we
    calculated earnings in the calendar year after the year in which
    beneficiaries completed VR. For example, whether the beneficiary completed
    VR in January or December 2000, earnings from January 2001 through
    December 2001 would have been used to determine earnings in the year after
    VR.

    Descriptive Analyses

    To depict the variation of agency performance in earnings outcomes of SSA
    beneficiaries completing VR from 2001 to 2003, we performed two
    descriptive analyses. First, we developed distributions of each earnings
    outcome. Second, we computed the means and ranges of these outcomes by
    year and agency type. With data from 78 agencies over 3 years (from
    persons who exited the state VR programs from 2001 to 2003), we had 234
    cases in our data file.^10 Both sets of analyses are presented in the
    findings section of the report.

    Section 3: Econometric Analyses

    To identify key factors related to the earnings outcomes of SSA
    beneficiaries completing VR programs, we used econometric methods to
    analyze data from various sources related to VR agencies and the SSA
    beneficiaries who exited them from 2001 through 2003. Our econometric
    analyses focused on the differences across agencies for the three
    different dependent variables: (1) the percentage of beneficiaries who had
    earnings during the year after leaving VR; (2) among those with earnings,
    the average beneficiary earnings level during the year after leaving VR;
    and (3) the percentage of beneficiaries that left the disability rolls as
    a result of finding work by the end of 2005.

    We began our econometric analysis with ordinary least squares (OLS) and
    logistic regression models to analyze differences in outcomes based on
    individual characteristics. That is, we started with as many observations
    as there were individuals in our study population, each observation being
    assigned the characteristics of the agency as well as of the individual.
    Given that our data were multilevel (i.e., included information on both
    individuals and agency-level characteristics), we used statistical
    techniques to assess the feasibility of using ordinary least squares and
    logistic regression at the individual level rather than hierarchical
    modeling techniques.^11 As a result of these analyses, we chose to use
    robust standard errors to account for clustering in agencies rather than
    hierarchical modeling techniques. However, preliminary analyses using the
    individual-level data to model binary outcomes and each individuals'
    earnings revealed that regression and logistic models frequently failed
    statistical tests when compared to a null model with no explanatory
    variables, and only accounted for a small fraction of the variability
    outcomes of interest to us.^12

    10We have only 232 observations in our model of earnings because we
    considered average earnings among only those beneficiaries with earnings
    in the year following their exit from VR. Two agencies did not have any
    beneficiaries with reported earnings from employment in 2002.

    ^11We used Stata's xtreg and rtlogit commands to calculate the intraclass
    correlation coefficient rho. These analyses revealed minimal clustering
    among individuals within agencies (rho of 0.02 and below); that is,
    individuals' characteristics and employment outcomes appeared to vary as
    much within agencies as across agencies. This suggests that inferences
    derived from OLS and logistic regressions with robust standard errors are
    not misleading as a result of failure to hierarchically account for
    clustering of individuals within agencies.

    Because our econometric models using individual-level data explained very
    little variation in earnings outcomes (i.e., low predictive power), we
    proceeded to model outcomes at the agency level. Specifically, we combined
    data on the aggregate characteristics of individuals within agencies (such
    as the percentage of female beneficiaries or Disability Insurance
    recipients within an agency) with agency-level data on structure,
    expenditures, and policies and practices. In other words, rather than
    assess whether individuals differed in the likelihood of getting a job or
    leaving the rolls or had different earnings, we analyzed whether the
    agencies' earnings outcomes varied as a function of the characteristics of
    the agencies, the aggregate characteristics of beneficiaries within each
    agency, and the characteristics of the states the agencies were located
    in.^13 Our dependent variables thus contained, for each agency in a given
    fiscal year, the average earnings level among those with jobs, the
    percentage at each agency who had earnings during the year after
    completing VR, and the percentage of those leaving the rolls due to work.

    As with our descriptive analysis, we had 234 cases in our data file, a
    number that was fairly small relative to the large number of agency
    characteristics whose effects we wanted to estimate.^14 We could not, as a
    result, fit models that estimated the effects of all of the
    characteristics of interest simultaneously to determine which were
    statistically significant. We therefore chose to proceed by first
    estimating, in a series of bivariate regression models, which state and
    clientele characteristics (or characteristics of the types of SSA
    beneficiaries served in each agency) were significant. After obtaining
    preliminary estimates, we aggregated sets of significant state and
    clientele characteristics into single models for each of the three
    outcomes, and reassessed the significance of their net effects when they
    were estimated simultaneously in a multivariate regression model.^15 We
    next tested the stability and magnitude of statistically significant
    coefficients for the state and clientele characteristics under different
    model specifications, and proceeded to introduce the agency
    characteristics (e.g., structure, management, expenditures, etc.) one at a
    time into these base models with the significant state and case mix
    characteristics. After determining individually significant agency
    characteristics, we used an iterative procedure to reassess agency-level
    effects by testing model stability and which variables were and were not
    significant when others were included, and retesting the significance of
    selected state, case mix, and agency characteristics that were marginally
    significant in prior models.^16 In all cases we used robust regression
    procedures to account for the clustering of cases within agencies (i.e.,
    the lack of independence within agencies over time), and weighted the
    cases in our analyses according to either the total number of
    beneficiaries in each agency in each year (for models of having earnings
    or leaving the rolls) or the total number of beneficiaries with earnings
    due to work in each year (for models of earnings).

    ^12For example, our multivariate models of earnings were only able to
    explain, at best, approximately 8 percent of the variation in individuals'
    earnings.

    ^13Although the alternative of looking at individual outcomes with
    individual data might have allowed us to control for individual
    characteristics somewhat better before estimating the effects of the state
    and agency characteristics, we believe modeling the variability in
    outcomes using the aggregate data was more appropriate given the objective
    of assessing which agency-level characteristics are related to employment
    outcomes. However, because aggregation reduces our degrees of freedom and
    may compound individual measurement error in variables such as earnings,
    we recognize that our estimated coefficients may not be as precise as ones
    generated using individual characteristics. See section 4 of this appendix
    for more information on measurement issues.

    ^14We have only 232 observations in our model of earnings because two
    agencies did not have any beneficiaries with reported earnings from
    employment in 2002.

    Ultimately, we obtained the models shown in table 6. Each of the models
    consisted of 7 to 9 characteristics that jointly accounted for between 66
    and 77 percent of the variability in each dependent variable. Although
    certain characteristics were significant in some specifications for each
    outcome, the limited degrees of freedom prevented us from including all
    but the most consistently significant variables with greatest stability
    across models. In the models that estimated factors affecting the
    percentage of SSA beneficiaries who had earnings and factors affecting
    average earnings, state characteristics accounted for a substantial
    portion of the explained variance. Although state characteristics were
    also important in the model estimating the percentage getting off the
    rolls by 2005, the year that beneficiaries exited the agency accounted for
    the greatest portion of the variance explained, a result reflecting that
    those who exited the rolls earlier had more time to do so.

    ^15Statistical significance was measured at a p-value <0.05 and marginal
    significance was measured at p-value <0.10.

    ^16Although we considered this full range of variables in the series of
    models leading to our final specifications for each outcome, not all
    variables are significant for each outcome. We used a variety of factors
    to decide which characteristics to include or exclude in the model for
    each outcome. We considered statistical significance, magnitude of each
    effect, stability of included coefficients across model specifications
    with different regressors, changes in the proportion of variance explained
    using F-tests for nested models (when appropriate), and theoretical
    considerations based on past research and input from agencies we surveyed
    and interviewed.

    Table 6: Coefficients for Multivariate Models Estimating the Effects of
    State and Agency Characteristics on Three VR Outcomes, and the Proportion
    of Variance Explained (R-Squared) by Each Model

    Significant explanatory
    variables for percentage of
    beneficiaries with earnings
    during the year after VR                           Robust standard
    (R-squared = 0.66)              Effect coefficient           error P-value
    Unemployment rate                            -2.22            .358   <.001
    Per capita income (per $10,000)               3.90            1.42    .008
    Population size (per 1 million)               -.40            .047   <.001
    Percentage of female                         -.508            .202    .014
    beneficiaries
    Combined agency                               8.08            1.95   <.001
    General agency                               10.35            2.22   <.001
    Proportion of SSA beneficiaries               .397            .140    .006
    served
    Percentage of counselors                      5.63            2.06    .008
    meeting CSPD requirements
    In-house benefits counselor                 -.3.61           1.251    .005
    Constant                                     60.19            9.89   <.001
    Significant explanatory
    variables for average earnings
    among SSA beneficiaries                            Robust standard
    (R-squared = 0.77)               Effectcoefficient           error P-value
    Per capita income (per $10,000)            1684.54          185.25   <.001
    Percentage of beneficiaries                 -82.16            3.61   <.001
    with mental impairments
    Percentage of beneficiaries on              -64.52            8.20   <.001
    Disability Insurance
    Agency integration with                     727.37          350.63    .041
    business community
    Degree of support/cooperation               858.15          249.87    .001
    with other agencies
    Percentage of expenditures on                12.54            3.95    .002
    training
    Proportion of SSA beneficiaries             -27.55           11.77    .022
    served
    Constant                                   9059.65          824.75   <.001
    Significant explanatory
    variables for percentage of
    beneficiaries leaving the
    disability rolls (R-Squared =                       Robuststandard
    0.76)                            Effectcoefficient           error P-value
    Exit year 2002                               -1.89            .170   <.001
    Exit year 2003                               -4.07            .194   <.001
    Per capita income (per $10,000)               1.98            .326   <.001
    Population size (per 1 million)              -.068            .014   <.001
    Percentage of beneficiaries on               -.040            .016    .021
    Disability Insurance
    Percentage of beneficiaries 46               -.078            .043    .073
    to 55 years of age
    Percentage of beneficiaries                  -.084            .016   <.001
    with mental impairments
    Percent of beneficiaries with                -.045            .010   <.001
    visual impairments
    Agency integration with                       1.62            .788    .044
    business community
    Constant                                     11.64            1.58   <.001

    Source: GAO analysis of SSA, Education, GAO survey, and data from various
    sources listed in table 3.

    Note: While earnings are coded in units of dollars, per capita income is
    coded in units of $10,000 so that the coefficient represents the effect of
    a $10,000 change in per capita income. Population is coded in per million
    persons. Percentage and proportion variables are coded between 0 and 100;
    this includes the percentage of beneficiaries with earnings, leaving the
    rolls, on Disability Insurance, and with mental and visual impairments, as
    well as the percentage of agency budget spent on training and the
    percentage of CSPD-certified counselors. Agency integration with the
    business community and support and cooperation with other agencies are
    scaled between 0 and 1.

    Taking each outcome one at a time, the coefficients in table 6 suggest the
    following:

               o The percentage of SSA beneficiaries who had earnings was
               significantly lower in more populous states and states with higher
               unemployment rates, and significantly higher in states with higher
               per capita incomes. The percentage of SSA beneficiaries who exited
               VR agencies and had earnings was also significantly lower in
               agencies in which greater percentages of the beneficiaries are
               female. Independent of these effects of the state in which the
               agencies are located, and the gender composition of the
               beneficiaries who exit the agencies, there were also significant
               net effects of certain agency characteristics. Agencies that
               served only blind beneficiaries had lower percentages of SSA
               beneficiaries who had earnings than combined agencies or general
               agencies that did not serve the blind. The percentage of SSA
               beneficiaries who had earnings was also higher in agencies that
               served a higher proportion of SSA beneficiaries and had a higher
               percentage of counselors who met CSPD requirements, but lower in
               agencies that had an in-house benefits counselor.

               o Among the SSA beneficiaries who had earnings, those who were in
               agencies in states that had higher per capita incomes had higher
               average incomes. Net of these effects, agencies that (1) were more
               integrated with the business community, (2) had a higher degree of
               support and cooperation from other agencies, and (3) spent more of
               their total budget on training had higher average annual incomes
               among SSA beneficiaries completing VR services. Agencies with a
               higher percentage of beneficiaries on Disability Insurance, with
               mental impairments, and higher proportions of SSA beneficiaries
               served had lower average annual incomes among SSA beneficiaries
               completing VR services.

               o With respect to leaving the disability rolls by 2005, our final
               model showed that beneficiaries who exited agencies more recently
               were less likely to leave the rolls by 2005. (See section 4 for an
               explanation of why this might be the case.) Net of this, agencies
               in states with larger populations had lower percentages of
               beneficiaries who left the rolls by 2005, while agencies in states
               with higher per capita incomes had higher percentages who left the
               rolls by 2005. The characteristics of clientele served by each
               agency had a significant effect on the percentage of SSA
               beneficiaries who got off the rolls by that year. Agencies with
               higher percentages of beneficiaries who were blind or had mental
               impairments had lower percentages getting off the rolls by 2005.
               Those agencies with a higher proportion of DI beneficiaries had
               lower percentages that got off the rolls by 2005. The only agency
               characteristic with consistent statistical significance was the
               integration with the local business community; a greater
               proportion of beneficiaries in agencies with better integration
               with businesses left the rolls.^17 Agencies with a greater
               proportion of beneficiaries from 46 through 55 years of age had
               fewer recipients leaving the rolls, but this effect is only
               significant at the 90 percent confidence level in our final model.

    Section 4: Limitations of our Analyses

    Our results cannot be generalized to the larger population of all SSA
    disability beneficiaries or all VR participants because we looked only at
    VR participants who were SSA beneficiaries. Because VR participation is
    voluntary, beneficiaries who participate in VR may have certain
    characteristics that make them different from other SSA beneficiaries and,
    therefore, more likely or less likely to succeed in the workforce.

    Because our primary goal was not to conduct an impact evaluation of the VR
    program, but rather to conduct a comparative analysis of earnings outcomes
    across state VR agencies to determine what might account for differences
    in state agency performance, we felt that our analysis did not require a
    control group of SSA beneficiaries that did not receive VR services.
    Nonetheless, as a secondary analysis goal, we attempted to identify such a
    group using the data that were available to us on SSA beneficiaries that
    had applied for but did not receive VR services. However, we were unable
    to identify a subset of this group that was sufficiently similar to the VR
    participants to feel confident that any differences in earnings outcomes
    that we found between them and those that completed the VR program would
    be attributable to the VR program and not to the differences in individual
    characteristics. Therefore, our findings do not allow us to report on the
    overall effectiveness of the VR program.

    ^17Several other agency characteristics, notably the percentage of
    expenditures on purchased services, had marginally significant effects and
    were not included in the final model.

    Our earnings data had several limitations that may have resulted in an
    under- or overestimation of beneficiaries' earnings. For example, although
    the beneficiary earnings data were provided to SSA by the Internal Revenue
    Service and are considered to be the most comprehensive and accurate
    measure of earnings available, they excluded several categories of workers
    who participated in alternative retirement systems and whose earnings may
    not have been reported to SSA.^18 Such omissions could have resulted in an
    under- or overestimate of beneficiary earnings. On the other hand, some
    earnings reported to SSA may have included income derived from work
    activity in a previous year, such as commissions or bonuses. Further, the
    earnings data included some forms of nonwork income, such as sick leave
    earnings and profit sharing. These additional sources of income could not
    be identified and, therefore, could result in an overestimation of
    beneficiaries' earnings in a particular year. The data did not allow us to
    estimate the magnitude of the effect of these factors on our analyses.

    Our findings that beneficiaries receiving DI and beneficiaries from later
    cohort years were less likely to leave the rolls are likely due to several
    factors related to program structure and the updating of data. First,
    under current program rules, DI beneficiaries are allowed a trial work
    period (9 months) and an extended period of eligibility (36 months) before
    they are considered off the rolls.^19 SSI beneficiaries who earn enough so
    that they do not receive a benefit for 12 months are taken off the rolls.
    Therefore, given that we measured whether beneficiaries left the rolls by
    2005, beneficiaries from earlier cohort years would have had more time to
    leave the rolls. Further, by 2005 many DI beneficiaries may not yet have
    entered or completed their extended period of eligibility or reached the
    point where they would have been considered off the rolls.

    In addition, delays in the reporting of earnings may also have contributed
    to our finding that relatively more SSI beneficiaries and beneficiaries
    from earlier cohort years left the rolls. There can be a significant
    delay--up to 3 years--between when beneficiaries begin work and when SSA
    is notified or learns of their earnings. This delay is more likely to
    occur with DI beneficiaries, whose earnings were reviewed on a yearly
    basis as compared to monthly earnings reviews for SSI beneficiaries during
    the time frame of our study. Because of this reporting delay, the TRF
    subfile data that indicated whether a beneficiary had left the rolls may
    not have contained completely up-to-date data, especially for later
    cohorts. For these two reasons, we did not include comparisons of the
    rates of departures from the disability rolls by exit year because they
    would be misleading.

    ^18Workers who may have been excluded include federal civilian employees
    hired before 1984 and certain state and local government employees.

    ^19The 9-month trial work period must occur within a 60-month period.

    With respect to earnings after VR, we included all earnings in the
    calendar year after VR, irrespective of the time gap between VR completion
    and the first month of earnings. Therefore, the start month for
    calculating earnings in the year after VR could have ranged from 1 to 12
    months after VR, depending on which month the beneficiary exited. In other
    words, beneficiaries who exited VR in January 2000 would have their 2001
    annual earnings calculated beginning in January 2001--12 months after
    their exit from VR. In contrast, beneficiaries who completed VR in
    December 2000 would have been out of VR for 1 month when their 2001 annual
    earnings calculation started in January 2001. We have no indication of
    clustering in earnings relative to VR completion and therefore expect a
    fairly even distribution of earnings over time. We do not expect the time
    lag in the earnings calculation to vary systematically by year or cohort.

    Finally, our analysis of the impact of agency characteristics on earnings
    outcomes of particular exit cohorts was limited by the time frames of our
    agency data. Specifically, we used information on the agency that
    pertained to the year before each exit cohort completed VR to explain the
    earnings outcomes of that exit cohort (e.g., agency data from 2000 were
    used to explain 2001 beneficiary outcomes). We did this for two reasons.
    First, beneficiaries, on average, receive services from VR for
    approximately 2 years. Therefore, for any given exit cohort, data on the
    agency from the year prior to exit will cover the most beneficiaries in
    that exit cohort. Second, although data from previous years might also
    explain beneficiary outcomes for a given cohort, we did not want to impose
    an inordinate burden on our survey respondents by collecting data on many
    years, especially those prior to 2000. In conducting preliminary tests of
    our survey questions, we also learned that the quality of the data may
    have been lower in earlier years because some agencies retain data for a
    limited time period.

    Appendix II: Comments from the Department of Education

    Note: GAO's comments supplementing those in the report text appear at the
    end of this appendix.

    See comment 1.

    See comment 3.

    See comment 2.

    See comment 6.

    See comment 5.

    See comment 4.

    See comment 7.

    See comment 9.

    See comment 8.

    The following are GAO's comments on the Department of Education's letter
    dated April 17, 2007.

    GAO Comments

    1. Education noted that more contextual information would help readers
    better understand the data and findings in the report. We added additional
    information to the report about VR reimbursement for successful SSA
    beneficiary rehabilitations, the role of disability determination services
    in VR referral, as well as a reference on where to find more information
    on the structure of federal disability programs.

    2. We disagree with Education that our measures of statistical
    significance do not necessarily translate into issues of programmatic
    significance for the VR program because it is a formula grant program or
    that the agency characteristics we identify as having a significant impact
    on agency-level performance with respect to SSA beneficiaries may be
    overstated. Given Education's important leadership role in overseeing the
    VR agencies, we believe that our findings are relevant to the guidance and
    information that Education may choose to provide to state VR agencies.
    While we acknowledge that many SSA disability beneficiaries will not be
    able to return to work and leave the rolls for a variety of reasons, such
    as the severity of their disability, we analyzed numerous versions of our
    models and only reported on the relationships that were consistently
    significant across many versions of the model. As such, we believe these
    relationships are valid and deserve careful consideration.

    3. Education stated that there is a significant difference between helping
    more SSA beneficiaries participate in the workforce versus helping more
    leave the disability rolls. While we agree, we believe that participating
    in the workforce is an important first step and improves SSA
    beneficiaries' potential for leaving the rolls.

    4. We agree with Education that our description of the states' CSPD
    certifications could be misconstrued. We clarified the CSPD language in
    the background, findings, and recommendation sections of the report.

    5. Education stated that overall collaboration between VR agencies and
    other agencies providing complementary services may improve outcomes, but
    that some collaboration resulting in better services to individuals with
    disabilities may not always support more or better rehabilitations for SSA
    beneficiaries. When we tested the effect of receiving support from
    specific agencies on SSA beneficiary outcomes, we did not find support
    from individual agencies to be significant. (See table 4 in app. I under
    "Integration with Outside Partners" for a list of variables we tested.)
    However, we found that when these relationships were aggregated, agencies
    that received a greater degree of support from more than one public agency
    had significantly higher levels of earnings among SSA beneficiaries.

    6. Education suggested that VR agencies with high proportions of SSA
    beneficiaries may also have high levels of collaboration with other
    agencies because SSA beneficiaries may require long-term supports to live
    in the community, which may in turn necessitate cooperation with other
    public programs. The department noted that this may account for our
    findings because benefit eligibility may be necessary to receive certain
    supports from outside agencies. We added a footnote with this potential
    explanation to the report.

    7. Education noted that we questioned the validity of certain
    self-reported data in this report, but deemed similar data acceptable in
    our 2005 VR report.^1 In relation to our 2005 report, this report
    references different self-reported data for different purposes.
    Specifically, this report refers to clients' earnings data at the time of
    application, whereas our 2005 report used clients' earnings data after
    exiting VR (which was not used in this report). More importantly, this
    report used data as part of an econometric model whereas our 2005 report
    used self-reported data for descriptive purposes. While it is always
    preferable to verify self-reported data, our reliability tests are limited
    to our intended use of the data, and the data's reliability for that
    purpose. Education said it was open to our recommendation, but sensitive
    to the reporting burden on the VR agencies. Our recommendation that
    Education explore cost-effective ways to validate self-reported data was
    based on the experience of some VR agencies that have obtained data
    successfully from official sources and not relied solely on self-reported
    information.

    8. Education disagreed with our recommendation on when economic conditions
    and state demographics should be considered in assessing agency
    performance. Instead of using this information to help set performance
    measures, the department said that it takes these factors into account
    when it monitors agency performance results and believes that its approach
    is effective. However, on the basis of the statistical significance of
    economic factors in our analysis, we believe that incorporating this
    contextual information in assessing performance measures is essential to
    provide state agencies with a more accurate picture of their relative
    performance. Education also stated that the VR program's statutory funding
    formula allocates relatively more funds to poorer states based on per
    capita income to offset the lack of other resources in the state. However,
    if the additional funds allocated to VR agencies located in states with
    low per capita incomes actually offset the lack of other resources in the
    state, we would not have found a significant relationship between per
    capita income and state VR performance. Finally, Education stated the
    overall state unemployment rate may not entirely correspond to the jobs
    available to SSA beneficiaries. In our analysis, however, this variable
    was highly significant in explaining the percentage of SSA beneficiaries
    with earnings for state VR agencies.

    ^1 [51]GAO-05-865 .

    9. We agree with Education that the report's title should indicate that
    our sample was limited to SSA beneficiaries and we modified the title
    accordingly.

    Appendix III: Comments from the Social Security Administration

    Note: GAO's comments supplementing those in the report text appear at the
    end of this appendix.

    See comment 5.

    See comments 1 and 2.

    See comment 2.

    See comment 1.

    See comment 3.

    See comment 6.

    See comment 2.

    See comment 5.

    See comment 4.

    See comment 2.

    See comment 8.

    See comment 7.

    See comment 2.

    See comment 9.

    The following are GAO's comments on the Social Security Administration's
    letter dated April 13, 2007.

    GAO Comments

    1. We disagree with SSA that many of the comments provided on our previous
    report ( [52]GAO-07-332 ) apply to this report because the methods and
    data we used differed significantly from our earlier report. Prior to
    submitting this report for agency comment, we carefully reviewed and
    incorporated any comments from the earlier report that were relevant.

    2. SSA stated that the report has methodological flaws that introduced
    aggregation bias and false correlations. It suggested that we should have
    focused on individual-level analysis or reported the results of both the
    individual- and aggregate-level analyses. We disagree, as the primary goal
    of our analysis was to analyze agency-level outcomes, not individual-level
    outcomes. Specifically, our objective was to understand what "may account
    for the wide variation in state VR agency outcomes with respect to SSA
    beneficiaries." In doing so, we used aggregated data, which is a widely
    used and, at times, necessary means of analysis throughout all social
    sciences. Because we used aggregated data, we did not attempt to infer the
    effects of individual behavior or individual outcomes and noted such in
    our report. For example, we did not find that a lower percentage of women
    beneficiaries had earnings relative to male beneficiaries. Rather, we
    found that agencies serving a higher proportion of women beneficiaries had
    lower percentages of beneficiaries with earnings relative to other
    agencies.

    We did not report the results from the individual-level analyses, as
    recommended by SSA, because we did not find them sufficiently reliable
    upon which to base findings. Specifically, we did not find the
    individual-level results to be reliable, as we were not able to control
    for some factors at the individual level--for example, severity of
    disability--that were crucial to an individual-level analysis, but not
    crucial to analyses at the aggregated level. Although we chose not to
    report individual-level results, they were, in fact, consistent with the
    results of our aggregate analyses.

    We conducted statistical tests prior to our agency-level analyses to
    ensure that our aggregate analyses were not biased by a failure to account
    for certain types of correlations between individuals within agencies. Our
    tests did not reveal such correlations. In the absence of such
    correlations, several respected authorities agree that aggregate-level
    analyses that incorporate aggregated individual-level characteristics will
    not result in biased estimates.^1 To further ensure our methods were
    appropriate and robust, our final report was reviewed and validated by an
    expert in statistical analysis.^2

    3. We agree with SSA that state agency rules about whether and how
    disability beneficiaries are referred to VR may have an affect on agency
    success rates, and controlled for it to the extent possible in our
    analysis. While we were not able to control for differences in the way
    states target beneficiaries for referral to VR as SSA suggested, we did
    include a variable reflecting the percentage of SSA beneficiaries served
    by a VR agency (computed as the percentage of all clientele served at that
    agency). This variable was significant in two of our three models.

    4. SSA had concerns that the Ticket to Work program was implemented during
    the time frame of our study and should have been controlled for in our
    analysis. Although there was a very slight overlap between the time frame
    of our study and the timing of the Ticket to Work program, we nevertheless
    conducted tests to determine whether the rollout of the Ticket program had
    an effect on VR agency outcomes for SSA beneficiaries. The rollout of the
    Ticket program was not significant and, therefore, we did not report its
    effect.

    5. SSA questioned the value of measuring state-level economic factors and
    the resultant implications for VR. Our findings on the influence of state
    economic characteristics were highly significant and are corroborated by
    previous research, and therefore we believe that implications and
    recommendations can be offered from our analysis of state economic factors
    on agency-level outcomes. However, nowhere in our report do we indicate
    that our findings suggest that during times of high unemployment, less
    funding should be allocated to VR agencies. To the contrary, we suggest
    that economic factors should be controlled, or accounted for, when
    assessing agency performance. Moreover, while we agree that economic
    conditions are associated with tax revenues, we found that total state
    agency expenditures on services (and several other expenditure variables
    listed in table 2 of app. I) were not significant predictors of
    agency-level earnings outcomes for SSA beneficiaries.

    ^1See, for example, Leigh Burstein, "The Analysis of Multilevel Data in
    Educational Research and Evaluation," Review of Research in Education,
    vol. 8, p. 158-233 (1980); Judith Singer, "Using SAS PROC MIXED to Fit
    Multilevel Models, Hierarchical Models, and Individual Growth Models,"
    Journal of Educational and Behavioral Statistics, vol. 24, no. 4 (1998);
    and Stephen W. Raudenbush and Anthony S. Bryk, Hierarchical Linear Models:
    Applications and Data Analysis Methods, second ed. (Thousand Oaks,
    California: Sage Publications, 2002), 99-159.

    ^2Herbert Smith, Professor of Sociology, Director of Population Studies,
    University of Pennsylvania.

    6. SSA had concerns about our findings on benefits counselors because they
    differed from those of other research. While prior research focused on the
    impact of benefits counseling in one state, our analysis focused on the
    impact of benefits counseling across all state agencies. Additionally, we
    noted that the time frame of our study was a period of transition for the
    benefits counseling program. Therefore, while we believe our findings are
    accurate, we also noted the contradictory findings in other research.

    7. We agree with SSA that the higher earnings thresholds for the legally
    blind allow them to earn more than other categories of workers with
    disabilities while still keeping their disability benefit and have
    modified our explanation of the results on beneficiaries who are blind.

    8. SSA stated that SSI beneficiaries generally lack a work history that
    qualifies them for DI benefits, and that this lack of work history is more
    likely to lead SSI beneficiaries into entry-level jobs, resulting in lower
    average annual earnings than DI beneficiaries. While we agree past work
    history can be a contributing factor, we found the opposite effect. We
    found that agencies with a higher proportion of SSA beneficiaries who were
    DI beneficiaries had lower average annual earnings among SSA beneficiaries
    and a lower percentage of beneficiaries leaving the rolls. We offer
    potential explanations for these results in the report.

    9. We incorporated SSA's technical comments as appropriate.

    Appendix IV: GAO Contacts and Staff Acknowledgments

    GAO Contact

    Denise M. Fantone, Acting Director, (202) 512-7215, fantoned@gao.gov

    Acknowledgments

    In addition to the contact named above Robert E. Robertson, Director;
    Michele Grgich, Assistant Director; Amy Anderson; Melinda Cordero; Erin M.
    Godtland; Jay Grussing; Robert Marek; Brittni Milam; Nisha R. Unadkat; and
    Rick M. Wilson made significant contributions to all phases of this
    report. In addition, Susan Bernstein, Cindy Gilbert, Lisa Mirel, Thomas
    McCool, Anna Maria Ortiz, Daniel A. Schwimer, Doug Sloane, and Shana B.
    Wallace provided technical assistance.

    Related GAO Products

    Vocational Rehabilitation: Earnings Increased for Many SSA Beneficiaries
    after Completing VR Services, but Few Earned Enough to Leave SSA's
    Disability Rolls. [54]GAO-07-332 . Washington, D.C.: March 2007.

    Vocational Rehabilitation: Better Measures and Monitoring Could Improve
    the Performance of the VR Program. [55]GAO-05-865 . Washington, D.C.:
    September 2005.

    SSA Disability: SGA Levels Appear to Affect the Work Behavior of
    Relatively Few Beneficiaries, but More Data Needed. [56]GAO-02-224 .
    Washington, D.C.: January 2002.

    SSA Disability: Other Programs May Provide Lessons for Improving
    Return-to-Work Efforts. [57]GAO-01-153 . Washington, D.C.: January 2001.

    Social Security Disability Insurance: Multiple Factors Affect
    Beneficiaries' Ability to Return to Work. [58]GAO/HEHS-98-39 . Washington,
    D.C.: January 1998.

    Social Security: Disability Programs Lag in Promoting Return to Work.
    [59]GAO/HEHS-97-46 . Washington, D.C.: March 1997.

    SSA Disability: Program Redesign Necessary to Encourage Return to Work.
    [60]GAO/HEHS-96-62 . Washington, D.C.: April 1996.

    Vocational Rehabilitation: Evidence for Federal Program's Effectiveness is
    Mixed. [61]GAO/PEMD-93-19 . Washington, D.C.: August 1993.

    (130534)

    GAO's Mission

    The Government Accountability Office, the audit, evaluation and
    investigative arm of Congress, exists to support Congress in meeting its
    constitutional responsibilities and to help improve the performance and
    accountability of the federal government for the American people. GAO
    examines the use of public funds; evaluates federal programs and policies;
    and provides analyses, recommendations, and other assistance to help
    Congress make informed oversight, policy, and funding decisions. GAO's
    commitment to good government is reflected in its core values of
    accountability, integrity, and reliability.

    Obtaining Copies of GAO Reports and Testimony

    The fastest and easiest way to obtain copies of GAO documents at no cost
    is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
    newly released reports, testimony, and correspondence on its Web site. To
    have GAO e-mail you a list of newly posted products every afternoon, go to
    www.gao.gov and select "Subscribe to Updates."

    Order by Mail or Phone

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent of
    Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
    copies mailed to a single address are discounted 25 percent. Orders should
    be sent to:

    U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
    D.C. 20548

    To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
    512-6061

    To Report Fraud, Waste, and Abuse in Federal Programs

    Contact:

    Web site: www.gao.gov/fraudnet/fraudnet.htm
    E-mail: fraudnet@gao.gov
    Automated answering system: (800) 424-5454 or (202) 512-7470

    Congressional Relations

    Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 U.S.
    Government Accountability Office, 441 G Street NW, Room 7125 Washington,
    D.C. 20548

    Public Affairs

    Paul Anderson, Managing Director, [AndersonP1@gao.gov (202) 512-4800
    U.S. Government Accountability Office, 441 G Street NW, Room 7149
    Washington, D.C. 20548

    www.gao.gov/cgi-bin/getrpt?GAO-07-521 .

    To view the full product, including the scope
    and methodology, click on the link above.

    For more information, contact Denise Fantone at (202) 512-4997 or
    fantoned@gao.gov.

    Highlights of [69]GAO-07-521 , a report to congressional requesters

    May 2007

    VOCATIONAL REHABILITATION

    Improved Information and Practices May Enhance State Agency Earnings
    Outcomes for SSA Beneficiaries

    State vocational rehabilitation (VR) agencies, under the Department of
    Education (Education), play a crucial role in helping individuals with
    disabilities prepare for and obtain employment, including individuals
    receiving disability benefits from the Social Security Administration
    (SSA). In a prior report (GAO-05-865), GAO found that state VR agencies
    varied in the rates of employment achieved for SSA beneficiaries. To help
    understand this variation, this report analyzed SSA and Education data and
    surveyed state agencies to determine the extent to which (1) agencies
    varied in earnings outcomes over time; (2) differences in state economic
    conditions, client demographic traits, and agency strategies could account
    for agency performance; and (3) Education's data could be used to identify
    factors that account for differences in individual earnings outcomes.

    [70]What GAO Recommends

    GAO recommends that Education promote certain promising practices
    identified in our analysis, reassess the data it collects on clients, and
    consider economic factors when measuring state agency performance.
    Education generally agreed with our recommendations, but disagreed that
    economic factors should be incorporated into performance measures. It
    considers these factors during monitoring and believes its approach to be
    effective. We maintain that these factors are critical to measuring
    agencies' relative performance.

    Our analysis of data on state agency outcomes for SSA beneficiaries
    completing VR found that state agencies varied widely across different
    outcome measures for the years of our review. For example, from 2001 to
    2003 average annual earnings levels among those SSA beneficiaries with
    earnings during the year after completing VR varied across state agencies
    from about $1,500 to nearly $17,000.

    Distribution of State Agency Average Annual Earnings for SSA Beneficiaries
    during the Year after VR

    Note: Earnings are in 2004 dollars.

    After controlling for a range of factors, we found that much of the
    differences in state agency earnings outcomes could be explained by state
    economic conditions and the characteristics of the agencies' clientele.
    Together state unemployment rates and per capita income levels accounted
    for roughly one-third of the differences between state agencies in the
    proportion of SSA beneficiaries that had earnings during the year after
    VR. The demographic profile of SSA clients being served at an agency--such
    as the proportion of women beneficiaries--also accounted for some of the
    variation in earnings outcomes.

    We also found that after controlling for other factors, a few agency
    practices appeared to yield positive earnings results. For example, state
    agencies with a higher proportion of state-certified counselors had more
    SSA beneficiaries with earnings during the year after completing VR.

    However, we were unable to determine what factors might account for
    differences in earnings outcomes at the individual level. This was due in
    part to Education's data, which lacked information on important factors
    that research has linked to work outcomes, such as detailed data on the
    severity of clients' disabilities. Although Education collects extensive
    client-level data, some key data are self-reported and not always verified
    by state agencies.

    References

    Visible links
      45. http://www.gao.gov/cgi-bin/getrpt?GAO-05-865
      46. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-97-46
      47. http://www.gao.gov/cgi-bin/getrpt?GAO-05-865
      48. http://www.gao.gov/cgi-bin/getrpt?GAO-07-332
      49. http://www.gao.gov/cgi-bin/getrpt?GAO-07-332
      50. http://www.gao.gov/cgi-bin/getrpt?GAO-05-626
      51. http://www.gao.gov/cgi-bin/getrpt?GAO-05-865
      52. http://www.gao.gov/cgi-bin/getrpt?GAO-07-332
      54. http://www.gao.gov/cgi-bin/getrpt?GAO-07-332
      55. http://www.gao.gov/cgi-bin/getrpt?GAO-05-865
      56. http://www.gao.gov/cgi-bin/getrpt?GAO-02-224
      57. http://www.gao.gov/cgi-bin/getrpt?GAO-01-153
      58. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-98-39
      59. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-97-46
      60. http://www.gao.gov/cgi-bin/getrpt?GAO/HEHS-96-62
      61. http://www.gao.gov/cgi-bin/getrpt?GAO/PEMD-93-19
      69. http://www.gao.gov/cgi-bin/getrpt?GAO-07-521
    *** End of document. ***