[Federal Register Volume 64, Number 56 (Wednesday, March 24, 1999)]
[Notices]
[Pages 14336-14345]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 99-7148]



[[Page 14335]]

_______________________________________________________________________

Part III





Department of Labor





_______________________________________________________________________



Employment and Training Administration



_______________________________________________________________________



Consultation Paper on Performance Accountability Measurement for the 
Workforce Investment System Under Title I of the Workforce Investment 
Act; Notice

  Federal Register / Vol. 64, No. 56 / Wednesday, March 24, 1999 / 
Notices  

[[Page 14336]]



DEPARTMENT OF LABOR

Employment and Training Administration


Consultation Paper on Performance Accountability Measurement for 
the Workforce Investment System Under Title I of the Workforce 
Investment Act

AGENCY: Employment and Training Administration, Labor.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: The purpose of this notice is to disseminate consultation 
papers for interested parties on the Performance Accountability 
Measurement System for Title I of the Workforce Investment Act. There 
are two papers. The first presents the broad framework for Core 
Measures of Performance and Customer Satisfaction specified in Title I, 
Section 136. The second presents the framework for Negotiating State 
Adjusted Levels of Performance as specified in Title I Section 136. 
These papers are to be used by States intending to implement the 
Workforce Investment Act as of July 1, 1999. The Department of Labor 
will work with States individually to ensure that there are no negative 
consequences if significant changes occur in these papers based on the 
comments received. Interested parties have 30 days to provide comments 
on these papers. Over the next several months additional consultation 
papers will also be disseminated for comment.

FOR FURTHER INFORMATION CONTACT: Mr. Eric Johnson, Workforce Investment 
Implementation Taskforce Office, U.S. Department of Labor, 200 
Constitution Avenue, NW, Room S5513, Washington, DC, Telephone: (202) 
219-0316 (voice) (This is not a toll-free number), or 1-800-326-2577 
(TDD). Information may also be found, or comments provided, at the 
website--http://usworkforce.org.

SUPPLEMENTARY INFORMATION: The Workforce Investment Act (WIA or Act), 
Pub. L. 105-220 (August 7, 1998) provides the framework for a reformed 
National workforce and employment system designed to meet the needs of 
the Nation's employers, job seekers and those who want to further their 
careers. Title I of WIA specifies Core Performance and Customer 
Satisfaction measures. Each Governor must submit a five-year strategic 
plan no later than April 1, 1999, to begin WIA programs by July 1, 
1999, and no later than April 1, 2000, to begin WIA programs by July 1, 
2000. The current law, the Job Training Partnership Act, is repealed 
effective July 1, 2000. States planning to implement during PY 1999 are 
to utilize these papers in addressing Performance Accountability in 
their plans.
    An important part of the five-year strategic plan is the 
establishment of performance levels for each of the core performance 
and customer satisfaction measures, which will be negotiated between 
the Governor and the Secretary of Labor. These levels will form the 
basis for incentives and sanctions as specified in Title I, section 136 
and Title V, section 503 of the Workforce Investment Act.
    The U. S. Department of Labor is establishing this performance 
accountability measurement system, and the process for reaching 
agreement on State adjusted levels of performance. These two 
consultation papers are part of that effort. Some of the questions on 
which the Department of Labor is seeking input are the following:
     Which services would be appropriately defined as self-
service/informational and thus not included in the core measures, and 
which services fall into the core services, intensive services or 
training;
     The point at which adult and youth registrants are counted 
for different performance measures (e.g., at a certain time after 
registration, during the reporting period, after completion of service, 
after program exit);
     The use of wage records for performance measurement 
considering availability, completeness, accuracy, timeliness and when 
wage records might be combined with supplemental sources (i.e., 
administrative records or survey data for performance purposes) 
considering the need for consistency, comparability and cost 
effectiveness;
     Who will be counted in the numerator and/or denominator of 
those measures expressed as rates, for example how should the employed 
and underemployed who receive services be accounted for;
     Identifying possible unintended effects resulting from 
definitions/policies around performance measurements;
     Identifying burdensome and unnecessary requirements that 
will provide limited benefit, but will be costly in terms of both 
record keeping requirements and processing;
     Using adjustment models in (1) negotiating State adjusted 
levels of performance to account for differences in service mix, 
participant characteristics and labor markets and/or (2) determining 
eligibility for incentives and consideration for sanctions comparing 
negotiation assumptions with actual information.
     Sources and types of information that would be useful in 
negotiating State adjusted levels of performance;
     The circumstances in which revisions to State adjusted 
levels of performance will be required by the Department, including 
special circumstances for early implementing States and the differences 
between the first program year and subsequent years covered by a 
State's 5-year plan. For example, if better data available in later 
years or if actual performance data shows that the assumptions under 
which State adjusted performance levels were negotiated are incorrect, 
should the Department require that approved levels for later years be 
changed.
    Please consider these issues as you review these consultation 
papers, and provide comments.

    Signed at Washington, DC, this 17th day of March 1999.
Raymond L. Bramucci,
Assistant Secretary of Labor, Employment and Training Administration.

Attachment 1--Performance Accountability Measurement for the 
Workforce Investment System

I. Introduction

    The Workforce Investment Act (WIA) calls for a comprehensive 
accountability system to assess the effectiveness of State and local 
areas in providing employability and training services. The Act 
requires:
     A focus on results defined by ``core indicators'' of 
performance;
     Customer Satisfaction with programs and services measured 
and related to results;
     A strong emphases on Continuous Improvement of Services;
     Annual performance levels and improvement plans developed 
during negotiations among Federal, State and local partners;
     Awards and Sanctions based on State performance; and
     State reporting and record keeping.
    In addition, States are required to provide annual reports to the 
Secretary of Labor with respect to progress in achieving State 
performance measures. The Act requires certain additional information 
be provided, such as cost of workforce investment activities and 
specified recipient data.
    This paper presents a draft framework for the workforce investment 
system core performance and customer service measures that apply to 
States in Title I of the WIA and will be used in determining State 
adjusted performance levels, eligibility for incentive grants or 
imposing sanctions. Additional papers on all of the above listed 
requirements

[[Page 14337]]

are being developed and will be provided for comment in the coming 
weeks. This paper is intended to elicit discussion about how success 
will be defined for workforce investment system activities and how it 
can efficiently and effectively be measured Statewide. The concepts 
within the paper build on previous Department of Labor (DOL) efforts 
such as the Workforce Development Performance Measures Initiative, the 
Labor Exchange Performance Measures Work Group, Simply Better!, 
Employment Service (ES) Reinvention and the Enterprise. It also 
incorporates input from State and local officials that was received at 
recent consultations focusing on WIA accountability and from other WIA 
briefings and communications.
    Please keep in mind that this document presents an overall 
framework. It does not fully address a number of the detailed technical 
and operational issues that were raised at recent consultations, nor is 
it intended to serve as reporting instructions. In addition to your 
feedback on this framework, we are also interested in input on 
technical and operational issues that may not have been addressed by 
this paper. All of this input will be used to develop further guidance 
and finalize the document for use by those States planning to implement 
the Workforce Investment Act before July 2000. This paper includes 
proposed definitions for the Core Indicators of Performance that will 
be used for State incentive grant eligibility determinations and 
sanctions.
    The paper is divided into three sections:
     Adult Performance Measures and Definitions that will apply 
separately to: (1) Adult Services, (2) Dislocated Worker services, and 
(3) Services to Eligible Youth 19 to 21 years old in Youth programs 
under Section 129 of WIA.
     Youth Performance Measures and Definitions for Services to 
Eligible Youth 14 to 18 years old in the Youth Program.
     Customer Satisfaction Measures for Participants and 
Employers.
    While the specific core performance indicators and customer 
satisfaction indicators outlined in this paper only apply by law to 
Title I of WIA, DOL may adopt them, as appropriate, for other DOL 
programs, and will work in cooperation with other Federal partner 
agencies to reach agreement where feasible on uniform measures. Thus, 
all partners are encouraged to review and comment on this draft 
framework. However, any changes to other programs whether internal or 
external to DOL would require appropriate actions based upon present 
Laws, Regulations and/or policies.
    Definitions to be used by all States and localities are provided 
for each of the core indicators and customer satisfaction measures to 
ensure comparability. Comparability of measures among States is 
important for two reasons. First, core indicators and the customer 
satisfaction performance levels are to be negotiated between the States 
and DOL. One of the factors affecting those negotiations are ``how the 
levels compare with State-adjusted levels of performance established 
for other States * * *'' Second, comparability also contributes to 
continuous improvement. Having standard definitions will allow States 
and localities to benchmark other States and localities to promote 
continuous improvement. Comparability also will facilitate the sharing 
of best practices within and among the States. Since the performance 
and accountability system under WIA includes incentives and sanctions, 
comparability is important to fairness and equity.
    Continuous improvement is a significant and required element of 
WIA. States and localities need to collect more substantial data than 
the core measures or other required measures under the Act to function 
in a continuous improvement environment. Therefore, it is important for 
State and local leadership to take advantage of the opportunity when 
developing their performance systems to go beyond Federal requirements.

II. Adult/Dislocated Worker Services

A. Workforce Investment Act (WIA) Requirements/Program Activity 
Categories for Reporting

    The WIA provides for a continuum of service delivery that includes 
three levels of services: (1) core services; (2) intensive services; 
and (3) training services. All persons will have access to core 
employment-related information and self-service tools without 
restrictions or additional eligibility requirements, assuming 
sufficient funds are available. Those core services that are not 
primarily informational and must be staff-assisted will require WIA 
registration. Intensive services and training will also require WIA 
registration. The intensive services are provided when a determination 
is made that unemployed individuals are or would be unable to obtain 
employment after receiving the basic core services, or when employed 
individuals are determined to be in need of these intensive services to 
obtain or retain employment that allows for self-sufficiency. 
Similarly, training services are only available after a determination 
that the individual is unable to obtain or retain employment that leads 
to self-sufficiency through intensive services.
    For accountability purposes, WIA establishes core indicators of 
performance in State and local WIA financed systems for participants in 
all workforce investment activities other than self-service and 
informational activities. This exception recognizes the low cost per 
unit of providing these services. WIA also requires that States and 
localities report on how participants in workforce investment 
activities other than training (except for self-service and 
informational activities) compare to participants in training 
activities. Therefore, the level of service individuals receive defines 
whether the individual will be counted in the core indicators and if 
so, how they will be categorized for reporting purposes.
    Many of the core services generally will be low cost, self-service 
and consist primarily of information and not require registration. In 
contrast, intensive services will be more costly, require significantly 
more staff investment, and thus, justify a different measurement system 
that calls for registering and tracking individuals throughout their 
program participation. For reporting, services are divided into--
     Core Services (for registered participants)
     Intensive Services
     Training
    Consistent with WIA, participants who use one-stop self-service 
facilities or only access information do not need to be registered and 
tracked. Access to some Core Services will be universally available 
through the Internet, at a One-Stop center or through a One-Stop 
partner. States and local Boards will be free to allow completely 
anonymous access to core services that are primarily information and 
universally available (for example, browsing a job bank or using a 
computer in the resource room). However, States and local Boards may be 
encouraged to request unique identifying information about customers 
who use the Internet (for example, current America's Job Bank account) 
and who use the universally available self-service capacity in One-Stop 
centers.
    For customers who are assessed for purposes of determining whether 
they require services or training that are not universally available, 
additional information will need to be collected as part of the 
assessment process. This information will include demographic data 
elements such as racial-ethnic

[[Page 14338]]

characteristics, veteran status and information on disabilities, and 
will assist in referring individuals to other partners' services and 
will be available for comparisons of program applicants and 
registrants.

B. What Services Fall Into What Category?

    The categorization of services is a State or local decision and 
will depend on the nature of the service. To serve as a guide and to 
assist in this identification, Table 1 includes most of the core, 
intensive, and training services described in Section 134(d). Each of 
the required WIA services is italicized. Frequently provided services 
that are in addition to those required by the legislation are not 
italicized. Given the wide variation in types of service that can be 
categorized as job search and placement assistance, and career 
counseling, finer distinctions have been made for these services. As 
soon as a participant moves from the self-service/informational level 
of service to registered service (core, intensive, training) core 
measures apply.

                                        Table 1.--Categories of Services
----------------------------------------------------------------------------------------------------------------
                                                                   C. Intensive services        D. Training
A. Core services-- Self-service and     B. Other core services         (registration           (registration
            information                (registration required)           required)               required)
----------------------------------------------------------------------------------------------------------------
Determination of eligibility to      Follow-up services,          Comprehensive and       Occupational skills
 receive assistance.                  including counseling for     specialized             training.
                                      registrants (those           assessment, including
                                      previously receiving         diagnostic testing
                                      intensive/training           and interviewing.
                                      services) after entering
                                      employment.
Outreach, intake (which may include  Individual job development.  Development of          On the Job Training.
 profiling), and orientation to the                                individual employment
 One-Stop center.                                                  plan.
Initial assessment of skill levels,  Job clubs..................  Group counseling......  Workplace training and
 aptitudes, abilities, and support                                                         cooperative education
 service.                                                                                  programs.
Labor Market Information...........  Screened referrals (testing  Individual counseling   Private sector
                                      and background checks done   and career planning.    training programs.
                                      before referral or when
                                      operating as the employers
                                      agent).
Consumer reports information and     ...........................  Case management.......  Skill upgrading and
 delivery system performance                                                               retraining.
 information.
Information on other One-Stop        ...........................  Short term pre-         Entrepreneurial
 partner services and supportive                                   vocational services.    training.
 services.
Information on filing UI claims....  ...........................  ......................  Job readiness
                                                                                           training.
Assistance in establishing WtW       ...........................  ......................  Adult education and
 eligibility and other non-WIA                                                             literacy activities
 training and education.                                                                   in combination with
                                                                                           training.
Resource Room usage................  ...........................  ......................  Customized training.
``How to'' group sessions (e.g.
 writing a resume).
Job referrals (informational, e.g.,
 job scouts, ES referrals in non-
 exclusive hiring arrangements,
 short-term or seasonal
 placements).
Internet browsing--job, info, and
 training searches.
Internet accounts--Career Kit,
 Personnel Kit.
Talent referrals (informational,
 e.g., talent scouts, ES staff
 referrals of resumes without
 further screening).
----------------------------------------------------------------------------------------------------------------

    The following considerations provided some of the rationale used in 
preparing this guide. First, ``self-service and informational 
activities'' are by their nature core services that do not require 
registration and tracking. A second consideration is the likely per 
unit cost of services. A number of placement activities are primarily 
informational in nature and relatively low cost. In these instances, 
the added cost of registration and participant tracking may not be 
justifiable. Thirdly, some services benefit participants but are 
undertaken primarily for their value to employers (e.g., assistance 
with recruitment for seasonal work--summer or holiday) are intended 
only to provide short term employment and do not necessarily increase 
worker earnings, retention or occupational skill attainment. Fourthly, 
these groupings of activities are intended to be clean, easy to 
administer, and applicable to all programs.

[[Page 14339]]

C. How to Measure Core Services, Intensive Services and Training

1. Core Indicators of Performance

----------------------------------------------------------------------------------------------------------------
                       Measure                                                Definition
----------------------------------------------------------------------------------------------------------------
Entry into Unsubsidized Employment:
    Entered Employment Rate (Sec 136(b)(2)(A)(i)(I))  The rate will be defined for cohorts of registered
                                                       participants unemployed at the time of registration. The
                                                       numerator will be the number of these registered
                                                       participants that are shown to have paid employment in
                                                       the quarter following registration or service completion.
                                                       The denominator will be all registered participants
                                                       unemployed at the time of registration who were active
                                                       during the reporting period (received services or
                                                       continuing from a prior period) but who are no longer
                                                       actively receiving services, other than post-employment
                                                       services. This includes enrolled participants who (1)
                                                       have obtained unsubsidized employment; (2) have withdrawn
                                                       from participation; or (3) who have completed training or
                                                       services. Individuals should be considered no longer
                                                       active and to have completed service if they have
                                                       received no services in the last quarter of the reporting
                                                       period, and are not scheduled to receive services in the
                                                       future.
                                                      Note: State and local officials opposed using a program
                                                       exit or termination to trigger reporting.
                                                      Records of all registered participants unemployed at the
                                                       time of registration, and not enrolled in a training
                                                       program at the end of the reporting period would be drawn
                                                       and matched against wage records to identify employment.
                                                       Dislocated workers as defined in WIA, Title I, subtitle
                                                       A, sec.101 (9) are included in the definition of
                                                       unemployed. A person is considered employed if his/her
                                                       social security number appears in the employer wage
                                                       report (no minimal wage requirement) in the quarter
                                                       following the one in which the seed record for matching
                                                       is drawn.
                                                      Not all jobs are covered by State UI wage records.
                                                       Therefore, a State or locality may supplement the results
                                                       of the wage record review by other methods and count as
                                                       employed any of these individuals in jobs not covered by
                                                       the State's UI wage records. Again, employment would be
                                                       determined based on employment in the quarter following
                                                       the one in which the seed record is drawn. Supplementary
                                                       information could include: use of the New Hire index,
                                                       surveys, self-reported hires or staff-reported hires
                                                       through an administrative record system.
                                                      Seed records not shown as employed would be matched
                                                       against administrative records to exclude from the
                                                       computation individuals who remain active, i.e, have
                                                       received services in the last quarter or are scheduled to
                                                       receive them in the future.
Retention in unsubsidized employment Six Months
 after entry into the employment:
    Six Month Retention Rate (Sec                     The rate is computed using information on the total number
     136(b)(2)(A)(i)(II)).                             of registered participants who have employment and who
                                                       appear in the wage records, and wage record information
                                                       for the second quarter thereafter (6 month rate). For
                                                       example, an individual completing training and placed
                                                       immediately in the first quarter of the program year,
                                                       would be recorded as employed in the second quarter. The
                                                       fourth quarter records would be queried to determine
                                                       retention.
                                                      Note: retention is not limited to the same employer.
Earnings Received in Unsubsidized Employment Six
 Months After Entry into Employment:
    Average Earnings Change in Six months (Sec        The average earnings change is measured as follows: the
     136(b)(2)(A)(i)(III)).                            wage record earnings for the registered participant in
                                                       the two quarters following employment (not counting the
                                                       quarter in which employment was recorded) less 50% of the
                                                       wage record earnings for the four quarters prior to
                                                       enrollment (not counting the quarter of enrollment). The
                                                       post-employment income can be with the same or other
                                                       employer in which the placement was first noted. The
                                                       measure is reported as an average (mean) gain. For
                                                       Incumbent workers and others who are employed at the time
                                                       of registration average earnings and retention would be
                                                       computed begining in the second quarter following the
                                                       quarter in which services are completed.
Educational Credential/Occupational Skills
    Credential Attainment Rate (Training Services     For adults entering employment after training and eligible
     Only) (Sec 136(b)(2)(A)(i)(IV)).                  youth 19 through 21 entering employment, post-secondary
                                                       education or advanced training after training, the
                                                       percent who attained a State-recognized credential
                                                       related to educational skill attainment (diploma, degree
                                                       or certificate) or attainment of an occupational skill
                                                       (license or certification) recognized by a State or a
                                                       Nationally-recognized industry trade body. Information in
                                                       administrative records or information gathered through
                                                       electronic interfaces with other data bases available or
                                                       surveys may be used. Additional guidance on acceptable
                                                       credentials and certificates will be provided.
----------------------------------------------------------------------------------------------------------------

2. Why These Measures and Definitions?
    Wage record versus surveys. WIA requires that wage records be used 
but does not exclude the use of surveys to supplement information. 
Where the same information is obtainable from both sources, 
instructions will provide that wage records be used both to ensure 
comparability among States and to minimize costs.
    Earnings change versus absolute earnings level. A change--gain/
loss--measure was chosen because of the different circumstances within 
the WIA service population and the difficulty of creating an absolute 
earnings target for all adults.
    Earnings change--prior period earnings. Six month comparisons are 
made with a full twelve month period prior to registration. (Fifty 
percent is used for the six month change for comparability of periods.) 
The twelve month period is used to minimize, to the extent possible, 
the impact on dislocated workers of reduced earnings as layoffs 
approach and any severance pay that may be added to earnings.
    Retention rate. While most of the measures use the registered 
population as the base, this measure chooses only those who are 
employed because the retention question makes no sense in another 
context.

[[Page 14340]]

    Credentials rate (Training Service only). Adults entering 
employment after receipt of core or intensive services also may acquire 
some credential as a result of WIA participation. However, given the 
few relative to the number who will be employed after receiving 
services, the cost of this added information collection is not 
justified.

III. Youth 14 to 18 Years Old

A. WIA Requirements/Guiding Principles

    WIA authorizes programs to provide services to prepare youth to 
enter the workforce or to advance to postsecondary education or other 
occupational skills training. Programs link academic and occupational 
learning. Service providers will have strong ties to employers and must 
also include tutoring, study skills training and instruction leading to 
completion of secondary school (including dropout prevention). Other 
elements of programs should include alternative school services, 
mentoring by appropriate adults, paid and unpaid work experience (such 
as internships and job shadowing, occupational skills training, 
leadership development, and appropriate supportive services.) Youth 
participants also will receive guidance and counseling and follow-up 
services for at least one year. Programs must also provide summer job 
opportunities linked to academic and occupational learning. The mix of 
year-round and summer activities is left to local discretion.
    Eligible youth are low-income, ages 14 through 21 with barriers to 
employment. WIA specifies different youth core indicators for older 
youth ages 19 through 21 and for youth ages 14 through 18. The older 
youth performance measures are identical to the adult program measures 
and were addressed earlier in the paper. The three required core 
indicators for youth ages 14 through 18 are--
     Attainment of basic skills and, as appropriate, work 
readiness or occupational skills;
     Attainment of secondary school diplomas and their 
recognized equivalent;
     Placement and retention in postsecondary education or 
advanced training, or placement and retention in military service, 
employment, or qualified apprenticeships.

B. Guiding Principles

     Performance measures should reflect the same flexibility 
available for program design and services and the varied successful 
outcomes recognized. WIA allows a wide variety of services that are 
offered to youth. This provides the opportunity for local programs to 
design and operate programs that meet local needs and respond to gaps 
in services as they are identified for their area. Indicators should 
not force certain designs to remain competitive in terms of 
measurement.
     Performance measures should accommodate a variety of 
different approaches to serving youth without forcing arbitrary time 
limits or sequencing of services. Measures should not determine when 
youth begin or end services by forcing measurements before participants 
are ready to complete their goals or require that a youth leave the 
program before credit can be taken for outcomes achieved. Research has 
shown that programs that establish an ongoing relationship with youth, 
and continue to serve them for several years while adjusting goals and 
services to reflect needs as youth age have the greatest success.
     Performance measures need to recognize that youth goals 
change as youth mature and must be age appropriate. The denominator for 
various rates should depend on the appropriateness of the goal as 
determined through individual service plans. For example, younger youth 
participants should receive services that encourage staying in or 
returning to school and keeping up academically. As the participants 
get older, goals will change and relate to getting a secondary school 
diploma, and, ultimately, placement and retention in post-secondary 
education, advanced training or employment.
     Indicators must (1) recognize the differing goals 
depending on the activities/services; (2) the age of the youth; and (3) 
allow comparison between activities/services that include modest levels 
of summer job opportunities and those that invest a large proportion of 
resources during the summer. WIA provides for summer employment 
opportunities directly linked to academic and occupational learning, 
but intentionally did not provide a separate funding stream or 
differing set of indicators. In addition, the summer employment 
opportunities element is not intended to be a stand alone program (WIA 
sec. 129(c)(2)(C)). Indicators should be designed so that achieving 
every goal established for every youth--whether for a year-round or 
summer employment opportunities only--would result in a 100 percent 
rate.

C. Establishing a Basis for Performance Measurement

    State and local officials with whom DOL consulted strongly believed 
that the core indicators of performance for participants should reflect 
completion of their activities, not necessarily completion of 
participation. By design, youth programs are intended to provide a 
continuum of services for youth resulting in attainment of several 
interim outcomes such as the acquisition of basic skills, work 
readiness, or occupational skills, award of a secondary school degree, 
and then, placement and retention in employment or advanced education/
training. Participant goals are reflected in individual service 
strategies and will be different from one youth to another, depending 
on the needs and interests of the youth.
1. Core Indicators of Performance

----------------------------------------------------------------------------------------------------------------
                       Measure                                                Definition
----------------------------------------------------------------------------------------------------------------
Attainment of basic skills and, as appropriate, work
 readiness or occupational skills:
    Skill Attainment Rate (Sec 136(b)(2)(A)(ii)(I)..  A rate computed by dividing the number of youth who
                                                       attained a higher level of proficiency with regard to
                                                       basic skills, and, as appropriate, work readiness skills
                                                       or occupational skills by the number of youth receiving
                                                       services or training for whom attaining basic skills,
                                                       and, as appropriate, work readiness skills or
                                                       occupational skills were goals to be achieved during the
                                                       reporting period. Goals are based on individual
                                                       assessments using widely accepted and recognized
                                                       measurement/assessment techniques. (Outcomes are counted
                                                       as they are achieved, not when the youth completes
                                                       program participation).

[[Page 14341]]

 
Attainment of secondary school diplomas or
 recognized equivalents
    Diplomas or Equivalent Attainment Rate (Sec       A rate computed by dividing the number of youth who
     136(b)(2)(A)(ii)(II)).                            attained a secondary school diploma or equivalent divided
                                                       by the number of youth for whom attaining a diploma or
                                                       certificate was a goal to be achieved during the
                                                       reporting period. This goal will generally be appropriate
                                                       for older youth 16 or 18 years old.
Placement and retention in postsecondary education
 or advanced training, or placement and retention in
 military service, employment, or qualified
 apprenticeships:
    Retention Rate (Sec 136(b)(2)(A)(ii)(III))......  Of those who are receiving follow-up services and for whom
                                                       placement and retention is a goal, the percent with
                                                       retention status at 30 days, 90 days, 180 days and one
                                                       year from beginning follow-up. This overall rate would be
                                                       an average of measures for all four periods.
----------------------------------------------------------------------------------------------------------------

2. Age as the Basis for Outcomes.
    These younger youth reporting requirements are for participants who 
are 14 to 18 when enrolled in the youth program. They are appropriate 
as long as the youth continues to receive services identified in their 
individual service plans and has not attained the outcomes established 
for that age group. If the youth were going to continue being served by 
the youth program after achieving goals established prior to age 19, 
the youth does not need to be ``terminated'' but instead the record 
would indicate that the youth transferred to the age 19 to 21 youth 
program. This would have the same effect as a ``termination'' for 
performance measurement purposes but would not disrupt services to the 
youth. Individuals ages 19 through 21 who have completed the goals 
established for them prior to turning 19 and those who enter the youth 
program as age 19 or older should have individual service plans leading 
to the attainment of the goals appropriate for that age group. These 
outcomes would be measured in accordance with the same principles 
established for the adult program.

D. Why These Measures and Definitions?

    Skill Attainment Rate; Attainment of secondary school diplomas and 
their recognized equivalents: and Placement in postsecondary education 
or advanced training, or placement in military service, employment, or 
qualified apprenticeships. All three core indicators use as their 
denominators the total number of youth age 14 through 18 for whom the 
particular outcome to be measured was a goal. This recognizes that 
depending on the age of the youth and their needs, individual goals 
will differ significantly and that the effectiveness and quality of a 
program should be measured by its experience in achieving goals 
established for its participants. It allows comparisons of programs and 
allows for program goals to be established without having to know the 
exact mix of ages of participants and relative investments in different 
activities. It also permits a 100 percent rate if all goals are 
achieved.

    Note: A youth participating in WIA youth activities/services for 
multiple reporting periods (years), could be counted during each of 
the periods (years) if goals established in each are achieved.

    Placement Rate and Retention Rate. Unlike the adult program where 
separate measures are provided for placement and retention, WIA calls 
for a single youth indicator that includes both placement and retention 
rates in one measure. The retention rate--for unsubsidized employment 
and further education--was chosen as the core indicator because of the 
difficulty in coming up with a single measure that measures placement 
and retention. A meaningful measure with both would be difficult 
because of complications coming up with a single denominator. Placement 
will be measured, but outside the core indicators.
    Unsubsidized employment. WIA specifies that adults be measured with 
regard to placement in unsubsidized employment. The Youth indicator 
specifies only placement in employment. Unsubsidized employment is 
being used for consistency and because it, and not subsidized 
employment, will lead to self-sufficiency.

IV. Customer Satisfaction

A. Workforce Investment Act Requirements

    In addition to the core measures, WIA, at sec. 136(b)(2)(B), states 
that ``the customer satisfaction indicator of performance shall consist 
of customer satisfaction of employers and participants with services 
received from the workforce investment activities authorized under this 
subtitle.'' The Statute also requires, at sec. 136(b)(3)(A)(i), that 
there be State adjusted levels of performance for customer satisfaction 
and that ``the levels of performance established * * * shall, at a 
minimum--
    (I) Be expressed in an objective, quantifiable, and measurable 
form; and
    (II) Show the progress of the State toward continuously improving 
on performance.''
    The Act draws a link between the core measures and customer 
satisfaction by indicating that negotiations between DOL and the States 
must take into consideration ``* * * the extent to which the levels 
involved will assist the State in attaining a high level of customer 
satisfaction.'' WIA further suggests that ``customer satisfaction may 
be measured through surveys conducted after the conclusion of 
participation in workforce investment activities.''

B. Methods To Measure Customer Satisfaction

    Customer satisfaction measures are important because they provide 
valuable information from customers for strategic planning and program 
operation. Such feedback to supervisors and staff can motivate high 
performance and continuous improvement. They also send a clear message 
to staff, management, and customers themselves that customers matter.

[[Page 14342]]

    There are a number of different methods to collect customer 
satisfaction information. The simplest approach is to train staff to 
listen to the customers they serve and to ask questions that elicit 
customer needs while they are providing service. Focus groups and group 
interviews are another strategy. A trained manager or staff person can 
circulate in the resource center where people are waiting and ask 
questions informally to gain a better understanding of customer needs 
and concerns. Suggestion boxes are also a way of gathering information. 
As part of a comprehensive continuous improvement strategy, 
organizations generally use a combination of strategies since each 
serves a somewhat different purpose and provides different types of 
information.
    To meet the customer satisfaction requirements of WIA, the 
recommended measures focus on customer satisfaction surveys. This is 
the only method that allows State and National aggregation of 
comparable, quantifiable data. The proposed measures present a general 
framework for developing a National customer satisfaction index for 
different customers.
    An index is a single score that is created by combining the scores 
from several questions that address different dimensions of the 
customer experience. The customer satisfaction index will be described 
in more detail in the proposed measures. Essentially, the index would 
provide a way to capture common customer satisfaction information 
across programs that could be aggregated to a State and National level. 
The proposed measures will continue to be modified as DOL receives 
feedback and validation through consultation with the workforce 
investment system.

C. Proposed Customer Satisfaction Measurement Strategy

    WIA requires measures of customer satisfaction for both 
participants and employers that are quantifiable, comparable across 
States, and that, along with the core indicators of performance, 
promote continuous improvement.
    The basic approach for gathering and reporting customer 
satisfaction measures that meet these requirements will be as follows--
     There will be separate surveys of participants and 
employers;
     Customer satisfaction indicators will be derived from 
State or locally conducted surveys that include a few embedded 
questions that enable comparisons across States, while allowing each 
State to design its own instrument;
     Guidelines will be issued to the States requiring the 
States to validate survey methods to ensure comparability across 
States;
     Participants surveyed will include only registered 
individuals who have completed their activity/service participation 
(excluding follow-up services). This includes individuals who have 
completed services and are now employed, those who have discontinued 
receiving services or training, either because they have withdrawn or 
completed;
     Employer surveys can be based on a random sample of 
employers using the system; and
     Results on the embedded questions, for both employers and 
participants, will be compiled on the State level and reported 
annually.
    While the Act requires reporting and comparisons across States, the 
primary value of effective customer feedback is to drive strategic 
planning and continuous improvement at the local level. DOL plans to 
play a strong, proactive technical assistance role that will provide 
States and localities with easy access to the information, instruments, 
tools and other resources that will enable them to use this feedback as 
a springboard toward high performance and quality services.
    ETA will finance the design and development of customer 
satisfaction instruments and methodologies that meet requirements, and 
make these available to the States. However, States will have the 
option of designing their own instruments and methods as long as the 
few questions that enable comparisons are embedded in the State 
instruments and the methodologies used to collect responses on those 
questions are within guidelines that DOL will develop. These guidelines 
will include suggestions for establishing State baseline levels to be 
used where they do not already exist.

Attachment II--Reaching Agreement on State Adjusted Levels of 
Performance Under Title I of the Workforce Investment Act of 1998; 
A Consultation Paper

    This Consultation Paper provides a framework regarding the approach 
and process for reaching agreement on State adjusted levels of 
performance under Title I of the Workforce Investment Act of 1998 to be 
incorporated in States' five-year strategic plans required by that Act. 
Comments are solicited on the overall framework and the approach. The 
paper does not address the additional indicators a State may identify 
in the State plan in accordance with Title I, because they are not 
subject to the agreement process.

I. Introduction

A. Broad Legal Framework

    In Section 136, the Workforce Investment Act (WIA) establishes a 
comprehensive performance accountability system for the Statewide and 
local workforce investment systems. Its purpose is to assess the 
effectiveness of States and local areas in achieving continuous 
improvement of their Federally-funded workforce investment activities, 
in order to optimize the return on investment of Federal funds in those 
activities.
    As part of the performance accountability system, WIA requires the 
Secretary and the Governor of each State to reach agreement on the 
respective State performance levels for the core indicators of 
performance and the customer satisfaction indicator. WIA requires that 
the State performance levels be expressed in an objective, 
quantifiable, and measurable form. The law also requires that the 
levels show the State's progress toward continuously improving in 
performance. The negotiated performance levels become ``State adjusted 
levels of performance'' and must be incorporated into the State's 5-
year plan. They become the basis for sanctions for failed performance. 
Together with adjusted levels of performance for adult education 
literacy programs and performance levels for vocational education 
programs, they become the basis for incentive grants.
    State adjusted levels of performance for the first three program 
years covered by a State 5-year plan are agreed to during the plan 
review and approval process. In reaching agreement on those levels, the 
Secretary and the Governor must take into account the expected levels 
of performance identified by the State in its 5-year plan. The 
Secretary and the Governor also must take into account the following 
three factors: (1) the extent to which the levels will help the State 
achieve a high level of customer satisfaction; (2) how the levels 
compare to those of other States, with consideration of, at least, 
differences in economic conditions, participant characteristics at 
entry into the program, and services to be provided; and (3) the extent 
to which the levels promote continuous improvement in performance and 
ensure optimal return on investment. State adjusted levels of 
performance for the fourth and fifth program years covered by a State 
5-year plan must be agreed to before the beginning of the fourth 
program year.

[[Page 14343]]

The State plan is not required to identify expected levels of 
performance for those years. In reaching agreement, the Secretary and 
the Governor must take into account the same three factors listed 
above. The Governor may request revision of the agreed to performance 
levels if unanticipated new circumstances in the State result in 
significant change in any of factors for any year.

B. Guiding Principles

    WIA reflects the Nation's commitment to accountability for results 
in government programs that is expressed in the Government Performance 
and Results Act. It embodies strong commitment to partnership among 
government entities in serving people in areas of workforce 
development. It promotes flexibility to provide services that provide 
maximum benefit to customers and high levels of customer satisfaction. 
These commitments are characterized by key principles that must be 
reflected in the process of reaching agreement on target levels for 
State performance measures.
    1. Performance expectations for related workforce investment 
programs and for related reporting purposes (e.g., reporting under the 
Government Performance and Results Act) should be aligned to facilitate 
better management of overall performance.
    2. State and Federal partners share the commitment to high 
performance and customer satisfaction, and continuous improvement in 
both.
    3. The process and considerations in reaching agreement on State 
performance levels must be consistent for all the States so as to 
assure fairness in the derivation of performance levels.
    4. States should have maximum flexibility to target populations for 
WIA services within the parameters of the law, and to develop and adopt 
innovative methods for accomplishing workforce investment objectives; 
this flexibility should be reflected in agreed target levels of 
performance.
    5. States should have maximum flexibility to develop reasonable 
methodologies for setting performance goals in the core measurement 
areas and customer satisfaction.

C. Assumptions

    Critical assumptions underlie the discussion of the process for 
reaching State/Federal agreement on State adjusted levels of 
performance. These assumptions relate to performance measures, 
incentives and sanctions, performance data and tools, and continuous 
improvement.
    1. A paper on Performance Accountability Measurement for the 
Workforce Investment System is being circulated for comment. The 
measures described here are drawn from that paper, and so are subject 
to change. There are seven measurements for the core indicators of 
performance. Four of these apply separately to activities under three 
funding streams (adult, dislocated worker, and youth ages 19-22) and 
three apply to activities for youth ages 14-18, for a total of fifteen 
measures. In addition to those fifteen measures for the core indicators 
of performance, there are two measures for the customer satisfaction 
indicator. Each State plan will include one State adjusted performance 
level for each of the seventeen measures. The indicators and their 
measures, grouped by program, are:
 Adult Program (four indicators)
     Entry into unsubsidized employment, measured by Entered 
Employment Rate,
     Retention in unsubsidized employment after entry into the 
employment, measured by Six Month Retention Rate,
     Earnings received in unsubsidized employment six months 
after entry into employment, measured by Average Earnings Change in Six 
Months,
     Attainment of educational credential/occupational skills 
credential for adults entering employment after training, measured by 
Educational Credential/Occupational Skills Credential Attainment Rate 
(Training Services Only);
 Dislocated Workers Program
(Four indicators--same as for the Adult Program)

     Entry into unsubsidized employment, measured by Entered 
Employment Rate,
     Retention in unsubsidized employment after entry into the 
employment, measured by Six Month Retention Rate,
     Earnings received in unsubsidized employment six months 
after entry into employment, measured by Average Earnings Change in Six 
Months,
     Attainment of educational credential/occupational skills 
credential for adults entering employment after training, measured by 
Educational Credential/Occupational Skills Credential Attainment Rate 
(Training Services Only);
 Youth Ages 19-22 Program
(four indicators--same as for the Adult Program, with a variation in 
the credentials indicator)

     Entry into unsubsidized employment, measured by Entered 
Employment Rate,
     Retention in unsubsidized employment after entry into the 
employment, measured by Six Month Retention Rate,
     Earnings received in unsubsidized employment six months 
after entry into employment, measured by Average Earnings Change in Six 
Months,
     Attainment of educational credential/occupational skills 
credential for youth ages 19-22 entering post-secondary education, 
advanced training, or employment after training, measured by 
Educational Credential/Occupational Skills Credential Attainment Rate 
(Training Services Only),
Youth Ages 14-18 Program
(three indicators)

     Attainment of basic skills and, as appropriate, work 
readiness or occupational skills, measured by Skill Attainment Rate,
     Attainment of secondary school diplomas and their 
recognized equivalents, measured by Diplomas and Equivalents Attainment 
Rate,
     Placement and retention in postsecondary education or 
advanced training, or placement and retention in military service, 
employment, or qualified apprenticeships, measured by Placement Rate;
Customer Satisfaction for Combined Programs
(two indicators)

     Participant satisfaction, measured by an index derived 
from several questions on customer satisfaction surveys,
     Employer satisfaction, measured by an index derived from 
several questions on customer satisfaction surveys.
    2. The Department of Labor expects that negotiations will lead to 
high State adjusted levels of performance to encourage high 
performance.
    3. Incentives will be awarded and sanctions will apply based on 
performance against State adjusted levels of performance beginning in 
the first year of WIA implementation. Only States that exceed their 
agreed to levels will be eligible for incentive awards. Sanctions based 
on first year performance will not include monetary penalties.
    4. The Department of Labor will establish a single estimate of 
National average performance for each performance measure. Initially, 
because only limited historical outcome data are available to develop 
information on likely outcomes for WIA core

[[Page 14344]]

performance indicators, the information will be subject to serious 
qualifications. For some indicators, rough estimates of the 
distribution of performance among States and simple adjustment models 
will be developed as tools for reaching agreement on State adjusted 
levels of performance. For other core performance indicators and the 
customer satisfaction indicator, the available data will not be 
sufficient at the outset to develop such tools. The Department expects 
that future negotiations will be informed by discussions with and the 
actual performance of the early implementing States.
    5. A commitment to continuous improvement is not simply an 
agreement to raise the State adjusted levels of performance for 
successive years and incrementally improve performance numbers. 
Continuous improvement is the process of building dynamic, high 
achieving systems within every organization through the ongoing 
systematic improvement of products, programs, services, and processes 
by both small increments and major breakthroughs. Continuous 
improvement encompasses a commitment to a systematic approach to high 
performance. Continuous improvement is driven by finding opportunities 
to do better, as well as by solving problems that need immediate 
correction. It becomes a regular part of daily work, and provides a 
method of eliminating problems at their source. Performance measures 
and customer satisfaction are integrated into a continuous improvement 
approach to focus on where to concentrate resources, or redesign 
programs or sequences of services in order to achieve better results.

II. Principal Stages of Agreement Process

    There are three principal stages of the process for reaching 
agreement on State adjusted levels of performance in State five-year 
plans. These stages are defined in terms of plan submittal and 
approval. The first stage precedes submittal of the State plan and 
includes the Department's provision of information, the State's 
planning and development of an ``expected level of performance'' for 
each of the prescribed indicators of performance and customer 
satisfaction, and preliminary State/Federal discussions. The second 
stage begins with the State's submittal of its plan including the 
expected levels of performance that serve as the starting point for 
negotiations between the State and Federal partners, and ends when the 
State adjusted levels of performance are agreed to by the State 
Governor and the Secretary of Labor and incorporated into the State 
plan. The third stage follows approval of the plan with the agreed 
levels and encompasses possible modification of the plan to revise the 
State adjusted levels of performance.

A. Stage One: Federal and State Information and Preliminary Discussion

    Before any meaningful activity can be accomplished relative to 
planning or setting State performance goals, the State and Federal 
partners will need an understanding about the process and its 
relationship to other processes in the performance accountability 
system, e.g., reporting, incentive and sanction policies, GPRA goals, 
etc. Ideally, this information would become available before a State 
engages in its strategic planning process or sets its ``expected levels 
of performance'' and must be available before the negotiation process 
begins. Both the State and the Federal partners must gather and assess 
information prior to States' submittal of their plans.
1. Federal Information
    The Department expects to release information in the following 
areas at the earliest possible time. Each of these items will be 
covered in papers to be developed and issued for comment.
     Specific measures will be identified for the core 
performance and customer satisfaction indicators. Definitions will be 
provided for those measures, and will include the scope of the 
measures, e.g., are they only Title I, all WIA referenced activities, 
etc.
     Information will be provided about performance measurement 
tools under development, such as estimates of the distribution of 
performance among States and simple adjustment models. The information 
will include sources of data, the expected usefulness of information to 
be developed, and for which measures, if any, there will be State-
specific estimates or adjustment models.
     The process for negotiating the measures will be 
established and communicated to the system. This will include 
expectations for how and when the discussions will occur as well as the 
kinds of information that must be available from the State to 
facilitate the discussions.
     Guidance will be issued about the levels that will be 
considered acceptable when negotiating adjusted performance levels, 
including specific information on:

--Policies that set criteria for evaluating expected levels of 
performance;
--Policies for award of incentives and related concepts for meeting and 
exceeding WIA Title I performance goals (note that this does not 
include the overall policy for consideration of performance WIA Title 
II and Carl D. Perkins Act programs); and
--Policies defining sanctions and related definitions for failure to 
meet standards.

     Guidance will be issued on determining the impact of 
adjusted levels of performance on attaining high levels of customer 
satisfaction.
     Policy will be set to describe the consideration of 
continuous improvement in the goal setting process.
     Specific data and tools including models, if available, 
will be provided for comparing the adjusted levels of performance among 
States. Data and tools will continue to be released as they are 
developed.
2. State Information
    WIA envisions an accountability process that takes into 
consideration unique State and local requirements and circumstances. As 
States engage in the planning process and develop the expected levels 
of performance they will identify in their State plans, they will 
gather information that will be useful in the subsequent negotiation 
process. States will obtain preliminary information about the economy, 
anticipate characteristics of the population to be served, and set 
strategies for determining service mix, since these must be considered 
in setting performance levels. States will explore pertinent data 
sources related to sequenced services and one-stop service systems and 
examine their utility in establishing a baseline for the negotiations 
process. States will develop information gleaned from an environmental 
scan to determine the progress local areas have made in developing a 
service delivery system as required in WIA. States will consider the 
strength of State/local partnerships among agencies and organizations 
that will support the system and strategies under consideration to 
strengthen and streamline the delivery system. States are encouraged to 
take into account, in developing their expected levels of performance, 
the results of the negotiation of local performance levels.
3. Preliminary Discussion
    The Department recognizes, with the States, that time is short for 
development of State plans, including performance levels, particularly 
for those States that expect to implement WIA in July 1999. The 
Department appreciates and continues to encourage

[[Page 14345]]

State and local involvement as WIA policies and procedures are 
developed. In this spirit of cooperation, preliminary discussion of 
performance management, including the development of performance 
levels, is welcomed. States are encouraged to contact their Regional 
Offices for discussion and technical assistance prior to plan 
submittal. The benefits of early discussion could include:
     Ensuring understanding of guidance, policy, data, and 
technical material provided by the Department prior to plan submittal;
     Benefitting from the experience of regional staff and 
other States;
     Tailoring the provision of technical assistance on 
performance accountability to meet local and regional planning needs;
     Developing a mutual understanding of State and Federal 
expectations and assumptions prior to plan submittal, to ensure 
development of a shared set of goals;
     Allowing maximum time (in advance of the up-to-ninety-days 
plan review period) for States and local areas to complete necessary 
planning and consultation on performance levels; and
     Smoothing the agreement process.

B. Stage Two: Formal Discussion and Agreement

    A State's submittal of its five-year plan to the Secretary of Labor 
triggers the up-to-ninety-days review period during which the Federal 
and State partners are to reach agreement on the State adjusted levels 
of performance for the core and customer satisfaction indicators. The 
agreed to levels will be incorporated into the plan prior to its 
approval. A State's plan will not be approved if agreement has not been 
reached. It is expected that the negotiations will take place between 
the States and the Department's Regional Offices, consistent with 
guidelines to be issued by the Department to the workforce development 
system.
    The State plan will include the State's expected level of 
performance for each core indicator and customer satisfaction 
indicator. The plan will also include an explanation of the derivation 
of each expected level. In the formal negotiations, States should be 
prepared to provide support for any data and data analysis used in 
arriving at the expected levels of performance. States should also be 
prepared to discuss any environmental or strategic issues that are 
expected to influence performance levels.
    In addition to the expected levels of performance identified in the 
State plan for the first three years covered by the plan, WIA specifies 
three factors that the Governor and the Secretary must take into 
consideration in the agreement process. The first is the extent to 
which the State adjusted levels of performance will help the State 
achieve a high level of customer satisfaction. The second is how those 
levels compare to the adjusted levels of other States, considering 
differences in economic conditions, characteristics of participants 
upon entry into WIA programs, and services to be provided. The third 
factor is the extent to which the adjusted levels promote continuous 
improvement in performance and ensure optimal return on investment. As 
mentioned earlier, the Department expects to issue guidelines for 
consideration of all of these factors, to ensure consistency and 
fairness in the agreement process. The performance levels, representing 
the anticipated results of the comprehensive workforce development 
plan, will be considered in the context of the entire plan. 
Particularly because of the anticipated limitations of available data 
at the outset of WIA implementation, the negotiation process itself is 
expected to be a learning experience for the State and Federal 
partners.

C. Stage Three: Modifications, and Years Four and Five

    WIA specifies that Governors may request revisions to State 
adjusted levels of performance for any of the five program years 
included in a State plan, based on unanticipated circumstances in their 
respective States that result in significant changes in factors 
including economic conditions, characteristics of participants, and 
services provided. WIA does not prohibit consideration of other 
factors. The Secretary will issue guidelines establishing objective 
criteria and methods for making revisions requested by Governors. These 
guidelines also will specify the conditions under which a State is 
required to revise the agreed to levels of performance. The revision 
process will be addressed in a separate paper which is expected to be 
issued for comment in late April 1999.
    Because of the transitional nature of the program for the first 
three program years and the lack of data from which predictions of WIA 
performance can be derived for each State, there must be allowance for 
changes in expected performance beyond the circumstances specified in 
WIA. Allowance for changes in performance expectations is particularly 
important because a State's performance measured against its State 
adjusted levels of performance will affect its eligibility for 
incentive grants and its susceptibility to sanctions. As the body of 
WIA experience grows--over time in individual States and as more States 
implement WIA--more information will become available to permit 
development of more useful performance management tools, including 
National figures, State-specific information, distributions of 
performance data across States, and adjustment models. The effects of 
continuous improvement approaches will be better understood and more 
predictable as their application is tested. Because of these 
anticipated changes, it is expected that State adjusted performance 
levels included at the beginning of a State's five-year plan will be 
able to be refined as time passes.
    Federal guidance will delineate circumstances in which the State 
adjusted levels of performance must or may be revised--upward or 
downward--for individual States or for all States. Some possibilities 
beyond those identified in the law are listed here.
     Performance levels are set for all States based on the 
pre-WIA wage record experience of a few States, and experience shows 
that the predictions were not valid.
     The operation of the one-stop system in a State varies 
significantly from that discussed during performance negotiations.
     Changes in State law, Statewide vision, or strategies have 
a significant impact on performance outcomes.
     Changes in Federal law or policy have a significant impact 
on performance outcomes.
    WIA requires that agreement be reached on State adjusted levels of 
performance for the fourth and fifth program years covered by a five-
year plan prior to the beginning of the fourth year. The State does not 
submit expected levels of performance for those years. The law seems to 
contemplate that experience under WIA in the first three years will 
provide a sufficient basis for setting levels for subsequent 
performance.

[FR Doc. 99-7148 Filed 3-23-99; 8:45 am]
BILLING CODE 4510-30-P