[Federal Register Volume 64, Number 150 (Thursday, August 5, 1999)]
[Notices]
[Pages 42818-42822]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 99-20119]



[[Page 42817]]

_______________________________________________________________________

Part VI





Department of Labor





_______________________________________________________________________



Employment and Training Administration



_______________________________________________________________________



Consultation Papers on Performance Accountability Under Title I of the 
Workforce Investment Act of 1998 (WIA); Notice

  Federal Register / Vol. 64, No. 150 / Thursday, August 5, 1999 / 
Notices  

[[Page 42818]]



DEPARTMENT OF LABOR

Employment and Training Administration, Labor


Consultation Papers on Performance Accountability Under Title I 
of the Workforce Investment Act of 1998 (WIA)

AGENCY: Employment and Training Administration, Labor.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: The purpose of this notice is to disseminate consultation 
papers for interested parties on the performance accountability system 
for title I of the Workforce Investment Act. There are two papers. The 
first paper provides a framework regarding the approach and processes 
for continuous improvement under title I of the Workforce Investment 
Act of 1998. The second paper provides a framework for the approach and 
processes for customer satisfaction measures under title I of the 
Workforce Investment Act. Interested parties have 30 days to provide 
comments on these papers.

FOR FURTHER INFORMATION CONTACT: Mr. Eric Johnson, Workforce 
Implementation Taskforce Office, U.S. Department of Labor, 200 
Constitution Avenue, NW, Room S-5513, Washington, D.C., Telephone: 
(202) 219-0316 (voice) (This is not a toll free number), or 1-800-326-
2577 (TDD). Information may also be found, or comments provided, at the 
website--http://usworkforce.org.

SUPPLEMENTARY INFORMATION: The Workforce Investment Act (WIA or Act) 
Pub. L. 105-220 (August 7, 1998) provides the framework for a reformed 
national workforce investment system designed to meet the needs of the 
nation's employers, job seekers and those who want to further their 
careers. One of the key reforms contained in the Act is the 
establishment of a comprehensive accountability system to assess the 
effectiveness of State and local areas in providing employment and 
training services. The Act requires:
     A focus on results defined by core indicators of 
performance;
     Measures of customer satisfaction with programs and 
services;
     A strong emphasis on continuous improvement;
     Annual performance levels developed as a result of 
negotiations among Federal, State and local partners;
     Incentive awards and financial sanctions based on State 
performance; and
     Reporting and dissemination of performance results.

The two papers contained in this notice focus on two of these 
requirements--continuous improvement and customer satisfaction.
    The Department is approaching the development of this new 
performance accountability system on two tracks. First, definitions of 
the core measures of performance and temporary reporting instructions 
have been developed and disseminated for those States who are 
implementing WIA in Program Year (PY) 1999. Second, the Department is 
working with States and local governments to develop definitions and 
reporting requirements for use in PY 2000 and beyond. Part of this 
process will include using the lessons learned from the early 
implementing States and working with the Department of Education and 
other Federal agencies to develop common definitions for performance 
measures across programs. In general, the Department is considering PY 
1999 to be a transition year. The comments received on these two papers 
will be used in developing the performance accountability system for PY 
2000 and beyond.
    Comments are solicited on the overall framework and approaches 
being proposed for customer satisfaction and continuous improvement 
under title I of WIA.

    Signed at Washington, D.C., this 30th day of July 1999.
Raymond L. Bramucci,
Assistant Secretary of Labor, Employment and Training Administration.

Attachment 1--Continuous Improvement Under Title I of the Workforce 
Investment Act of 1998

I. Introduction

A. Legal Framework

    The Workforce Investment Act of 1998 envisions a high performance 
workforce investment system in this country--a system that is customer-
driven, results-oriented, flexible, and continuously improving. The 
Act's purpose is clearly stated as: To provide workforce investment 
activities that increase participants' employment, retention, earnings, 
and skill attainment and as a result:
     Improve the quality of the workforce;
     Reduce welfare dependency; and
     Enhance the productivity and competitiveness of the 
nation.
    The Act envisions a workforce investment system that strives for 
high performance rather than settling for compliance levels of 
performance, and that delivers unparalleled levels of services to 
customers--job seekers, workers, and employers. Although WIA has 
numerous references to continuous improvement, this consultation paper 
focuses on three major provisions contained in Section 136 of the 
legislation:
     A comprehensive performance accountability system will 
include an assessment of the effectiveness of state and local areas in 
achieving continuous improvement of workforce investment activities. 
Section 136(a).
     The Governor/Secretary agreement on State adjusted levels 
of performance must take into account the extent to which those levels 
promote continuous improvement in performance. Section 
136(b)(3)(A)(iv)(III).
     States must conduct ongoing evaluations of workforce 
investment activities to promote and implement methods for continuously 
improving them. Section 136(e)(1).

B. Guiding Principles

    The U.S. Department of Labor (DOL) is using the following as 
guiding principles in designing a system-wide approach for continuous 
improvement.
     DOL's role in continuous improvement is primarily one of 
leadership carried out through an effective technical assistance 
effort.
     For the workforce investment system to strive toward 
performance excellence, continuous improvement practices must be 
embraced at all levels--local, State, and DOL Regional and National 
Offices.
     DOL will integrate existing quality initiatives to drive 
continuous improvement through a technical assistance strategy that 
includes award and recognition efforts, access to information on best 
practices, and the availability of a variety of tools.
     The Malcolm Baldrige Criteria for Performance Excellence 
will be used as the framework for continuously improving performance in 
the workforce investment system.

C. Malcolm Baldrige Criteria

    The Malcolm Baldrige Criteria for Performance Excellence and the 
Baldrige Scoring Guidelines are proposed as the framework for enabling 
organizations within the workforce investment system to advance toward 
high performance. This framework is widely accepted as the standard for 
defining performance excellence in public and private organizations. 
The Criteria and Scoring Guidelines are excellent diagnostic 
instruments that can help leaders identify organizational strengths and 
key areas for improvement and work to achieve higher levels of 
performance. DOL will provide resources and technical assistance to

[[Page 42819]]

state and local organizations that are interested in using the Baldrige 
Criteria to help improve performance. The following is excerpted from 
the 1999 ``Criteria for Performance Excellence,'' and includes for 
informational purposes the relative point value assigned to each 
category:

------------------------------------------------------------------------
                                                                 Point
                        Award category                           value
------------------------------------------------------------------------
Leadership...................................................        125
Strategic Planning...........................................         85
Customer and Market Focus....................................         85
Information and Analysis.....................................         85
Human Resource Focus.........................................         85
Process Management...........................................         85
Business Results.............................................        450
                                                              ----------
      Total..................................................      1,000
------------------------------------------------------------------------

II. Approach to Continuous Improvement

A. Overview of the Approach

    According to leading Baldrige experts, continuous improvement is 
the systematic and ongoing improvement of products, programs, services 
and processes by small increments and major breakthroughs. Continuous 
improvement is the process of building dynamic, high achieving systems 
within every organization, and becomes embedded in the way the 
organization conducts its daily activities.
    DOL's role in the continuous improvement process is primarily based 
on providing leadership and technical assistance. In striving to 
improve performance as measured by the performance and customer 
satisfaction indicators, states and localities will need resources, 
information and technical assistance to help them continuously improve 
organizational effectiveness. The approach to continuous improvement 
proposed in this consultation paper envisions that DOL will play a 
strong, proactive role in providing States and localities with 
information, resources, tools, training and technical assistance to 
help them enhance their performance. DOL will also apply these tools to 
continuously improve the effectiveness of ETA National and Regional 
Offices.
    DOL's Continuous Improvement Strategy is aimed at improving 
outcomes for the customers of the workforce investment system by 
enhancing system-wide performance. The objectives of the strategy are 
to:
     Effectively align system-wide resources to achieve 
performance excellence.
     Recognize and award top performers within the system.
     Provide organizations and individuals with learning 
opportunities to acquire the skills needed to operate in a high 
performance mode.

B. Continuous Improvement in State Workforce Investment Plans

    A rigorous approach to continuous improvement must be applied at 
all levels of the workforce investment system in order for that system 
to achieve the high levels of performance envisioned in the Workforce 
Investment Act. For States to develop a Statewide workforce investment 
system that incorporates a rigorous approach to continuous improvement, 
each State needs to start with a snapshot or baseline of its system 
capacity--its ``as is'' capacity at the point in time when the State 
Plan is developed. Ideally, States would establish both an ``as is'' 
state for each organization's capability to become a high performance 
organization (organizational effectiveness), as well as the ``as is'' 
state for each organization's current program results and outcomes.
    In the spirit of partnership and shared accountability, State 
officials and DOL officials would have this data before them as the 
basis for establishing the baseline. From the State's perspective, the 
baseline or starting point for continuous improvement is simply defined 
as, ``where you are now.'' (This process has been further defined in 
the consultation paper on Reaching Agreement on State Adjusted Levels 
of Performance.)
    The State's continuous improvement strategy becomes its approach 
for closing the gap between the current ``as is'' capacity and a time-
sensitive ``desired state'' set forth in the State's plan. This 
approach addresses both the voluntary ``organizational improvement'' 
strategy and the more traditional compliance-oriented strategies for 
meeting minimum WIA specified performance measures. This offers states 
the opportunity to propose a rigorous and comprehensive approach to 
continuous improvement--one that establishes an effort to develop and 
improve organizational capacity (systems and processes) thus enabling 
committed organizations to deliver high performance, customer-focused 
services, as well as meeting all other requirements of the Act.

C. Voluntary Approach to Assessment and Benchmarking

    DOL's role is to make available to States the resources, tools and 
services that will help them advance toward high performance through a 
rigorous continuous improvement strategy. The basic tools and services 
would include organizational assessment tools, resources to aid in the 
development of improvement plans, best practices, and benchmarking for 
continuous improvement services. ``Benchmarking'' is the use of 
information and data on processes and results that represent best 
practices and the highest levels of performance.
    As part of its continuous improvement strategy, DOL would gather 
and make easily accessible to States and local organization benchmarks 
of the highest levels of performance both in processes and results 
within the workforce investment system, and for similar processes and 
results for organizations outside the system. Benchmarks represent the 
very essence of high performance business practices--comparing your 
organization to the very best in class and striving continuously to 
attain that level of performance. It is a voluntary practice carried 
out by the best organizations as a fundamental component of their 
continuous improvement strategy.

D. Supporting Continuous Improvement Activities

    Under WIA, States are to ensure that the principle of continuous 
improvement is embedded in Statewide workforce investment activities. 
Again, this would represent the regimen for achieving the systematic 
and ongoing improvement of workforce investment programs, services, and 
processes by small increments and major breakthroughs. This continuous 
improvement regimen will foster enhancements in performance levels 
desired by each level of the system.
    The State's Workforce Investment Plan must include a description of 
the State's strategy for developing and operating this continuous 
improvement approach. While each State has latitude to use a range of 
resources, tools and approaches for accomplishing this, the States are 
encouraged to work with the Employment and Training Administration 
(ETA) Regional Office to take advantage of resources available from or 
brokered through ETA's Continuous Improvement Strategy.
    Generally, DOL is seeking comment on the following strategy to 
support the local, State, Regional and National organizations in 
continuous improvement--
     Establish a system of organizational and individual 
learning to acquire skills needed to support high performance within 
the workforce investment system.
     Utilize the Malcolm Baldrige Criteria for Performance 
Excellence as a

[[Page 42820]]

proven and rigorous methodology to transform local, State, and National 
workforce investment organizations.
     Establish an award and recognition system in support of 
high performing organizations at all levels.
     Work closely with early implementing States as partners to 
begin the system-wide transformation process toward performance 
excellence.
     Provide local and State organizations, Regional Offices 
and National Office with easily accessible information on benchmarks 
and best practices, as well as affordable and effective assessment 
tools.

Attachment II: Customer Satisfaction Under Title I of the Workforce 
Investment Act of 1998

I. Introduction

A. Legal Framework

    In addition to the core measures, the Workforce Investment Act of 
1998 [WIA Section 136(b)(2)(B)] states that ``the customer satisfaction 
indicator of performance shall consist of customer satisfaction of 
employers and participants with services received from the workforce 
investment activities authorized under this subtitle.'' The Act 
[Section 136(b)(3)(A)(i)] also requires that there be State-adjusted 
levels of performance for customer satisfaction and that ``the levels 
of performance established * * * shall, at a minimum--
    (1) Be expressed in an objective, quantifiable, and measurable 
form; and
    (2) Show the progress of the State toward continuously improving in 
performance.''
    WIA draws a clear link between the core indicators of performance 
and customer satisfaction. The levels of performance attained for the 
core indicators must ``assist the State in attaining a high level of 
customer satisfaction'' (WIA Section 136(b)(3)(a)(iv)(I)). WIA further 
states that ``customer satisfaction may be measured through surveys 
conducted after the conclusion of participation in workforce investment 
activities'' (WIA Section 136(b)(2)(B)).
    Effective high performance organizations listen to their customers 
and build their organization around meeting their customers' 
expectations. Determining a customer's expectations and satisfaction is 
an integral part of a continuous improvement strategy. Under the 
Workforce Investment Act, customer satisfaction is both a process of 
identifying and listening to customers, as well as an outcome for 
measuring program success.
    WIA emphasizes the importance of a customer-driven workforce system 
by including customer satisfaction as a required measure, along with 
the core indicators of performance. Customer satisfaction measures 
provide feedback to supervisors and staff about how their actions 
affect customers, giving them critical information to motivate and 
guide continuous improvement. Customer satisfaction feedback also sends 
a clear message to staff, management, and customers that customers 
matter.

B. Guiding Principles

    DOL is using the following guiding principles in designing a 
system-wide approach for measurement of customer satisfaction:
     Customer satisfaction is the foundation of an 
organization's strategy for continuous improvement.
     Customer satisfaction should be measured after completion 
of the service and should be quantifiable.
     Customer satisfaction surveys need to contain a set of 
required questions to form a customer satisfaction indicator.
     Comparability is an important element in negotiating 
customer satisfaction performance levels and in providing opportunities 
for benchmarking and sharing best practices.
     States and local organizations are encouraged to add 
customized questions to inform their efforts to align resources or 
redesign processes to achieve better results.

II. Approach

A. Overview of the Approach

    The Act, in requiring a customer satisfaction indicator for 
employers and participants, presents a general framework for developing 
a national approach. Customer satisfaction indicators are a specific 
part of the performance accountability system and are the foundation of 
an organization's strategy for continuous improvement. They provide a 
guide to achieving the vision and goals of the Act, and provide a 
focused and structured process for listening to and learning from 
customers.
    To meet the customer satisfaction requirements for Title I, DOL 
proposes the use of customer satisfaction surveys. There are two 
purposes for surveying customers. The first is to produce an outcome 
measure for each State as part of the performance accountability 
system. This will be accomplished by a small set of required questions 
that will form a customer satisfaction index. The second purpose is to 
gain customer feedback to help in improving processes and services. 
This will be accomplished through a set of recommended questions 
addressing each service component and any additional questions that the 
State and local areas choose to ask, depending on their particular 
needs and service mix.
    DOL will provide guidelines for collecting customer satisfaction 
data that will lay out the strategy and standards (e.g., sample size, 
response rate) for implementing the survey while providing as much 
flexibility for the states and localities as possible. The survey will 
contain the required questions that form the indicator. In addition, to 
cover many of the most commonly delivered services, the guidelines will 
suggest sets of questions that States may choose to use along with the 
indicator questions. The advantage of using these questions will be 
that they provide additional opportunities for benchmarking and 
learning from the best practices of others.

B. Proposed Customer Satisfaction Strategy

    Consistent with the Workforce Investment Act, measures of customer 
satisfaction:
     Must address participants and employers;
     Must be quantifiable;
     Must be able to track progress toward improvement;
     Must be comparable across states;
     May be measured at the conclusion of participation; and
     Must promote continuous improvement in performance along 
with the core measures.
    The Act calls for assessment of two customer categories: (1) 
participants, and (2) employers. Consistent with the approach taken for 
core measures, two options are presented. The first option is to report 
the participant indicator for each of the four groups:
     Adults
     Dislocated Workers
     Youth 19-21 served with youth funds, and
     Youth 14-18.
    The second option is to aggregate the four groups to provide a 
single indicator of participant satisfaction.
    The advantages of reporting each of the four groups separately are 
to:
    1. Allow for a more comprehensive analysis of results. An analysis 
by group will provide an assessment of the degree to which core 
indicator performance contributes to customer satisfaction.
    2. Allow program managers to evaluate the degree to which they are 
satisfying different customer segments.
    The advantage of the second option is that it will simplify 
customer surveying

[[Page 42821]]

and reporting, and will emphasize high expectations for all of the 
groups. It should be noted that, under both options, the four groups 
identified above would also include those participating in incumbent 
worker training. Customer surveying for other services that are not 
covered under Title I would be at the discretion of other one-stop 
partner programs.
    For employers, it is proposed that services to employers be grouped 
into the following three service categories: (1) informational, (2) 
labor exchange, and (3) special services such as rapid response. 
Capturing customer satisfaction within each of these three categories 
will allow a clearer picture of service to employers and is one way to 
expand the system's ability to be accountable for services to a 
significant customer base. While States would not be required to report 
the three customer indicators for employers at a National level, they 
may utilize this method as a way to better understand their employer 
customers.

C. Collecting Customer Satisfaction Information

    There are a number of different methods to collect customer 
satisfaction information.
     The simplest approach is to train staff to listen to the 
customers they serve and to ask questions that elicit customer needs 
while they are providing service.
     Focus groups and group interviews are another strategy.
     A trained manager or staff person can circulate in the 
resource center where people are waiting and ask questions informally 
to gain a better understanding of customer needs and concerns.
     Suggestion boxes are also a way of gathering information.
     Telephone surveys of customers are used to gather specific 
information.
    To meet the WIA customer satisfaction requirements for Title I, the 
method proposed in this paper is customer satisfaction surveys. This is 
the most effective method that allows state and national aggregation of 
comparable, quantifiable data.
    As part of a comprehensive continuous improvement strategy, 
organizations will use a combination of strategies in addition to the 
proposed surveys, since each serves a somewhat different purpose and 
provides different types of information.

D. Proposed Measures

    The customer satisfaction indicator will be derived from surveys 
that must have a minimum set of common questions asked in a common 
format to assure comparability. These common questions are used to form 
an index, which is a single score. An index has the advantage of 
addressing different dimensions of the customer's experience, and is 
more reliable than a single question. The creation of an index provides 
a proven methodology to capture common customer satisfaction 
information across programs and organizations that can be aggregated to 
a State and National level. The responses of the embedded questions 
will be rolled up to the State level and reported annually at a 
specified time. This approach will continue to be modified as the 
Department receives feedback and validation through consultation with 
the workforce investment system.
    Satisfaction for all customers in all service categories will be 
measured through a set of 3-5 questions that together form the 
indicator. We propose that the surveys include these three questions:
     ``Overall, how satisfied were you with the services 
received?'' (Ranging from 1--Very Dissatisfied to 10--Very Satisfied)
     ``How likely would you be to refer others to these 
services?'' (Ranging from 1--Not Very Likely to 10--Very Likely)
     ``If you were in a similar situation again, how likely 
would you be to use these services?'' (Ranging from 1--Not Very Likely 
to 10--Very Likely)
    The above questions provide an indicator sensitive enough to record 
change but less prone to random fluctuations common to indicators that 
are composed of a single question. [This protects States from being 
sanctioned when random error depresses the indicator's performance 
level and prevents states from being rewarded for high performance 
resulting solely from random error.] The satisfaction score will be 
reported on a 0-100 scale. To simplify reporting to the Federal level, 
scores for each service category can be aggregated into two 
satisfaction indices, one for participants and one for employers.

E. Comparability Across States

    Comparability is important for several reasons. First, customer 
satisfaction performance levels are negotiated along with the core 
measures. One of the factors affecting those negotiations are ``how the 
levels compare with state adjusted levels of performance established 
for other States * * *.''
    Comparability also provides for fairness in determining incentives 
and sanctions. Additionally, comparability contributes to continuous 
improvement across the system. Having comparable measures will allow 
benchmarks to be developed to promote continuous improvement. 
Comparability will also facilitate the sharing of best practices within 
and among the States.

F. When To Measure

    Consistent with WIA, it is proposed that customer satisfaction be 
measured at completion of the service. For continuous improvement 
purposes, it is particularly important to measure customer satisfaction 
as close to the point of service for the following reasons:
     The immediacy of a person's impression makes a significant 
difference in terms of what he/she will remember;
     The highest response rate is obtained at point of service;
     Due to the time delay to track outcome-related data (e.g., 
the core indicators), this immediate customer feedback provides much 
needed real time data for staff and program managers.
    The point in time will vary based on the type of customer and level 
of service received.
Participant Customers
    For self-help/information and core services, the survey will be 
conducted at the point of contact, immediately after the service is 
provided. For intensive and training services, the participant will be 
surveyed after the completion of services (this does not mean 
necessarily that they have ``exited'' or been ``terminated'' from a 
program). Additional surveying may be conducted as part of follow-up to 
determine other aspects of satisfaction. Such surveys are proposed to 
be optional, given the additional reporting burden they would create.
Employer Customers
    For informational services, the survey will be conducted at the 
point of contact, immediately after the service is provided. For labor 
exchange and special services, the employer will be surveyed after the 
completion of services.

G. Using Customer Satisfaction in a Continuous Improvement Process

    The customer satisfaction indicators, in addition to being a 
specific part of the performance accountability system, are also the 
foundation of an organization's strategy for continuous improvement. 
The indicators provide a guide to achieving the vision and goals of the 
Act. Additional questions of local importance to customers, program

[[Page 42822]]

operators and service providers deepen the understanding of how to 
reach these goals.
    By adding customized questions, organizations can use customer 
satisfaction as part of an integrated continuous improvement approach. 
They can determine where to focus more resources, or redesign programs 
or sequences of services in order to achieve better results. This use 
of customer satisfaction will not be federally mandated in order to 
maintain local flexibility, and to recognize differing approaches in 
program designs that vary depending upon the service mix and each 
area's economic and demographic conditions.

H. Definition of Measures

Measurement of Participant Customers
    The degree to which participant customers are satisfied with the 
core, intensive and training services provided by the workforce 
investment system.
Measurement of Employer Customers
    The degree to which employer customers are satisfied with the 
informational, labor exchange, and special services provided by the 
workforce investment system.

I. Pilot Testing

    DOL will work with a number of pilot sites to better determine the 
range of customer satisfaction levels (i.e., baseline data), and to 
explore technical issues of survey timing, methodology, and 
questionnaire construction. The sites will be selected based on 
interest and previous experience with customer satisfaction surveys. 
DOL will use the results of the pilot testing and the feedback from 
this consultation paper to issue guidance or technical standards for 
the survey methodology.

[FR Doc. 99-20119 Filed 8-4-99; 8:45 am]
BILLING CODE 4510-30-P