Immigration Services: Better Contracting Practices Needed at Call
Centers (30-JUN-05, GAO-05-526).				 
                                                                 
The U.S. Citizenship and Immigration Services (USCIS) bureau	 
within the Department of Homeland Security (DHS) provides	 
toll-free telephone assistance through call centers to		 
immigrants, their attorneys, and others seeking information about
U.S. immigration services and benefits. As the volume of calls	 
increased--from about 13 million calls in fiscal year 2002 to	 
about 21 million calls in fiscal year 2004--questions were raised
about USCIS's ability to ensure the reliability and accuracy of  
the information provided at call centers run by an independent	 
contractor. This report analyzes: (1) the performance measures	 
established by USCIS to monitor and evaluate the performance of  
contractor-operated call centers; (2) how performance measures	 
were used to evaluate the contractor's performance; and (3) any  
actions USCIS has taken, or plans to take, to strengthen call	 
center operations.						 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-05-526 					        
    ACCNO:   A28580						        
  TITLE:     Immigration Services: Better Contracting Practices Needed
at Call Centers 						 
     DATE:   06/30/2005 
  SUBJECT:   Contract administration				 
	     Contract performance				 
	     Contractors					 
	     Customer service					 
	     Evaluation 					 
	     Evaluation methods 				 
	     Federal procurement				 
	     Immigration					 
	     Internal controls					 
	     Performance measures				 
	     Quality assurance					 
	     Service contracts					 
	     Standards						 
	     Performance-based contracting			 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-05-526

United States Government Accountability Office

GAO

                       Report to Congressional Requesters

June 2005

IMMIGRATION SERVICES

              Better Contracting Practices Needed at Call Centers

GAO-05-526

June 2005

IMMIGRATION SERVICES

Better Contracting Practices Needed at Call Centers

[IMG]

  What GAO Found

USCIS developed seven performance measures intended to assess the
performance and overall quality of responses provided by customer service
representatives at contractor-operated call centers. These measures
include how quickly calls were answered and the accuracy of information
provided. The contract between USCIS and its contractor stipulated that
the contractor could earn financial incentive awards if the average
monthly performance met or exceeded the standards on a quarterly basis at
each of four call centers. Conversely, financial deductions could be made
if the standards were not met.

USCIS did not finalize the terms regarding how the contractor's actual
performance would be calculated, or scored, before awarding the contract.
This limited USCIS's ability to exercise performance incentives (positive
or negative) because the parties could not reach agreement on performance
terms. USCIS suspended the use of financial incentives while the parties
negotiated the issue. Agreement was not reached after 16 months, however,
USCIS determined that the contractor had failed to meet standards for 4 of
the 7 performance measures in the fourth quarter of 2004 and took action
to reduce its payments for services. The contractor objected, citing the
lack of agreement on the performance measurements and the impact of
workload increases, but USCIS disagreed and stated it would reduce
payment. In a separate but related matter, USCIS failed to meet
contractual, regulatory, and GAO standards pertaining to how the
contractor's performance would be documented-especially with respect to
any deficiencies. Finally, USCIS exercised its option to extend the call
center contract through May 2006, to allow time to solicit and award new
call center contracts. USCIS said it intends to finalize performance
measurement terms in the new contracts.

USCIS used contractor performance data it collected over the course of the
contract to identify opportunities to improve customer service and call
flow, among other things. Several initiatives were launched as a result.

              USCIS Call Flow and Call Volume for Fiscal Year 2004

Source: Developed by GAO from USCIS data.

                 United States Government Accountability Office

Contents

  Letter

Results in Brief
Background
USCIS Established Performance Measures to Assess Contractor's

Call Center Performance

USCIS Evaluated Contractor's Performance but Suspended Use of
Financial Incentives for More Than 2 Years Due to Performance
Measurement Dispute

USCIS Has Used Results of Monitoring Efforts to Identify
Opportunities to Improve Customer Service and Call Flow at All
Call Centers

Conclusions
Recommendation for Executive Action
Agency Comments and Our Evaluation

                                       1

                                      2 4

                                       9

14

20 22 22 22

Appendix I Scope and Methodology

Appendix II	Criteria and Methodology Used by Contractor to Measure Quality
of Tier 1 Calls Monitored

Appendix III	Criteria and Methodology Used to Measure Customer
Satisfaction with Customer Service Representatives 28

Appendix IV	Criteria and Methodology Used by Independent Consulting Firm
to Measure Quality of Calls Monitored

Appendix V Mystery Shopper Scenario Example 30

  Appendix VI	Comments from the Department of Homeland
  Security 31

           Appendix VII Comments from Pearson Government Solutions 34

  Appendix VIII GAO Contact and Staff Acknowledgments

     Tables                                                           
                 Table 1: Call Quality Monitoring or "Soft Skills"         27 
                     Table 2: Accuracy of Information Provided             27 
                     Table 3: Accuracy of Capturing Information            27 

  Figures

Figure 1: Organization of USCIS's National Customer Service Center and
Call Centers 5

Figure 2: Tier 1 Customer Service Representatives Processing
Immigration-Related Calls at a Contractor-Operated Call Center 6

Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004 7

Abbreviations

USCIS U.S. Citizenship and Immigration Services
COTR contracting officer's technical representatives
CSR customer service representative
DHS Department of Homeland Security
FAR Federal Acquisition Regulation
IIO immigration information officer
INS Immigration and Naturalization Service
NCSC National Customer Service Center
OFPP Office of Federal Procurement Policy
PRS Performance Requirements Summary
VA Department of Veterans Affairs

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office Washington, DC 20548

June 30, 2005

Congressional Requesters:

The U.S. Citizenship and Immigration Services (USCIS) bureau within the
Department of Homeland Security (DHS) provides telephone assistance to
customers calling about U.S. immigration services and benefits. USCIS
customers-immigrants and their attorneys or family members, employers, and
others-can call toll-free for information on such topics as how to become
U.S. citizens, obtain work visas, initiate the naturalization process,
obtain legal forms, and determine the status of applications for benefits.
This telephone assistance service helps customers avoid making visits to
USCIS offices, which could require customers to take time off from work
and travel long distances. Telephone calls for assistance to USCIS have
increased in recent years, from about 13 million in fiscal year 2002 to
about 21 million in fiscal year 2004. These calls are handled in a variety
of ways: by an interactive voice response system;1 by four call centers
managed by a private contractor (Pearson Government Solutions) engaged by
USCIS; and by two call centers operated directly by USCIS.

You expressed interest in how USCIS monitors and evaluates
contractoroperated call centers to ensure that they operate reliably and
provide accurate information to their growing number of customers. This
report addresses the following: (1) What performance measures did USCIS
establish to monitor and evaluate the performance of contractor-operated
call centers? (2) How were these performance measures used to evaluate the
contractor's performance? (3) What actions, if any, did USCIS take or plan
to take to strengthen call center operations?

To address these objectives, we obtained and analyzed information from
USCIS officials in Washington, D.C., from contractor officials in
Arlington, Virginia, and at one of four contractor-operated call centers.2
At these locations, we interviewed officials and collected and analyzed
pertinent documentation, including descriptions of monitoring and
evaluation programs, contract requirements for call center operations, and
summaries of call center evaluation results. We assessed the reliability
of

1An automated voice system with a touch-tone menu. 2The contractor
considers the locations of its call centers to be proprietary information.

  Results in Brief

telephone call volume data provided to USCIS by a telecommunications
vendor, as well as USCIS and contractor staffing data. To carry out our
data reliability assessments, we (1) reviewed information about the data,
systems that produced the data, and data quality control procedures, and
(2) interviewed USCIS and contractor officials knowledgeable about the
data as necessary. We determined that these data were sufficiently
reliable for purposes of this report.

We conducted our work between May 2004 and May 2005 in accordance with
generally accepted auditing standards. (See app. I for details about our
scope and methodology.)

USCIS developed seven performance measures that were intended to assess
the performance of and overall quality of responses provided by customer
service representatives at contractor-operated call centers. These
measures were described in a contract signed by both parties in January
2002. Two performance measures, for example, were to assess the accuracy
of information provided to callers and how quickly calls were answered.
Based on provisions in the contract, the contractor could earn financial
incentive awards if certain standards or goals set for each performance
measure were met or exceeded on a quarterly basis at each of the call
centers-for instance, if callers waited an average of 30 seconds to 36
seconds for their calls to be answered. Conversely, the contract provided
for negative financial incentives (penalties), whereby USCIS could deduct
from payments owed to the contractor on a quarterly basis if the standards
were not met.

USCIS collected data on the contractor's performance on a monthly basis
since June 2002, when the contract was awarded. However, these data were
not used for the purpose of applying financial incentives (either positive
or negative) until the fourth quarter of 2004. The use of financial
incentives was suspended because USCIS awarded the contract without
finalizing how the contractor's actual performance would be calculated, or
scored, for the various performance measures. The two parties negotiated
on this issue over a period of about 16 months after the contract was
awarded before abandoning the effort. Thereafter, USCIS determined that,
for the fourth quarter of 2004, the contractor had failed to meet
standards for four of the seven performance measures, and announced its
intention to reduce payments to the contractor for services. The
contractor objected to USCIS's decision on the grounds that the
performance measurements had not been finalized and that changes in call
center workloads affected the basis for applying financial incentives.
USCIS disagreed and stated it

would reduce payment to the contractor. Although this dispute had not been
resolved, USCIS exercised its option to extend the current call center
contract through May 2006 to allow time to solicit and award new call
center contracts. The exercise of this option has no effect on the
contract's performance measurement terms, which is the source of the
parties' disagreement. In a separate but related matter, USCIS failed to
meet contractual, regulatory, and GAO standards pertaining to how the
contractor's performance was documented-especially with respect to any
deficiencies. For example, USCIS's failure to obtain the contractor's
written acknowledgement of USCIS-identified performance deficiencies did
not meet the notification procedures established by the contract,
documentation requirements of the Federal Acquisition Regulations, and
GAO's standards for internal control. Consequently, USCIS did not maintain
a complete and reliable record of the contractor's performance needed to
ensure accountability.

USCIS used the contractor performance data it collected monthly over the
course of the contract to identify opportunities to improve customer
service, help better respond to customer inquiries, and manage the flow of
calls into call centers. Several initiatives have been launched as a
result. For example, USCIS has implemented new call-routing procedures
designed, in part, to route customers' calls to service representatives
more quickly and reduce delays (on-hold time) that arise when call volume
is heavy. These and other initiatives were undertaken by USCIS concurrent
with its efforts to negotiate contract differences with the contractor. It
is too early to assess the impact of these initiatives.

To improve USCIS's efforts for evaluating contractor performance and
encourage quality services at call centers, we recommend that the
Secretary of Homeland Security require the Director of USCIS take the
following two actions: (1) finalize contract terms related to specific
performance measurement requirements, before awarding new
performance-based call center contracts; and (2) maintain readily
available written records of performance assessments and performance
evaluation meetings with the contractor.

DHS and the contractor provided formal comments and technical comments on
a draft of this report, which we have incorporated, as appropriate. DHS
generally agreed with our recommendations. The contractor said the report
accurately summarizes the complex nature of CIS's call center program and
several challenges created by significant post-award changes to that
program.

Background 	The Homeland Security Act of 2002 established USCIS within
DHS.3 USCIS is responsible for several functions transferred on March 1,
2003, from the former Immigration Services Division of the Immigration and
Naturalization Service (INS) under the Department of Justice.4 These
functions include providing services or benefits to facilitate entry,
residence, employment, and naturalization of legal immigrants; processing
applications for U.S. citizenship/naturalization; and rendering decisions
about immigration-related matters.

                             Call Center Operations

The USCIS Information & Customer Service Division is responsible for
operating the National Customer Service Center (NCSC), which was
established in 1997 to provide nationwide assistance by telephone to
customers calling about immigration services and benefits. When a customer
calls the NCSC toll-free number (1-800-375-5283), the call is received by
the interactive voice response system. The system features automated,
self-service options 24-hours a day, 7 days a week. If the system cannot
address a customer's concerns or needs or if a customer requests live
assistance, then the call is generally routed to one of the four NCSC
contract call centers, known as Tier 1.5 These four centers are operated
by the contractor, Pearson. If a question posed by a customer is
particularly complex or otherwise cannot be answered at the Tier 1 level,
the call is transferred to one of the two USCIS-operated call centers,
known as Tier 2.6 Figure 1 shows the organization of NCSC, including the
call centers.

3Pub. L. No. 107-296, S: 451 (Nov. 25, 2002).

4Throughout this report, we use USCIS to refer to both the new USCIS and
the former Immigration Services Division of INS.

5If a customer attempts to obtain live assistance outside normal business
hours, the customer is told when to call back to obtain the assistance.

6In some cases, customers' calls are routed directly from the interactive
voice response system to Tier 2, bypassing Tier 1.

Figure 1: Organization of USCIS's National Customer Service Center and
Call Centers

Source: Developed by GAO from USCIS data.

In fiscal year 2004, almost half of 21.1 million calls made to NCSC were
handled and completed by the interactive voice response system and the
rest were generally routed to Tier 1. Customer service representatives
(CSR) at Tier 1 respond to inquiries in English or Spanish. The CSRs focus
primarily on providing administrative information to customers by using a
series of scripts provided by USCIS. For example, if a customer needs what
USCIS considers basic information, such as USCIS local offices' hours of
operations, eligibility requirements, and procedures to follow, such
questions are to be answered by CSRs at Tier 1 call centers using specific
scripts. In addition, CSRs are to refer customers to USCIS service centers
and local offices, for such things as changes of address and appointment
scheduling at USCIS application support centers. (Some of these tasks may
alternatively be performed by customers through the USCIS Web
site-www.uscis.gov.) As of April 2005, the four Tier 1 call

centers employed over 450 CSRs.7 Figure 2 shows CSRs processing calls at a
Tier 1 call center.

Figure 2: Tier 1 Customer Service Representatives Processing
Immigration-Related Calls at a Contractor-Operated Call Center

Source: USCIS contractor.

At the two USCIS-operated Tier 2 call centers, calls are handled by
immigration information officers (IIO)-immigration specialists with
indepth knowledge of immigration laws, non-immigrant visas,
naturalization, asylum and refugee status, and other related policies and
procedures. As of April 2005, the Tier 2 call centers operated by USCIS
had 111 IIOs. About 5 percent, or about 590,000, of the calls going to
Tier 1 CSRs were

7The contractor considers the precise number of CSRs at its call centers
and their locations to be proprietary information.

rerouted to Tier 2 IIOs in fiscal year 2004. Figure 3 shows the call
volume handled by the interactive voice response system, Tier 1 call
centers, and Tier 2 call centers during fiscal year 2004.

         Figure 3: USCIS Call Flow and Call Volume for Fiscal Year 2004

                   Source: Developed by GAO from USCIS data.

a"Completed calls" refer to calls that have ended, not necessarily to
calls where customers' questions or issues have been resolved.

    Performance-Based Service Contract

In January 2002, USCIS awarded a performance-based service contract for
the management of four Tier 1 call centers.8 In making this award, USCIS
obtained acquisition services from the Department of Veterans Affairs
(VA), and the contracting officer who signed and was responsible for
administering the contract was a VA employee working on behalf of USCIS.9
The contracting officer's technical representative (COTR), a USCIS
employee, was also responsible for administering the contract. In
commenting on a draft of this report, DHS noted that by agreement of both
the VA and USCIS, on April 20, 2005, USCIS assumed responsibility for
administering the contract. The contract was awarded for a base year,

8The term "contract" in this report refers specifically to "Delivery Order
No. 591," which was awarded to the contractor following a competitive
procedure through the General Services Administration's Federal Supply
Schedule.

9 The contract was awarded before DHS was established while what is now
USCIS was part of INS.

beginning on June 1, 2002, plus 4 option years (1-year renewable
extensions of the contract, three of which have been exercised as of June
2005). Through calendar year 2004, USCIS paid $64.6 million to the
contractor for the Tier 1 call center operations.

According to the Office of Federal Procurement Policy (OFPP) in the Office
of Management and Budget, performance-based service contracts are designed
to focus on results.10 Their purpose is to ensure that contractors are
given the freedom to determine how to meet the government's performance
objectives, that appropriate quality levels are achieved, and that payment
is made only for services that meet these levels. This type of contract is
to emphasize standards for customer service and measurement of performance
and may offer financial incentives, both positive and negative, to
encourage quality performance. According to OFPP, call centers are suited
to this type of contract because, among other things, they emphasize
achieving results by meeting customer service standards. According to
OFPP, with performance-based service contracts, incentive payments made to
an independent contractor are to be contingent on the contractor's ability
to meet the government's performance standards; the contract does not
specify how those standards are to be met. Thus, the contractor retains
discretion in determining how to meet performance standards specified in
the contract, for example, how many CSRs to hire to ensure calls are
answered within a contractually specified time. Other elements suggested
for using a performance-based service contract include (1) identifying the
agency's needs and addressing those needs with performance requirements
that describe required service results; (2) establishing performance
standards that describe the required performance level; and (3)
establishing a quality assurance plan for assessing contractor performance
in order to ensure that the contractor has performed in accordance with
the standards.

10OFPP and the National Performance Review coordinated a federal
interagency working group to develop generic performance-based concepts
for government agencies to use in the procurement of call center services.
The group developed a reference source to help program and acquisition
officials draft performance requirements and standards related to call
center performance-based contracts: Performance Based Concepts for
Telephone Call Center Contracting, Office of Federal Procurement Policy
(May 27, 1997).

  USCIS Established Performance Measures to Assess Contractor's Call Center
  Performance

USCIS used a multi-faceted approach to monitor and evaluate the quality of
information and service provided by CSRs to customers calling
contractor-operated Tier 1 call centers.11 This approach used seven
performance measures. USCIS obtained performance data from the
contractor's monitoring of selected telephone calls; customer satisfaction
surveys; and a telecommunications vendor (telephone company).12 In
addition, USCIS used an independent consulting firm to monitor CSRs'
telephone calls and conduct a "mystery shopper" program assessing CSRs'
responses to customers.

    USCIS Set Call Center Performance Measures for the Contractor to Meet

In order to monitor and evaluate the performance of the four
contractoroperated Tier 1 call centers, USCIS planned to use seven
performance measures. These measures were to evaluate the quality of
customers' telephone interactions with CSRs; the accuracy of information
provided to callers over the telephone; the accuracy of callers'
information recorded by CSRs; callers' levels of satisfaction; how quickly
CSRs handled calls (two measures); and the number of calls abandoned by
customers put on hold. According to USCIS officials, USCIS established the
performance measures based on a review of industry standards for both
government and private-sector call center operations. The measures were
described in a section of the contract called the Performance Requirements
Summary (PRS).

Under the PRS, these performance measures comprised one of three
components upon which the contractor's performance score was based. The
other two components were the standard, or goal, set for each measure,
identifying the performance levels the contractor was expected to meet
(e.g., callers will wait an average of 30 to 36 seconds before their calls
are answered), and the performance calculation that USCIS would use to
analyze performance data (e.g., total delay of all calls divided by the
total number of calls).13 The PRS listing of the seven performance
measures included a "sample calculation" for each of the measures, and

11USCIS monitors and evaluates operations at both Tier 1 and Tier 2 call
centers. This report focuses on USCIS's evaluation of Tier 1
contractor-operated call centers.

12This telecommunications vendor is also responsible for providing the
interactive voice response system and routing customers' telephone calls
from this system to Tier 1 and Tier 2 call centers.

13In this report, we collectively refer to these three components as
"performance measurement requirements."

stated that "actual calculations [are] to be determined during Contract
negotiations."14 USCIS officials said they intended to negotiate and
finalize the calculations after a 4-to 6-month phase-in period, and the
contract was awarded with this provisional language.

As to the performance measures and their related standards or goals, three
of USCIS's performance measures are call quality monitoring, accuracy of
information provided, and accuracy of capturing information. Data on these
measures are to be collected by the contractor's quality assurance staff,
who are to randomly monitor two calls per day for each CSR. (CSRs are not
to know when they are being monitored.) The data collected are to be
reported to USCIS on a monthly basis. Details on these three measures
follow:

o  	Call quality monitoring. Calls are to be monitored by the contractor's
quality assurance staff to assess the CSRs' "soft skills," that is, their
ability to interact with customers, establish customer rapport, maintain
composure during a call, speak with clarity and professionalism, and other
factors. Call quality monitoring data are to be captured on a standardized
form. CSR responses for each of nine different "soft skills" are scored as
percentages, with scores for the most highly valued skills, such as
"active listening"-that is, whether the CSR was deemed to be attentive
when listening to the customers-given more weight than the scores for
other skills. The nine scores (i.e., percentages) are combined for a total
"soft skills" score, with 100 percent as the highest possible score. The
performance standard stated in the PRS for this measure is that all calls
monitored achieve an average score of 90 percent to 95 percent after the
nine "soft skills" scores (i.e., percentages) for each call are combined.
(See app. II for additional details on the criteria and methodology used
to determine soft skills scores.)

o  	Accuracy of information provided. Calls are to be monitored by the
contractor's quality assurance staff to determine, among other things,
whether CSRs provided accurate and complete responses. Using a

14The "sample" terms of the contract were never finalized, leading to a
dispute between USCIS and the contractor that remains unresolved (see p.
15). Examples of "sample calculations" listed for different measures in
the PRS were "total delay of all calls divided by total number of calls"
for determining the average length of time it takes before calls are
answered, and "total number of points achieved divided by total number of
points available for all quality monitoring scores" for determining call
quality. One sample calculation was listed for each measure.

standardized form, the staff score CSRs on five different efforts, such as
whether the CSR used software tools appropriately and whether, when the
callers were asked directly, they indicated that their needs had been
satisfied. The five efforts are scored as percentages, with more weight
given to the scores for certain efforts, such as "provides complete
response." The scores are then combined for a total "accuracy of
information provided" score, with 100 percent as the highest possible
score. The performance standard stated in the PRS for this measure is that
all calls monitored achieve an average score of 95 percent to 97 percent
after the five accuracy scores (i.e., percentages) for each call are
combined. (See app. II for additional details on the criteria and
methodology used to determine accuracy of information provided.)

o  	Accuracy of capturing information. Calls are to be monitored by the
contractor's quality assurance staff to determine, among other things,
whether CSRs accurately record and verify the callers' information. The
staff assess this measure by scoring four efforts, including whether a
referral to a local USCIS service center or local office was completed
appropriately and correctly. The four efforts are scored as percentages,
with more weight given to the scores for certain efforts, such as
"verifies caller's information." The scores are then combined for a total
"accuracy of information provided" score, with 100 percent as the highest
possible score. The performance standard stated in the PRS for this
measure is that all calls monitored achieve an average score of 95 percent
to 97 percent after the four accuracy scores (i.e., percentages) for each
call are combined. (See app. II for additional details on the criteria and
methodology used to determine accuracy of capturing information.)

A fourth performance measure of call quality-customer satisfaction- was
assessed by an independent consulting firm. Customer satisfaction surveys
were conducted on a monthly basis to determine if customers were satisfied
with the service that CSRs provided.15 At least 375 callers are to be
randomly selected to be interviewed each month from a population of 10,000
randomly identified callers who called within the 30 days prior to the
survey. To measure satisfaction with CSRs, customer responses to four
interview questions about CSRs are compiled, and the overall percentage of
respondents indicating satisfaction is calculated. The

15The customer satisfaction surveys ask about customer satisfaction with
the interactive voice response system and IIOs, as well as CSRs.

performance standard stated in the PRS for this measure is 80 percent to
85 percent of the customers surveyed indicating overall satisfaction with
the CSRs' service. (See app. III for additional details on the criteria
and methodology used to determine customer satisfaction.)

Three other performance measures involve the collection of statistical
data by the telecommunications vendor for determining how quickly calls
are answered. The performance measures and standards in the contract for
assessing how quickly CSRs answered customers' calls are as follows:

o  	Service level. The telecommunications vendor under contract with USCIS
is to collect information on the number of calls answered by CSRs in 20
seconds or less, that is, the number of callers who spoke to a CSR within
20 seconds after getting through the interactive voice response system.
The performance standard stated in the PRS for this measure involves two
factors: half-hour increments and the length of time it took CSRs to
answer calls. The standard is that for 80 percent to 85 percent of the
half-hour increments measured, 80 percent of the calls are to be answered
in 20 seconds or less.

o  	Average speed of answer. The telecommunications vendor under contract
with USCIS is to collect information on the length of time it takes for
CSRs to answer customers' calls after they are routed to Tier 1 by the
interactive voice response system; that is, how long callers are on hold
before a CSR answers their call. The performance standard stated in the
PRS for this measure is that, for all calls routed to Tier 1, callers will
wait an average of 30 seconds to 36 seconds.

o  	Abandoned calls. The telecommunications vendor under contract with
USCIS is to collect information on the number of calls abandoned by
customers after getting through the interactive voice response system and
waiting for a CSR to answer, that is, the number of times that customers
hang up the telephone while waiting for a CSR. The performance standard
stated in the PRS for this measure involves two factors: half-hour
increments and how frequently callers abandon their calls. The standard is
that for 85 percent to 95 percent of the half-hour increments measured, 1
percent to 2 percent of the calls are expected to be abandoned before a
CSR answers.

The contract stated that the contractor would be eligible to earn
financial incentive awards if the average monthly performance met or
exceeded the

standards on a quarterly basis at each call center, and allowed USCIS to
make deductions from payments to the contractor if the average monthly
performance fell below the standards.16 According to the contract, the
contractor is not eligible for an incentive award for a particular quarter
if one of the performance standards is not met by one call center, and
USCIS may make a deduction from payments to the contractor in that case.
In addition, USCIS may, at its sole option, elect to include or waive
financial incentives as it deems appropriate.

                               USCIS Carried Out
                               Additional Quality
                               Assurance Efforts

In addition to the performance data collected by the contractor's own
quality assurance staff, an independent consulting firm, and the
telecommunications vendor, USCIS took two additional steps to measure call
center performance for quality assurance purposes. First, to help ensure
that the contractor's scoring of call-quality performance measures was
reliable, USCIS used another independent consulting firm to validate the
results of the contractor's efforts by monitoring two calls per month for
each CSR. Data were gathered and provided to USCIS on a monthly basis.
(See app. IV for additional details on the criteria and methodology used
by the independent consulting firm to conduct call monitoring.)

Second, in April 2003, USCIS engaged the same independent consulting firm
to carry out a "mystery shopper" program to assess the completeness and
accuracy of CSRs' answers to callers. Under this program, an independent
consultant places random calls-1,200 each month-to Tier 1 call centers
using various scripts provided by USCIS. As of April 2005, the scripts
used in these calls covered 32 different scenarios, or types of calls, and
100 new scenarios were being developed. The calls are conducted in English
and Spanish. (See app. V for an example of a mystery shopper scenario.)

16Up to 10 percent of payments can be added to or deducted from the
contractor's billed amounts, depending upon whether or not the performance
standards are met.

  USCIS Evaluated Contractor's Performance but Suspended Use of Financial
  Incentives for More Than 2 Years Due to Performance Measurement Dispute

USCIS did not reach agreement with the contractor on how to apply the
performance measurement requirements described in the PRS before awarding
the performance-based service contract. USCIS suspended all financial
incentives, positive or negative, while the parties negotiated this issue
over a period of about 16 months without reaching agreement. After
negotiations were abandoned, USCIS determined that, for the fourth quarter
of 2004, the contractor had failed to meet four of seven performance
measures and merited a payment deduction. The contractor disagreed on the
grounds that the performance measurements had not been finalized and that
changes in call center workloads affected the basis for applying financial
incentives. In a separate matter, USCIS failed to ensure that all
contractual, regulatory, and GAO standards pertaining to the documentation
of the contractor's performance were fulfilled.

    USCIS Awarded the Call Center Contract without Finalizing Performance
    Measurement Requirements

The performance measurement requirements described in the PRS were not
completely finalized before the contract was awarded. The language
referring to "sample" calculations for determining how performance would
be measured remained in the contract after it was in force. In commenting
on a draft of this report, DHS said that at the time of the contract
award, USCIS management believed it was appropriate to let the winning
vendor have some input into the performance measurement methodology since
this contract represented a transition to performance-based contracting
for call center operations.

The negotiations between USCIS and the contractor on this issue began in
January 2003 (after a phase-in period) and continued intermittently until
April 2004, when they were abandoned. While negotiations were taking place
and after they were abandoned, USCIS obtained monthly data relating to the
contractor's performance on the seven performance measures and compared
those data to the standards.17 USCIS considered the measures and standards
themselves to be nonnegotiable; the contractor, on the other hand,
considered them as part of the "sample calculations" and, thus,
negotiable. For over 2 years, USCIS did not use any of the resulting
performance scores for the purpose of calculating financial incentive
awards or payment deductions under the contract because the terms of the
PRS remained unresolved between the parties.

17USCIS started collecting data on the contractor's performance in June
2002, the first month of the contractor's operations. We are not
presenting the performance data used by USCIS because the performance
measurement terms of the PRS were unresolved between USCIS and the
contractor as of April 2005.

The contractor maintained that the performance scores were "potential
scores" and were to be used by the parties in reaching an agreement on how
to structure the PRS.

On September 1, 2004, the contracting officer, representing USCIS, sent a
letter to the contractor advising that USCIS would begin evaluating the
contractor's performance and determining a financial incentive award or
payment deduction for the fourth quarter of the calendar year (October 1
through December 31). USCIS officials told us they decided to take this
action because they had concluded that negotiations with the contractor
were unlikely to result in an agreement on the PRS. The contractor
objected to USCIS's decision to carry out this evaluation. By letter dated
November 29, 2004, the contractor stated that, under the terms of the
contract, USCIS could not unilaterally determine the performance
measurement requirements because all aspects of the requirements were
negotiable, including the performance standards.

The contractor further stated that an evaluation of its performance must
take into account certain changes that took place to the work required
under the contract. For example, the contractor stated that the number of
USCIS-provided scripts, containing information for CSRs to address
callers' inquiries, had grown to more than 2,300 pages from approximately
400 script pages in June 2002.18 According to the contractor, these
changes significantly increased the average amount of time needed to
handle a call and affected the contractor's ability to meet the
performance standards imposed by USCIS. According to the contractor,
USCIS's unilateral imposition of performance measurement requirements that
did not account for the changed work requirements was inconsistent with
Federal Acquisition Regulation (FAR) 16.402(g), which provides that "[i]t
is essential that the Government and contractor agree explicitly on the
effect that contract changes (e.g., pursuant to the Changes clause) will
have on performance incentives."

Nevertheless, by letter dated February 11, 2005, USCIS's contracting
officer notified the contractor of the evaluation results for the period
of October through December 2004. The results showed that the contractor
met the standards for three of the seven performance measures and did

18USCIS revised the scripts to provide new and updated immigration-related
information on a variety of topics. In its technical comments on a draft
of this report, DHS said that the contractor had been compensated for the
increased script content and average call duration.

not meet the standards for the other four measures. USCIS determined that,
as a result of this performance, payments due to the contractor for
services would be reduced. The letter noted that the contractor could
submit its own data regarding performance during this period. Following
the review of any data submitted, USCIS would take action to make the
appropriate payment deduction, waive the payment deduction, or pay an
appropriate incentive award.

The contractor requested, by letter dated February 25, 2005, that USCIS
waive implementation of the financial incentives, both positive and
negative. The contractor reiterated its position that USCIS's unilateral
implementation of the performance measurement requirements as currently
written in the contract, without sufficient regard for substantial changes
to the contract and the changing nature of the program, was not
appropriate. The contractor stated that it was ready to resume
negotiations on this subject so that fair and equitable financial
incentives would be established. The contractor further stated that it had
determined the payment deduction was incorrectly calculated by USCIS.

USCIS's contracting officer responded, by letter dated April 15, 2005,
that the government would not agree to waive implementation of the
financial incentives and a deduction would be made from the next payment
to the contractor. The letter stated that USCIS did not unilaterally
create and impose the performance measurement requirements, which were
included in the negotiated contract that USCIS and the contractor agreed
to. Regarding the contractor's assertion that the average amount of time
needed to handle calls had significantly increased, the letter noted that
the performance measurement requirements would apply regardless of the
average length of calls at any given time.

According to FAR and OFPP guidance19 on performance-based service
contracting, the precise method for measuring performance should have been
agreed upon between USCIS and the contractor before the contract was
signed and implemented. FAR S: 16.401 states that performance-based
service contracts should establish "reasonable and attainable
[performance] targets that are clearly communicated to the contractor."
According to OFPP, performance measurement techniques (i.e., how
performance will be assessed to determine whether standards have been

19Performance Based Concepts for Telephone Call Center Contracting, Office
of Federal Procurement Policy (May 27, 1997).

met) are essential elements of performance-based service contracting and
should be clearly stated. In addition, according to OFPP, performancebased
service contracts emphasize that all aspects of an acquisition be
structured around the purpose of the work to be performed, that
appropriate performance quality levels are achieved, that payment is made
only for services that meet these levels, and that financial incentives
are awarded to encourage quality performance.

    USCIS Plans to Solicit New Tier 1 Call Center Contracts with Changes to
    Improve Performance

Although the disagreement between the two parties had not been resolved,
USCIS exercised its option to extend the current call center contract for
another year through May 31, 2006, to allow time to solicit and award new
call center contracts.20 The exercise of this option has no effect on the
contract's performance measurement terms, which is the source of the
parties' dispute. USCIS officials said they plan to award new
performancebased service contracts for Tier 1 operations to two vendors,
with the two vendors fully operational by June 2006, to improve the
handling of

21

customers' calls to Tier 1.

USCIS officials told us they intend for the new contracts to include
certain changes meant to improve Tier 1 call center operations and to
incorporate OFPP guidance on performance-based contracting. USCIS
officials told us that, unlike the current contract, the new PRS will
clearly specify how contractor performance will be assessed and will not
leave any terms open for post-award negotiation. In addition, USCIS
officials said the new contracts will include independent call monitoring
and the mystery shopper program as performance measurement tools to assess
the quality of the Tier 1 CSRs' responses to customers, including the
accuracy and reliability of the information provided. At the time of our
review, USCIS officials said that the solicitation was going through DHS's
contract review process and DHS had not issued the solicitation for a new
contract containing these changes. DHS said in its comments on a draft of
this report that the solicitation was with the DHS Procurement Office for
review and issuance.

20USCIS officials said that DHS will conduct the acquisition and USCIS
will administer the contract, without seeking inter-agency assistance from
the VA as it previously did with the current call center contract.

21The current contractor may compete for the new contracts.

    USCIS and Contractor Did Not Fully Document Quality Assurance Activities

As part of its quality assurance responsibilities under the current
contract, USCIS is to keep written records of observations about the
contractor's performance based on periodic evaluations comparing
performance data to standards in the PRS. USCIS's contracting officer's
technical representative (COTR), who is responsible for administering the
contract, is to use these written observations to notify the contractor if
there are deficiencies-specifically, if the contractor does not meet the
performance standards. The contractor is required to sign and date such
observations to acknowledge that the COTR apprised it of any deficiencies.

USCIS and contractor officials said they met at least quarterly (monthly,
since October 2004) to discuss performance, performance data, and other
items. USCIS officials said they provided the contractor with
documentation containing performance and other data to discuss at these
meetings. USCIS officials said some of this documentation identified
performance deficiencies. However, contractor officials said they viewed
the performance data as "potential scores" to be considered during
negotiations. To the extent that USCIS considered the performance data as
notification of deficiencies, it did not follow contractual procedures
requiring the COTR to obtain the contractor's signature acknowledging
notification of the deficiencies. In addition, neither USCIS nor the
contractor kept minutes of these meetings.

According to FAR S: 46.104(c), the government should maintain, as part of
the performance records of a contract, suitable records reflecting the
nature of its contract quality assurance actions. With respect to any
performance deficiencies, the government's records should include, among
other things, the number and type of defects observed and any actions to
correct deficiencies. Further, according to GAO's standards for internal
control in the federal government, for an agency to run and control its
operations, it must have relevant, reliable information relating to
internal events.22 All transactions and other significant events need to
be clearly documented, and the documentation should be readily available
for examination. This information should be recorded and communicated to
management and others within the agency who need it to carry out their
responsibilities. GAO's standards provide a framework for establishing and
maintaining internal control and for identifying and addressing major

22GAO, Standards for Internal Control in the Federal Government,
GAO/AIMD-00-21.3.1 (Washington, D.C.: November 1999); Internal Control
Management and Evaluation Tool, GAO-01-1008G (Washington, D.C.: August
2001).

performance challenges. Appropriate and effective internal control is a
key factor in helping agencies better achieve program results.

The contract also requires the contractor to provide a quality assurance
plan. The plan that was developed by the contractor describes the
contractor's approach and strategy for ensuring the delivery of
high-quality service. As part of the plan, the contractor is to conduct
formal, biweekly internal performance review meetings to help with the
identification and correction of performance deficiencies. These meetings
are to be attended by contractor and USCIS officials, with contractor
staff reporting on quality performance issues, and are to be in addition
to the quarterly (now monthly) meetings discussed above. Minutes of the
meetings are to identify action items, responsibilities, and solution time
frames, and the minutes are to be published for USCIS review. However, a
contractor official said that these meetings never took place. According
to both contractor and USCIS officials, the quarterly meetings were used
to discuss operations and performance and to focus senior management
attention on any performance issues.

Under FAR S: 37.602-2, the government's quality assurance surveillance
plans should include actions to help ensure that the contractor carries
out its quality control obligations. By failing to ensure that the
contractor held and documented performance review meetings as required by
the contractor's quality assurance plan, USCIS did not meet its quality
assurance obligations under FAR S: 37.602-2 and GAO's internal control
standards. In addition, USCIS's failure to obtain the contractor's written
acknowledgment of USCIS-identified performance deficiencies did not meet
the notification procedures established by the contract, documentation
requirements of FAR S: 46.104(c), and GAO's standards for internal
control.

In its comments on a draft of this report, DHS noted that the contract was
administered by a component of the VA until April 20, 2005, and DHS said
the VA was provided with documentation discussed at quarterly and monthly
performance assessment meetings between USCIS and the contractor.
According to DHS, the lack of a clear understanding between USCIS and the
VA regarding their roles contributed to the fact that formal documentation
and evaluations were not always properly maintained and formally
transmitted to the contractor. DHS acknowledged that the agency procuring
a service is ultimately responsible for the contract and, thus, USCIS
should have clarified its and the VA's roles.

  USCIS Has Used Results of Monitoring Efforts to Identify Opportunities to
  Improve Customer Service and Call Flow at All Call Centers

USCIS used contractor performance data, including the results of surveys,
call monitoring, and the mystery shopper program, to identify
opportunities to improve customer service, including improving
callresponse times, help CSRs and IIOs better respond to customer
inquiries, and manage the flow of calls into call centers. Following are
examples of initiatives that USCIS recently implemented, or was planning
to implement, as of April 2005. It is too early to assess the impact of
these initiatives.

o  	USCIS implemented "intelligent call routing" for Tier 1. With
"intelligent call routing" in place since February 2004, telephone calls
routed from the interactive voice response system to Tier 1 are now routed
to the next available CSR, at any of the four Tier 1 contractoroperated
call centers. Previously, telephone calls to Tier 1 were routed to the
next available CSR in the call center that resided in the same geographic
region of the country as the caller. USCIS officials said that by June
2005 they will also have implemented "intelligent call routing" for calls
transferred from Tier 1 to the next available IIO at either of the two
Tier 2 call centers.

o  	USCIS implemented "overflow routing." USCIS started its "overflow
routing" initiative in October 2004, enabling certain general call types,
identified as "English Other" and "Spanish Other," to be routed directly
from the interactive voice response system to Tier 2 USCIS-operated call
centers, bypassing the Tier 1 contractor-operated call centers.
Previously, all calls handled by IIOs at Tier 2 were first routed from the
interactive voice response system to Tier 1, where CSRs then transferred
the calls to Tier 2. USCIS officials said they expect the change will
result in 1 to 5 percent of all calls being routed directly to Tier 2,
which should help when Tier 1 CSRs cannot handle the call volume.

o  	USCIS implemented interactive voice response system routing of certain
telephone calls to USCIS service centers. USCIS changed its automated
interactive voice response system in December 2004 so that certain types
of customers' telephone calls-for example, certain issues concerning new
permanent residents, cases already approved or denied, and pending
cases-are now routed directly to USCIS service centers, bypassing CSRs at
Tier 1. Previously, all customers' telephone calls that needed to be
handled by USCIS service centers were routed by the interactive voice
response system to Tier 1. Then, after talking with the customers, the
CSRs referred the customers to the service centers (whose employees have
access to case paperwork) via e-mail. CSRs were allowed to transfer
customers' telephone calls to service

center personnel only when customers requested emergency and expedited
handling of applications.

o  	USCIS implemented a portfolio management system. Private attorneys,
paralegals, and other representatives can use the USCIS Internet Web site
to check the status of their clients' immigration cases using a USCIS
receipt number. Under the system, USCIS also notifies the representatives
via e-mail when a case status changes; for example, when actions are
taken, such as the approval or denial of an application. As of April 2005,
over 300,000 customers, attorneys, and other representatives had used this
system.

o  	USCIS said it is planning to implement a referral management system.
Currently, Tier 1 CSRs send, via e-mail, service request referrals to
USCIS service centers and local offices for customers who call wanting to
change addresses, schedule and reschedule appointments at application
support centers, order forms, and resolve problems. After a referral is
made, NCSC does not know whether the service center or local office
responded to the customer in a timely manner or even responded at all. To
better monitor this process, USCIS plans to implement a referral
management system, with such service request referrals placed in a
database and assigned a tracking number. The system is to (1) determine
the proper service center or local office to process the referral, (2)
assign the case to an adjudicator, (3) update the case on a daily basis,
and (4) report once a month on case status. The referral management system
is planned to be accessible to customers on USCIS's Internet Web site so
they can make and track their own service request referrals. In addition,
customers without Internet access are to be able to call on the telephone
and CSR's will access the USCIS Web site and create referrals for them.
USCIS plans that the referral management system will be fully operational
during the summer of 2005.

o  	USCIS is planning a customer service portal on USCIS's Web site. USCIS
has a long-term goal of giving customers Internet access to information
contained in the "scripts" used by Tier 1 CSRs to answer customers'
questions. USCIS plans to establish a customer service portal on the USCIS
Internet Web site, providing access to the information. The goal is to let
customers with Internet access look up information themselves without
having to call NCSC on the telephone, navigate the interactive voice
response system, and wait for CSR's to answer. USCIS had not set a time
frame for implementing this initiative.

Conclusions 	Immigration call centers are a vital information referral
source used millions of times by immigrants and other interested parties
seeking to obtain needed documents, regulatory information, up-to-date
status information on immigration-related benefits and applications, and
other information. To ensure that it serves its customers effectively and
efficiently, USCIS appropriately used a performance-based contract, but
its failure to finalize all aspects of the performance requirements before
the contract was awarded hampered its ability to exercise performance
incentives in the contract. As a result, USCIS lost the opportunity during
the life of the contract to help ensure that it received the maximum level
of service from the contractor.

In addition, USCIS did not meet standards promulgated by federal
acquisition regulations, GAO, and the contract itself pertaining to
documenting the contractor's performance between 2002 and 2004, and
adequately documenting notification of the contractor when the government
perceived deficiencies in its performance. Failure to generate adequate
documentation could impair USCIS's ability to conduct future contract
negotiations and to preserve a complete and reliable record of contract
performance needed to ensure accountability.

  Recommendation for Executive Action

To improve USCIS's efforts for evaluating contractor performance and
encourage quality services at call centers, we recommend that the
Secretary of Homeland Security require the Director of USCIS take the
following two actions: (1) finalize contract terms related to specific
performance measurement requirements before awarding new performance-based
call center contracts; and (2) maintain readily available written records
of performance assessments and performance evaluation meetings with the
contractor.

                                Agency Comments
                               and Our Evaluation

DHS and the contractor provided formal comments and technical comments on
a draft of this report, which we have incorporated, as appropriate. In its
formal comments, DHS generally agreed with our recommendations. DHS said
the draft solicitation for the new contracts specifically identifies
performance requirements that are non-negotiable. DHS further stated that,
as recommended by GAO, written records of performance assessments and
performance evaluation meetings will be maintained and readily available
for review by all interested parties. In its formal comments, the
contractor provided additional language to further clarify this report.
The contractor said the report accurately summarizes the complex nature of
CIS's call center program and several challenges

created by significant post-award changes to that program. DHS's and the
contractor's formal comments are shown in appendixes VI and VII,
respectively.

We are sending copies to the Director of USCIS and other interested
parties. We will also make copies available to others upon request. In
addition, the report will be available at no charge on GAO's Web site at
http://www.gao.gov.

If you or your staff have any questions regarding this report, please
contact me at 202-512-8777 or [email protected]. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on the
last page of this report. Key contributors to this report are listed in
appendix VIII.

Paul L. Jones, Director
Homeland Security and Justice Issues

List of Requesters

The Honorable John Conyers, Jr.
Ranking Minority Member
Committee on the Judiciary
House of Representatives

The Honorable Bennie G. Thompson
Ranking Minority Member
Committee on Homeland Security
House of Representatives

The Honorable Sheila Jackson Lee
Ranking Minority Member, Subcommittee on Immigration, Border

Security, and Claims
Committee on the Judiciary
House of Representatives

The Honorable Loretta Sanchez
Ranking Minority Member, Subcommittee on Economic Security,

Infrastructure Protection, and Cybersecurity
Committee on Homeland Security
House of Representatives

The Honorable Janice D. Schakowsky
House of Representatives

                       Appendix I: Scope and Methodology

To determine what performance measures the U.S. Citizenship and
Immigration Services (USCIS) established to monitor and evaluate the
performance of contractor-operated call centers, we interviewed USCIS
headquarters officials in Washington, D.C.; Tier 1 contractor officials in
Arlington, Virginia, and at a contractor-operated call center;1 and an
official representing an independent consulting firm under contract to
USCIS and located in Fairfax, Virginia. We also collected and analyzed
pertinent USCIS and contractor documentation. We collected and analyzed
information on the various types of monitoring and evaluation programs
used by USCIS, including internal call monitoring, independent call
monitoring, customer satisfaction surveys, the mystery shopper program,
and telephone call data provided by a telecommunications vendor.

To find out how USCIS used the performance measures to evaluate the
contractor's performance, we interviewed USCIS headquarters officials in
Washington, D.C., and Tier 1 contractor officials in Arlington, Virginia.
We also collected and analyzed pertinent USCIS and contractor
documentation.

To determine what actions, if any, USCIS took or planned to take to
strengthen call center operations, we interviewed USCIS headquarters
officials in Washington, D.C., and Tier 1 contractor officials in
Arlington, Virginia, and at a contractor-operated call center. We also
collected and analyzed pertinent USCIS and contractor documentation.

We assessed the reliability of telephone call volume data provided to
USCIS by a telecommunications vendor, as well as USCIS and contractor
staffing data. To carry out our data reliability assessments, we (1)
reviewed information about the data, systems that produced the data, and
data quality control procedures, and (2) interviewed USCIS and contractor
officials knowledgeable about the data as necessary. We determined that
the call volume and staffing data were sufficiently reliable for the
purposes of this report.

1The contractor considers the locations of its call centers to be
proprietary information.

Appendix I: Scope and Methodology

We conducted our work between May 2004 and May 2005 in accordance with
generally accepted auditing standards.

Appendix II: Criteria and Methodology Used by Contractor to Measure Quality of
Tier 1 Calls Monitored

Table 1: Call Quality Monitoring or "Soft Skills"

                                            Scoring range       Section score 
                             Criteria             (0 - 3)           (Percent) 
                    Establish rapport                   3 
                   Maintain composure                   3 
                 Conversational style                   3 
                     Active listening                   3 
                  Efficient call flow                   3 
                              Opening                   3 
                         Call on hold                   3 
                     Call transferred                   3 
                              Closing                   3 
                    Percent compliant                                    100% 

                           Source: USCIS contractor.

                   Table 2: Accuracy of Information Provided

                                               Scoring range    Section score 
                                     Criteria         (0 - 3)       (Percent) 
            Uses software tools appropriately               3 
                   Provides accurate response               3 
                   Provides complete response               3 

            Provides USCIS content only            3                   9.0 
             Satisfies caller's needs              3                  27.0 
                 Percent compliant                                    100% 
             Source: USCIS contractor.                     

Table 3: Accuracy of Capturing Information

                                            Scoring range   Section score 
                    Criteria                       (0 - 3)      (Percent) 
             Completes referral when                       
                   appropriate                           3           12.0 
       Completes referral record correctly               3           28.0 
          Captures caller's information                  3           30.0 
          Verifies caller's information                  3           30.0 
                Percent compliant                                    100% 

Source: USCIS contractor.

Appendix III: Criteria and Methodology Used to Measure Customer Satisfaction
with Customer Service Representatives

USCIS and an independent consulting firm jointly developed a telephone
survey to measure customer satisfaction with the three levels of NCSC call
center service-interactive voice response system, Tier 1 CSRs, and Tier 2
IIOs. To carry out the survey each month, representatives of the
independent consulting firm call 375 randomly selected customers. To
assess the customers' satisfaction with the CSRs, the representatives read
several statements and ask questions for the customers to rate their
experiences with CSRs. For the customer satisfaction performance measure
required in the contract, USCIS collects and summarizes data on the
customers' responses to the four statements below. The customers are asked
to rate their agreement with each of the statements using a scale of 1 to
7 (1 is strongly agree and 7 is strongly disagree).1

1. The representative seemed to fully understand my questions.

2. The representative was polite.

3. The representative did not rush me.

4. The representative answered my questions promptly.

1The complete scale for the four statements is as follows: 1-Strongly
agree; 2-Mostly agree; 3-Somewhat agree, 4-Mixed, Neither agree nor
disagree; 5-Somewhat disagree; 6-Mostly disagree; and 7-Strongly disagree.

Appendix IV: Criteria and Methodology Used by Independent Consulting Firm to
Measure Quality of Calls Monitored

An independent consulting firm scored CSRs on 23 separate quality
assurance factors as follows.

                              Service orientation

Greeted customer
Verified customer
Established rapport
Used customer name
Expressed empathy
Maintained composure
Expressed commitment
Offered additional assistance
Terminated call appropriately

Issue identification 	Obtained information Listed actively

Issue resolution 	Took responsibility Provided thorough information
Summarized actions

Communications 	Projected enthusiastic tone Conveyed confidence Spoke with
clarity Used appropriate language

Call management 	Controlled call Used time efficiently Minimized "dead
air" Extended hold courtesies Transferred call appropriately

Source: USCIS.

Page 30 GAO-05-526 Immigration Services

                         Appendix VI: Comments from the
                        Department of Homeland Security

Appendix VI: Comments from the Department of Homeland Security

Appendix VI: Comments from the Department of Homeland Security

Appendix VII: Comments from Pearson Government Solutions

Appendix VII: Comments from Pearson Government Solutions

Appendix VIII: GAO Contact and Staff Acknowledgments

GAO Contacts

  Staff Acknowledgments

(440318)

Paul L. Jones 202-512-8777

In addition to the above, Darryl W. Dutton, Ronald G. Viereck, Brian J.
Lipman, Christine F. Davis, Amy L. Bernstein, and Michele C. Fejfar made
key contributions to this report.

  GAO's Mission

Obtaining Copies of GAO Reports and Testimony

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone 	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

  To Report Fraud, Contact:

Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470

Gloria Jarmon, Managing Director, [email protected] (202)
512-4400Congressional U.S. Government Accountability Office, 441 G Street
NW, Room 7125 Relations Washington, D.C. 20548

Public Affairs 	Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

                           PRINTED ON RECYCLED PAPER
*** End of document. ***