Every organization has certain stressful jobs that particularly influence the organization’s performance. The following case presents an evaluation of a training program for one such job: dispatchers at an oil company. The program had several objectives, with a focus on improved customer service skills. The evaluation approach was comprehensive, using several data collection tools at four levels of evaluation. The results are very impressive.

Background
International Oil Company, one of the five major oil companies worldwide, has three dispatching centers. The following evaluation was conducted for the company’s central dispatch, located in Los Angeles, California. The dispatch center’s employees include 11 dispatchers, and a delivery supervisor.

The dispatchers are responsible for helping service station dealers who are unable to use the automated gasoline-ordering system for whatever reason. Dealers call central dispatch to place orders according to their gasoline needs, These dispatchers handle approximately 600 telephone calls per shift during peak times (eight to 10 telephone calls per dispatcher, per hour). Of the total calls received, 60 percent are from dealers needing information or order changes; the other 40 percent are from drivers needing advice or
assistance.

Because most of the company’s gas stations are open 24 hours a day, delivery times are divided into four quadrants. A dispatcher tries to schedule delivery during the specific quad requested by the dealer. After a dispatcher calls the distribution terminal to place the order, the driver plans the delivery route to deliver the load during the requested quad. The driver loads the tank wagon to the dealer’s product mix
specifications and then delivers the gasoline. Safety and economic precautions mandate that the dealer take the full load ordered. If the dealer is unable to do so, the driver must “pull out” without unloading the gasoline and return to the terminal to adjust the product mix for the next station on the schedule. When this happens, dealers with high volumes or small tanks may run out of gasoline before the next scheduled
delivery.

The Customer Service Design and Development Committee designed and developed the training that is the subject of this case study. The committee members were Jeanne’Connor-Green, who at the time of this evaluation worked for the oil company and who now works for Intercom; Jim Burton, manager of the company’s Retail Learning Center in Los Angeles; Bob DeGroot, developer, Sales training America (now Sales Training International); Ashley Thorton, program evaluator, ROI Evaluation Services. O’Connor-Green managed the design
process.

After only a few months on the job, it had become evident to the new central dispatch manger that dispatchers were not performing to the company’s expectations. He complained that dispatchers:

• Did not help out each other on the phone (i.e., were not working together as a team)
• Had poor telephone etiquette
• Had excessive absences
• Would sometimes become angry with dealers
• Would not double-check orders
• Did not listen to each other
• Did not solve problems well
• Had poor rapport with the dealers

In addition to these problems, central dispatch management was complaining that there were too many gasoline pullouts and customer complaints. Because of these performance and business problems, Burton was asked to develop training for the dispatchers. He brought in two consultants, Thorton and DeGroot.

Program Description
Burton, Thorton, and Degroot interviewed Greg Gaines, Operations and Delivery Support, and John Garner, dispatch manager, both with the oil company’s Western Distribution region, as part of a formal needs analysis. Gaines and Garner wanted a training program that would result in reduced pullouts (which would lower delivery costs); dispatchers who were willing and able to serve customers and handle problems in a safe, efficient, professional way; a teamwork environment to serve customers and handle problems better; and a “we make the difference” attitude displayed by all central dispatch employees.

In addition to central dispatch management, both dispatchers and dealers were interviewed. During the interviews, dispatchers said they wanted support from management in handling problems and to learn how to reduce or avoid stress on the job, to stay calm during difficult telephone calls, and to voice problems and get them solved before frustration led to “blowing up.” The dealers wanted both the good and the bad news. They wanted efficient and dependable gasoline ordering and accurate information on time of delivery. They also wanted dependable, hard-working dispatchers who treated dealers with consideration and courtesy. In addition to the interviews, area dealers received a survey to measure their current satisfaction. As a result of these interviews and the survey, Thorton and DeGroot designed and developed a customer service training program.

Objectives
Three types of program objectives were developed for all of the oil company’s training programs:

• Business objectives indicate what the return to the oil company will be once students apply the knowledge and skills acquired in training.
• Application objectives indicate what the student will do on the job as a result of instruction.
• Learning objectives specify what students must do to show that learning has taken place.

From the training needs identified by Gaines and Garner, two business objectives were identified:

• A reduction in the number of pullouts to lower delivery costs
• A reduction in customer complaints.

The course was designed in seven modules, with an application objective for each module:

• Telephone etiquette- Participant will use central dispatch’s telephone etiquette standards.
• Rapport building- Participants will use rapport-building techniques to improve communications
• and reduce customer complaints.
• Listening to customers- Participant will listen to customers in order to identify problems and focus conversations toward solutions.
• Problem solving-Participant will demonstrate trying to meet the customers’ needs by while solving the problem while the customer is on the telephone, calling the customer back to explain what is being done, or explaining why the problem cannot be solved and offering alternatives.
• Diffusing anger- Participant will defuse dealers’ anger and frustration during difficult telephone calls and will refocus conversations toward solving problems.
• Working as a team-Participants will view the total job of central dispatch as their responsibility and work together as a team to solve problems and meet the customers’ needs.
• Dealing with stress- Participant will identify personal responses to business-related stress and then take positive action to dissipate that stress before it results in negative consequences.

Several learning objectives were developed for each of the seven modules, and those objectives are shown in Table 1.

Program Participants and Delivery
The program’s trainees were dispatchers with varying levels of experience and education. For example, one dispatcher had been working for the oil company for 30 years, while another was new to the job. Some dispatchers had high-school diplomas, whereas others had attended several years of college. Central dispatch must be staffed 24 hours a day, 365 days a year; all dispatchers worked in rotating eight-
hour shifts, unless working overtime.

The training was facilitated by Burton at the Retail Learning Center in Los Angeles (at no cost to participants) and was presented over a three-week period in four-hour-long blocks. This format allowed dispatchers a chance to practice newly learned skills and then respond to any problems or issues that needed to be discussed. A variety of delivery methods were used in the training, including lecture, discussion, case studies, role plays, and games.

Models and Techniques
Most of the oil company’s products training is evaluated for reaction, learning, application, and business results. The few exceptions are awareness programs and programs that are taught only once, both of which are evaluated only for reaction and learning. In the company’s training department, evaluation is part of the design process, and an evaluation plan is presented before development starts. Evaluation plans
are similar to proposals. They describe which objectives are to be evaluated, when they are to be evaluated, and which evaluation design will be used. They also describe where the evaluation will take place, which methods will be used (tests, surveys, observations, company records, etc.), and who will be involved in the evaluation.

The effectiveness of this particular training was measured on four levels:

1. Business results-What are the bottom-line results for the oil company, for example, in terms of money, time, and goodwill?
2. Performance results-are the participants applying what they have learned?
3. Learning results-Did participants learn what they were supposed to learn?
4. Reaction results-What do the participants think about the training?

The evaluation plan proposed that business results would be determined by comparing company records of the numbers of pullouts and customer complaints, before and after training. A survey would also be sent to dealers to measure their satisfaction with the service they received from dispatchers, before and after training.

To determine if application objectives were being met, the participants would be observed on the job, six months after training. The application objective that dealt with stress was to be measured by comparing absenteeism records before training and six moths later.

To determine the learning results of training, three methods would be used: case studies, games, and exercises. Case studies are a way to evaluate immediately whether specific learning objectives are being met. Games recreate an event, so that participants can manipulate a situation without risk, then analyze what happened. Exercises direct participants toward specific objectives and encourage high
participant involvement (Abelle, 1986).

A generic reaction questionnaire, developed by the oil company’s training department, would be used to assess reaction results. This questionnaire is used to measure all participants’ reactions to training for any course designed by the department. The advantage of having one questionnaire is that all courses may be compared with one another. In addition to the generic questionnaire, dispatchers and mangers were to be interviewed to determine their reaction to the training six months after it was completed.

Costs
The training program cost central dispatch management $60,000. This figure included the costs of situational and needs analysis, program design, program development (materials, manuals, etc.), and evaluation of the program. In fact, $60,000 may not be the complete cost of training, because the trainer’s salary and other costs (e.g., training facility) were not included. This figure was the amount used to calculate the return on investment (ROI), however, because no other costs were available.

Data Analysis and Results
At the company’s request, this evaluation, this evaluation used only measures of central tendency and frequency distributions to show results. The participant population (11) was so small that no sampling was required.

Business Results
As outlined in the evaluation plan, business results were obtained by comparing pullout records, dealer complaints, and the results of dealer surveys, before and after training. The evaluation of these business objectives took place 11 months after training, a month earlier than was planned.

PULLOUT RECORDS. Each pullout cost the oil company $250. Table 2 shows both the number of monthly pullouts and their cost. A reduction in the number of pullouts resulted in a savings of $354,750 during the first 11 months following training. Although the number of pullouts had shown a decreasing trend since July 1990 (see Figure 1), that decrease became even more dramatic after the training intervention in August 1991. The rise in the number of pullouts in December 1991 happened when several dispatchers went on vacation and untrained temporary employees filled in for them.

DEALER COMPLAINTS. Before training, concern about the potential effects of frequent dealer complaints prompted the dispatch manager to keep a log of these complaints. The log indicated that there were approximately three to five complaints each month before training, although in the moth immediately before training (July 1991), there were seven complaints. For six months after training (August 1991 to
January 1992), the log indicated a total of only six complaints; there have been no complaints since January 1992.

Based on an average salary for management, each dealer complaint cost the oil company approximately $50 worth of time. During the 11 months immediately before training, those complaints cost the company approximately $2,200. In the 11 months immediately after training, complaints cost the oil company $300. That represents a savings of $1,900. These figures do not take into consideration the cost of
the drivers’ or the dealers’ time.

Not only were complaints reduced significantly (85 percent) since the training intervention in August 1991, but Garner received several letters and telephone calls from dealers complimenting dispatchers on a job well done. Other benefits of decreased customer complaints included reduced dispatcher stress, improved customer relations, and an improved level of customer service.

DEALER SURVEY. A survey was sent to the 50 members of the Company’s Dealer Council, and 38 dealers responded. Table 3 shows responses for each question on this survey. In addition to indicating that dealers were satisfied overall with the performance of the dispatchers, the survey showed that dealers’ requirements were being met by dispatchers, dispatchers displayed proper telephone etiquette, and the
overall performance of dispatchers was very good.

The results of the survey suggested the following opportunities for improvement:

1. Dealers wanted more help from dispatchers in identifying and solving problems.
2. Dealers wanted more notification of gallon changes when load mix would be changed by more than 300 gallons.
3. Dealers wanted more notification if delivery would not arrive in the quad for which it was ordered.

Application Results
When a training program is implemented, it is expected that all learned skills will be used on the job; however, skills are used only as they are needed, and some skills are used more than others. Observation of dispatchers and a comparison of absentee records before and after training indicated that the dispatchers were meeting all the application job objectives.

OBSERVATION. The dispatchers were observed on two consecutive days for two hours each day. Figure 2, a frequency distribution graph, shows the results of those observations. The dispatchers applied the knowledge and skills they were taught on the job. There was only one incident when a dispatcher did not use “proper telephone etiquette.” The phones were extremely busy, two callers were on hold, and the dispatcher answered, “Good morning, central Dispatch,” instead of “Goof morning, Central Dispatch. How may I help you?”

ABSENTEEISM RECORDS. The performance objective for dealing with stress could not be measured by observing dispatchers on the job. Instead, the absentee rate was used to determine if the objective was met. If training reduced stress, the absentee rate would be expected to decline after training. Figure 3 shows that although a decline in absenteeism began in February 1991, that decline increased dramatically after training in August 1991. Each day of work missed by a dispatcher cost the oil company $200. During the 11 months immediately before training, absences cost the oil company $5,200. The cost of absences for the 11 months immediately after training dropped to $1,200, a cost savings of $4,000.

Learning Results
Each time a learning objective was mastered, the objective was checked off a list by both the participant and the instructor. Learning results were assessed by using case studies, games, and role-playing exercises. All participants mastered 100 percent of the learning objectives.

Reaction Results
Reaction to training is usually assessed only by administering the Products Training Generic Questionnaire immediately after a training program. In this instance, however, an additional method, the interview, was used. Interviews with management and dispatchers were conducted six months after training to find out if they believed the training had met the needs they had originally identified.

REACTION QUESTIONNAIRE. The reaction questionnaire is divided into five sections. The design section contains questions addressing length of the course, amount of information, content, pace, and course timing in relation to job assignment. For example, questions in the section on method of delivery request information about job aids, case studies, role plays, pre- and postcourse assignments, lectures, and practice time. Questions about location, the training room, and equipment appear in the learning environment section. The instructor section asks the respondent to rate the instructor on knowledge of subject matter, presentation skills, classroom management, and other issues under his or her control. The questionnaire ends with an overall training section. Participants answer on a 5-point scale, ranging from 1 (needs improvement) to 5 (excellent). The results for each section of the Products Training Generic Reaction Questionnaire are shown in Table 4.

MANAGEMNET REACTION INTERVIEW. In interviews, central dispatch managers were asked to rate how well dispatchers met expectations after training. The questions were based on problem areas identified in the original needs analysis. Ratings were on a scale of 1 to 10, with 10 being the best possible score. Figure 4, shows both the mean ratings obtained for these problem areas during the needs
analysis and the mean results for the posttraining interviews.

DISPATCHER REACTION INTERVIEW. Before training and six months after training, dispatchers were asked to rate their overall job satisfaction and how well their needs were being met. Ratings were on a scale from 1 to 10, with 10 being the best possible score. Figure 5 shows the results of these interviews.

Return on Investment
The oil company’s western region central dispatch saved $360,650 during the first 11 months after training. With 460,000 as the cost of training, a net value of $300,650 was generated. The ROI was calculated as follows:

Net benefits/program costs = $300,650/$60,000 = 501%

Because of this good result, the course was taught to the other two central dispatch locations, with similar outcomes.

Conclusions and Recommendations
Several barriers had some effect on both the business and the application results of training. For example, dispatchers can schedule deliveries but have no control over the drivers. During the observation period, a dealer called to ask about the status of his gasoline delivery. After several telephone calls, the dispatcher found that the driver did not want to get involved with rush-hour traffic and had decided
(without notifying central dispatch) to change his delivery schedule. The dispatchers claim that type of incident happens frequently.

Also, each plant has a different delivery procedure. The oil company has no control over the common carriers, and, therefore, there is no standardization for deliveries. Finally, some dealers do not “stick” the tanks (i.e., measure how much gasoline is left) and therefore do not order correctly. If a dealer misreads or fails to read the stick and orders 300 gallons when his tank will hold only 200, then the arriving
truck will have to pull out. Thus, both the oil company and the dealer lose time and money.

• A net benefit of approximately $300,000 during the first 11 months after training
• A reduction in pullouts, saving the oil company $354,750 during the first 11 months after training
• A reduction in absenteeism, saving the oil company $4,000 during the first 11 months after training
• An 85 percent reduction in customer complaints, saving the oil company $1,9000 during the first 11 months after training
• Decreased dealer complaints, resulting in
  • -reduced dispatcher stress
  • -improved customer relations
  • -improved customer service
• Several letters and phone calls from dealers to management complimenting dispatchers on a job well done
• Knowledge and skills taught in the classroom being applied on the job
• An overall increase in dispatchers’ job satisfaction

Management believed the training had resulted in positive changes; dispatchers felt that training had made their jobs easier and that they had more management support than before. In short, all training needs of dealers, management, and dispatchers had been met. Based on these results, additional programs were recommended, including follow-up training for dispatchers and a basic training program for all carriers
(even the common carriers not directly employed by the oil company, if possible) to teach them the company’s expectations and requirements. In addition, management requested that all new dispatch employees participate in the training program. Another recommendation was for a training program to let dealers know what to expect from central dispatch. This training could be in the form of a booklet, job aid, or module in the Dealer Management Development Program, which is provided to all new dealers.

Questions for Discussion
1. Is it necessary to develop program objectives at different levels? Explain. Review the objectives for this program at each level. Are they all measurable?
2. Critique the methods used to evaluate the program at each level.

3. Do you think the cost estimate for the program is adequate? Explain.
4. What is your reaction to the application results? Explain.
5. How credible are the business results? Explain.
6. Is it possible to isolate the effect of the training program on business results? How?

The Author
Rebecca Payne is an evaluation consultant, the owner and founder of ROI Evaluation Services, and an associate with Pinebrook Holub and Associates, an organizational training and development consulting company. Payne holds a master’s degree from the University of Houston in training and development, with a specialty in program evaluation. She has conducted other evaluation research for both the government and private business, has facilitated evaluation workshops and seminars for the American Society for Training and Development, and is past research director for the Houston Chapter. Payne can be contacted at the following address: ROI Evaluation Services, 22210 Coriander, Katy, TX 77450.

Reference
Abella, K.T. (1986). Building successful training programs. Reading, MA: Addison-Wesley.