This morning one of the questions posed by readers of IQ Six Sigma posed the following question:
“My department is charged with creating a "culture of process improvement" within our zone. We're struggling with what that looks like once we've created this culture. Looking at the Toyota model, they challenge employees to look for PI opportunities every day. What exactly does that look like, and what measurements should we consider (i.e. number of PI suggestions with managers being held accountable for X number per quarter, etc.) I'd like some ideas.”
My "short" answer (admittedly, this answer could have--and has--filled books):
Well, one thing you for sure don't want to do is set some quota for suggestions. You may already be faced with an uphill battle, because the leadership at your organization is actually the entity that has to create that culture of process improvement. If they are just rolling it downhill like any other MBO, it suggests that they don't know what they are doing.
Toyota does challenge employees with looking for improvement ideas. One of the ways they do that is by implementing them. Most suggestion boxes go unheeded by employees because they go unheeded by management. At companies like Toyota, they use mechanisms such as Quality Function Deployment to communicate the voice of the customer to everyone in the organization. It allows people on the production line a clear line of sight to the mind of the customer and the organization's leadership.
How do you establish this culture? Well, if you have to do it locally, start by knowing that you may not be as successful as you would if your leaders were leading. Empowerment is a big piece of the pie...you have to let people know they are empowered to make changes. You have to have mechanisms in place that let changes be approved at the lowest possible level. This doesn't mean that any line worker should be empowered to make design changes that require retooling the entire line without some study, but small local changes should be able to be made and standardized locally, as long as they don't suboptimize the system.
So, start by listening to people. I once found an operator potting an assembly with epoxy, using a pneumatic syringe...one of the primary quality characteristics in this assembly was that the epoxy had to be free from air bubbles! This line worker had been telling people about it for some time, but no one would listen; after all, an engineer had designed that workstation--who was this uneducated line worker to question the engineers? So, again, listen! Your people have the answers to most of your quality problems. It may take some time before they will talk (because it's a culture change for them, too).
It's not enough just to listen, though, you have to act! If you don't act on what you hear, and act promptly and visibly, soon you won't have anything to listen to. If you listen and act, you'll soon find that you can't keep up with the suggestions for improvement. That will be the beginning of changing the culture to one of improvement.
You also have to be a champion. You have to be out there talking it up, walking the talk, aggressively and visibly removing obstacles to improvement. Align whatever passes for reward and recognition in your zone with PI, to let people know that it's important. Constantly let people know what you value; proactively seek (and take) opportunities to demonstrate those values and beliefs. Measure important process and throughput measures...use SPC so you don't make boneheaded decisions about those measures.
As to what to measure to gage progress along the cultural change path...well, there are lots of things you can measure. Probably the most important are results and employee morale. If your error rates, rework rates and scrap rates are going down and your throughput is going up, it's working. You can also measure suggestions received; but you should use that number as the basis for a perhaps more important metric: percentage of suggestions implemented. This is certainly not an exhaustive list...there are numerous things you can measure. Deming said that the most important numbers are unknown and unknowable; this is what makes measuring what we can measure so important.
Standardize, do 5S, start holding 5-10 minute meetings at every cell every day, to go over quality metrics, suggestions entered, suggestions implemented (and get ideas for implementing suggestions), recognize people for advancing continuous improvement.
Thursday, March 11, 2010
Monday, February 8, 2010
Bonus Plans
In one of my LinkedIn Discussion Groups, we have been going back and forth on the idea of bonus schemes for a couple of weeks now. Today, we got a thoughtful post from John, who said that "Incentives and reinforcement are part of what I design." He offered insights as to how a system might be designed. I responded to one of his ideas.
He pointed out that "bonuses have been factored into sales compensation since the dawn of time because we know that vigorous sustainted effort is required," then asked, "Why here and not in all key jobs?" One of his reasons: "Execs are unfamiliar with the ways that objective measures can be designed for staff, managers, and production people," and goes on later to suggest that "Incentives need to be based on objective measures of performance, and that "ALL incentives are ultimately individual."
While these ideas seem to make some common sense, things that we've learned over the last 30 years or so suggest that they bear some scrutiny. Here's my reply:
_________________________________________________
I think Scott points to a couple of drawbacks to many bonus schemes. There are some problems with one of his fixes, though.
Let's talk about objective criteria: sometimes they do exist, but it's not as often as we think, and it's never (an I do mean NEVER) as clear-cut as we think. Anyone who's ever seen the Red Bead Experiment can attest to that. It's also almost never possible to separate the performance of the person from the performance of the system in which they operate. So, even when we talk about "anyone who reaches the goal gets the bonus," we assume that it's possible for everyone to reach that goal, completely independent of all the factors that drive the system.
Let me illustrate with an example from my days in the Military:
An Army school convenes twice per year, and runs for 5 months. One class starts in late Fall, the other in late Spring. Each class is led and instructed by two soldiers. During a study of these classes 10-11 years back, one of these instructor teams clearly excelled, by all the “objective” criteria used to measure performance: very low dropout rates, very high academic achievement with very little remediation, almost no legal or medical problems, excellent advancement rates for graduates, etc. The other team, however, didn’t fare so well; their dropout rates were very high, most of their students struggled to pass the weekly exams (despite extensive remediation and night study), they had numerous problems reported from both base security, military police and community police, a high incidence of sick days, and most students who graduated required a lot of extra work to gain adequate proficiency, once they arrived at their units.
Of course, the team with the highest scores on all the criteria won Instructor of the Quarter/Year, Soldier of the Quarter/Year and other achievement awards given by the training command, and were consistently ranked in the top 5 by their commanders—all this, of course, led to rapid advancement for these soldiers
The low-scoring team ended up at the bottom of the heap, in the “not ranked” category, and received letters of reprimand for their poor performance.
Eventually, someone noticed that this difference in performance transcended the soldiers themselves…ALL the Fall classes were better, and ALL the Spring classes were worse. As it turned out, there was a great logical explanation for all of it.
The classes that convened in the late Fall comprised students who had come into the Army right after High School graduation, many on delayed entry programs. They had enlisted for this particular specialization. They were highly qualified and highly motivated, both for the Army and for this school. In contrast, the Spring classes were made up of people for whom the Army was something to do after they had failed to find a job, and who had been put into this class to fill a quota. Some had needed waivers to get into the Army; many had required waivers to get into the class.
Ironically, if you looked at the workloads for the instructor teams, the hardest-working and most creative teams were those for the Spring class. They had to be, just to survive. They had to conduct remedial sessions at night study, as well as before classes, lunchtimes, weekends, etc. They had to continually push the envelope to find new and better ways to get these challenged students to learn. The other team largely skated through the duty…very little extra time, no extra thought needed.
This same sorry story still happens every day in Military recruiting. Recruiters in very populous areas in more patriotic-leaning states have very few problems meeting quota. They get awards, advancements, etc. Those in rural areas work many times harder and often don't make quota, and are forced to accept low evaluations and sometimes humiliating "remedial" sessions where senior recruiters come in and yell at them like drill sergeants ...many of these are just back from Iraq or Afghanistan.
He pointed out that "bonuses have been factored into sales compensation since the dawn of time because we know that vigorous sustainted effort is required," then asked, "Why here and not in all key jobs?" One of his reasons: "Execs are unfamiliar with the ways that objective measures can be designed for staff, managers, and production people," and goes on later to suggest that "Incentives need to be based on objective measures of performance, and that "ALL incentives are ultimately individual."
While these ideas seem to make some common sense, things that we've learned over the last 30 years or so suggest that they bear some scrutiny. Here's my reply:
_________________________________________________
I think Scott points to a couple of drawbacks to many bonus schemes. There are some problems with one of his fixes, though.
Let's talk about objective criteria: sometimes they do exist, but it's not as often as we think, and it's never (an I do mean NEVER) as clear-cut as we think. Anyone who's ever seen the Red Bead Experiment can attest to that. It's also almost never possible to separate the performance of the person from the performance of the system in which they operate. So, even when we talk about "anyone who reaches the goal gets the bonus," we assume that it's possible for everyone to reach that goal, completely independent of all the factors that drive the system.
Let me illustrate with an example from my days in the Military:
An Army school convenes twice per year, and runs for 5 months. One class starts in late Fall, the other in late Spring. Each class is led and instructed by two soldiers. During a study of these classes 10-11 years back, one of these instructor teams clearly excelled, by all the “objective” criteria used to measure performance: very low dropout rates, very high academic achievement with very little remediation, almost no legal or medical problems, excellent advancement rates for graduates, etc. The other team, however, didn’t fare so well; their dropout rates were very high, most of their students struggled to pass the weekly exams (despite extensive remediation and night study), they had numerous problems reported from both base security, military police and community police, a high incidence of sick days, and most students who graduated required a lot of extra work to gain adequate proficiency, once they arrived at their units.
Of course, the team with the highest scores on all the criteria won Instructor of the Quarter/Year, Soldier of the Quarter/Year and other achievement awards given by the training command, and were consistently ranked in the top 5 by their commanders—all this, of course, led to rapid advancement for these soldiers
The low-scoring team ended up at the bottom of the heap, in the “not ranked” category, and received letters of reprimand for their poor performance.
Eventually, someone noticed that this difference in performance transcended the soldiers themselves…ALL the Fall classes were better, and ALL the Spring classes were worse. As it turned out, there was a great logical explanation for all of it.
The classes that convened in the late Fall comprised students who had come into the Army right after High School graduation, many on delayed entry programs. They had enlisted for this particular specialization. They were highly qualified and highly motivated, both for the Army and for this school. In contrast, the Spring classes were made up of people for whom the Army was something to do after they had failed to find a job, and who had been put into this class to fill a quota. Some had needed waivers to get into the Army; many had required waivers to get into the class.
Ironically, if you looked at the workloads for the instructor teams, the hardest-working and most creative teams were those for the Spring class. They had to be, just to survive. They had to conduct remedial sessions at night study, as well as before classes, lunchtimes, weekends, etc. They had to continually push the envelope to find new and better ways to get these challenged students to learn. The other team largely skated through the duty…very little extra time, no extra thought needed.
This same sorry story still happens every day in Military recruiting. Recruiters in very populous areas in more patriotic-leaning states have very few problems meeting quota. They get awards, advancements, etc. Those in rural areas work many times harder and often don't make quota, and are forced to accept low evaluations and sometimes humiliating "remedial" sessions where senior recruiters come in and yell at them like drill sergeants ...many of these are just back from Iraq or Afghanistan.
Monday, January 11, 2010
Some Problems with Conditional Probability
A lot of people in my statistics classes struggle with conditional probability; you may be in the same boat. If you are, though, please don’t feel alone. A lot of people get this (and simple probability, for that matter) wrong. If you read "Innumeracy" by Poulos or "The Power of Logical Thinking" by Vos Savant, you'll see examples of how a misunderstanding or misuse of this topic has put innocent people in prison and ruined many careers. It's one of the reasons I'm passionate about statistics; it's counterintuitive for me, too. It's not easy to work out in your head, unless maybe you do it all the time. I always have to build a table.
When confronted with conditional probability, my advice is that you be completely process-driven; identify what's given, then follow the process and the formulas religiously. After a while, you can start to see it intuitively, but it does take a while. It's all about what you are given, and how you define things.
In my MBA stats class, one of the problems that always stumped the students was a conditional problem:
“Pregnancy tests, like almost all health tests, do not yield results that are 100% accurate. In clinical trials of a blood test for pregnancy, the results shown in the accompanying table were obtained for the Abbot blood test (based on data from "Specificity and Detection Limit of Ten Pregnancy Tests" by Tiitinen and Stenman, in the Scandanavian Journal of Clinical Laboratory Investigation, 53, Supplement 216). The disclaimer in the journal stated that other tests are more reliable that the test with results given in this table.
“1. Based on the results in the table, what is the probability of a woman being pregnant if the test indicates a negative result?
“2. Based on the results in the table, what is the probability of a false positive; that is, what is the probability of getting a positive result if the woman is not actually pregnant?”
Everyone would just try to look at it as though there were no conditions...they would say, 5/80 for question 1, and 3/80 for question 2. The first question, though, is asking "what is the chance of being pregnant, given a negative result?" There were 16 negative results, and of those, 5 were pregant. So the answer is 5/16, or 31.25%. For the second question, it's what is the probability of a positive, given that the woman is not pregant. In this case, there are 14 non-pregnant women, and 3 of those got a positive result. So that's about 21.42%.
These numbers, and this idea, are really important--that is, they carry real-world import. Some statisticians make their living explaining these concepts to juries. People get fired or arrested because of false positives on urinalysis and other tests, because there is a general impression that they are far more reliable than they actually are.
Let’s look at a different example. In the military, people are given random drug screenings. The test is “certified 99% accurate.” I was always told that this means that if you do drugs, and you’re tested, it will catch you 99 percent of the time. We think, “logically,” that this means there is only a one percent false negative rate…that the fact that someone who does drugs doesn’t get caught one percent of the time indicates that one percent false positive rate. Worse, we assume that if the “false negative rate” is only 1 percent, the false positive rate must also be one percent…it’s just common sense, right?
But “common sense” isn’t…it’s neither common nor truly sensical. Look at it this way…suppose we test 100,000 service members. Suppose further that .1% or 1 in a thousand service members actually do drugs. We might get this:
Tables like this are informative, but they don’t tell the whole story. You can see from this that the company is technically correct…at least in this case, of 100 people who did drugs, 99 were caught and 1 was not. But a false positive rate and a false negative rate are made up of more. To get to the whole story, it’s also good to do the marginals, or row and column totals:
Numbers like this, the numbers of people tested, are very important. This helps us figure out our givens. The false negative rate is not the number of people who did drugs and tested negative. It’s the number out of all the people who tested negative who actually did drugs. In this case, the false negative rate is much better than advertised…it’s 1/98,902, or .00001, about one in 10,000 who do drugs and get tested get away with it.
The consequences, though, are on the false positive side…this is where people get turned away for employment, get fired, etc. In the case of the military, a lot of people end up in a lot of trouble with the random urinalysis program. While we want to be cautious, and we don’t want a lot of druggies flying or controlling aircraft or tanks or other deadly weapons, we should also be concerned that we might be ruining careers unnecessarily. If we look at the table, the “common sense” interpretation of the false positive rate would be 999/100000, or 0.999 percent, very close to the one percent that we assumed initially. But, as astounding as it may seem, considering the number of people that are convicted each year because of this assumption, this is entirely incorrect!
The actual false positive rate consists of the number of people incorrectly identified as drug users, or the number of non-drug users out of the total number of positives. In this case, that’s 999 out of 1,098, or 90.98%! In other words, your chance of actually being a drug user, given a positive result on this “99% accurate” test, is only 9.02%!
Yes, it’s tricky. No, it’s not easy. But it’s important. It touches lives. Juries, lab technicians, doctors and nurses, lawyers, employers, employees and patients who don’t understand this put either themselves or others in peril every day.
When confronted with conditional probability, my advice is that you be completely process-driven; identify what's given, then follow the process and the formulas religiously. After a while, you can start to see it intuitively, but it does take a while. It's all about what you are given, and how you define things.
In my MBA stats class, one of the problems that always stumped the students was a conditional problem:
“Pregnancy tests, like almost all health tests, do not yield results that are 100% accurate. In clinical trials of a blood test for pregnancy, the results shown in the accompanying table were obtained for the Abbot blood test (based on data from "Specificity and Detection Limit of Ten Pregnancy Tests" by Tiitinen and Stenman, in the Scandanavian Journal of Clinical Laboratory Investigation, 53, Supplement 216). The disclaimer in the journal stated that other tests are more reliable that the test with results given in this table.
| Positive Result | Negative Result | |
| Subject is pregnant | 80 | 5 |
| Subject is not pregnant | 3 | 11 |
“1. Based on the results in the table, what is the probability of a woman being pregnant if the test indicates a negative result?
“2. Based on the results in the table, what is the probability of a false positive; that is, what is the probability of getting a positive result if the woman is not actually pregnant?”
Everyone would just try to look at it as though there were no conditions...they would say, 5/80 for question 1, and 3/80 for question 2. The first question, though, is asking "what is the chance of being pregnant, given a negative result?" There were 16 negative results, and of those, 5 were pregant. So the answer is 5/16, or 31.25%. For the second question, it's what is the probability of a positive, given that the woman is not pregant. In this case, there are 14 non-pregnant women, and 3 of those got a positive result. So that's about 21.42%.
These numbers, and this idea, are really important--that is, they carry real-world import. Some statisticians make their living explaining these concepts to juries. People get fired or arrested because of false positives on urinalysis and other tests, because there is a general impression that they are far more reliable than they actually are.
Let’s look at a different example. In the military, people are given random drug screenings. The test is “certified 99% accurate.” I was always told that this means that if you do drugs, and you’re tested, it will catch you 99 percent of the time. We think, “logically,” that this means there is only a one percent false negative rate…that the fact that someone who does drugs doesn’t get caught one percent of the time indicates that one percent false positive rate. Worse, we assume that if the “false negative rate” is only 1 percent, the false positive rate must also be one percent…it’s just common sense, right?
But “common sense” isn’t…it’s neither common nor truly sensical. Look at it this way…suppose we test 100,000 service members. Suppose further that .1% or 1 in a thousand service members actually do drugs. We might get this:
| Do Drugs | Don't Do Drugs | |
| Test Positive | 99 | 999 |
| Test Negative | 1 | 98,901 |
Tables like this are informative, but they don’t tell the whole story. You can see from this that the company is technically correct…at least in this case, of 100 people who did drugs, 99 were caught and 1 was not. But a false positive rate and a false negative rate are made up of more. To get to the whole story, it’s also good to do the marginals, or row and column totals:
| Do Drugs | Don't Do Drugs | Total in Row | |
| Test Positive | 99 | 999 | 1098 |
| Test Negative | 1 | 98,901 | 98,902 |
| Totals | 100 | 99,900 | 100,000 |
Numbers like this, the numbers of people tested, are very important. This helps us figure out our givens. The false negative rate is not the number of people who did drugs and tested negative. It’s the number out of all the people who tested negative who actually did drugs. In this case, the false negative rate is much better than advertised…it’s 1/98,902, or .00001, about one in 10,000 who do drugs and get tested get away with it.
The consequences, though, are on the false positive side…this is where people get turned away for employment, get fired, etc. In the case of the military, a lot of people end up in a lot of trouble with the random urinalysis program. While we want to be cautious, and we don’t want a lot of druggies flying or controlling aircraft or tanks or other deadly weapons, we should also be concerned that we might be ruining careers unnecessarily. If we look at the table, the “common sense” interpretation of the false positive rate would be 999/100000, or 0.999 percent, very close to the one percent that we assumed initially. But, as astounding as it may seem, considering the number of people that are convicted each year because of this assumption, this is entirely incorrect!
The actual false positive rate consists of the number of people incorrectly identified as drug users, or the number of non-drug users out of the total number of positives. In this case, that’s 999 out of 1,098, or 90.98%! In other words, your chance of actually being a drug user, given a positive result on this “99% accurate” test, is only 9.02%!
Yes, it’s tricky. No, it’s not easy. But it’s important. It touches lives. Juries, lab technicians, doctors and nurses, lawyers, employers, employees and patients who don’t understand this put either themselves or others in peril every day.
Friday, November 20, 2009
More on Six Sigma Metrics
There is an excellent article in this month's Six Sigma Forum magazine about the process sigma. The authors have examined a lot of the literature about the metric and come to some very interesting (and, in my opinion, accurate) conclusions. Six Sigma Forum offers an email address to respond to each article; this was my response:
I was very excited to see this article. I have been questioning this for years, and had just started doing some research with a view toward writing a similar article. I heartily agree with most of the authors’ points. I think they did a great job of catching at least the high level of the controversy and their enumeration of the advantages, disadvantages and myths should be made required reading for anyone in a Six Sigma role, especially Master Black Belts and Black Belts.
An area to which I had planned to give a bit more attention is the Statistical Process Control component of the metrics equation. Before you can make any assumptions about capability or process performance over time, you must measure it over time, and it must display a reasonable degree of statistical control. Only then do you have the assumption of homogeneity of data that makes any assumptions about an underlying distribution valid.
A foundational basis of SPC also sharpens the focus of the discussion surrounding the shift, and the short-term/long-term question. While it is possible for a process in a state of statistical control to have some underlying shifts that are not detected using Shewhart charts, sustained mean shifts of up to 1.5 sigma will almost certainly be detected within 10 subgroups following the shift, if the four most common Western Electric Zone Tests are applied.
Now, if you’re taking four samples per day for your monitoring subgroup, from a high volume operation—say, 2,000 units per hour—that might mean 9 days before the signal shows up; you’d have run approximately 64,000 units from a process whose mean had shifted. In those situations, CUSUM or other schemes more sensitive to gradual sustained shifts might be more appropriate.
Having said all that, though, it’s unlikely that shifts of that sort will go completely undetected in a well-monitored process. What we are essentially saying is that some assignable-cause variation is going to show up randomly, and for time periods too short to be detected by our charts. In that case, I believe that the local measures of dispersion used for control chart factors provide a reasonable way to operationally define short-term and long-term variation, if you must. R-bar/d2 and S-bar/c4, used to calculate control limits, provide very good estimates of short-term (within-subgroup) variation. Comparing that estimate with the standard deviation for the entire set of data will reveal whether any significant shifting has taken place. This would provide a fairly unambiguous test for shifts. Whether and how you want to define and measure the magnitude of any shift detected using this method could be another discussion; the fact that this argument is taking place without a method is another source of confusion.
It seems that we not only have to discriminate between short and long-term sigma, but we have to have a “short-term sigma assuming long-term data” and “long-term sigma.” Apparently, we can also have negative process sigmas, with DPMO greater than one million! Just look at the commonly-used sigma calculator at www.isixsigma.com, and click on the link for more information about the calculations. Their explanation, that sigma is just a z-score, shows how far we have come from an understanding of capability in some of these discussions. I think most of this falls under Wheeler’s category of victories “of computation over common sense.”
I can understand if you want to gig yourself 1.5 sigma to make your process sigma align with the one in all the tables, accepting the Motorola shift. What I don’t understand is why you would then decide that, in the longer term, it’s going to shift another 1.5 sigma (this seems to be the logic behind the “benchmark Z” used in some software packages these days.) So…now six sigma is actually three sigma by default? What’s the point?
I recently taught a Six Sigma certification exam prep course for my local ASQ chapter, and the primer for that course—a popular reference used by a huge number of applicants for that ASQ certification—suggested that, given binomial data (a proportion defective), you should use a log transformation to force the data into an approximation of the Poisson. I don’t know why anyone would do this, unless you really have to be able to have that negative process sigma and a DPMO of more than one million. For years, I have been doing just the opposite; transforming Poisson data to Binomial using e-DPU to estimate DPMO.
This brings up another excellent point from your authors: we need to figure out how we are going to count units and opportunities. My own belief is that we should limit ourselves to definitions that end up providing an estimate of proportion defective. An opportunity, in that case, would be the most discrete thing we could count; in other words, there could be no more than one defect per opportunity. This would get rid of the “negative sigma” nonsense.
I strongly endorse their recommendation that we use DPMO. The procedure they outline for finding DPMO is straightforward and useful. Calculating DPMO this way would provide a reasonable estimate. If we want to err on the side of safety, we might continue to use the 1.5 sigma shift for high-volume processes, or no shift for lower-volume processes. DPMO is a more intuitive metric, and would keep people from having to go to the table to translate DPMO to process sigma, and then decode it again later for anyone who wants to know what it means. That’s unnecessary rework, something we’d all like to avoid.
I was very excited to see this article. I have been questioning this for years, and had just started doing some research with a view toward writing a similar article. I heartily agree with most of the authors’ points. I think they did a great job of catching at least the high level of the controversy and their enumeration of the advantages, disadvantages and myths should be made required reading for anyone in a Six Sigma role, especially Master Black Belts and Black Belts.
An area to which I had planned to give a bit more attention is the Statistical Process Control component of the metrics equation. Before you can make any assumptions about capability or process performance over time, you must measure it over time, and it must display a reasonable degree of statistical control. Only then do you have the assumption of homogeneity of data that makes any assumptions about an underlying distribution valid.
A foundational basis of SPC also sharpens the focus of the discussion surrounding the shift, and the short-term/long-term question. While it is possible for a process in a state of statistical control to have some underlying shifts that are not detected using Shewhart charts, sustained mean shifts of up to 1.5 sigma will almost certainly be detected within 10 subgroups following the shift, if the four most common Western Electric Zone Tests are applied.
Now, if you’re taking four samples per day for your monitoring subgroup, from a high volume operation—say, 2,000 units per hour—that might mean 9 days before the signal shows up; you’d have run approximately 64,000 units from a process whose mean had shifted. In those situations, CUSUM or other schemes more sensitive to gradual sustained shifts might be more appropriate.
Having said all that, though, it’s unlikely that shifts of that sort will go completely undetected in a well-monitored process. What we are essentially saying is that some assignable-cause variation is going to show up randomly, and for time periods too short to be detected by our charts. In that case, I believe that the local measures of dispersion used for control chart factors provide a reasonable way to operationally define short-term and long-term variation, if you must. R-bar/d2 and S-bar/c4, used to calculate control limits, provide very good estimates of short-term (within-subgroup) variation. Comparing that estimate with the standard deviation for the entire set of data will reveal whether any significant shifting has taken place. This would provide a fairly unambiguous test for shifts. Whether and how you want to define and measure the magnitude of any shift detected using this method could be another discussion; the fact that this argument is taking place without a method is another source of confusion.
It seems that we not only have to discriminate between short and long-term sigma, but we have to have a “short-term sigma assuming long-term data” and “long-term sigma.” Apparently, we can also have negative process sigmas, with DPMO greater than one million! Just look at the commonly-used sigma calculator at www.isixsigma.com, and click on the link for more information about the calculations. Their explanation, that sigma is just a z-score, shows how far we have come from an understanding of capability in some of these discussions. I think most of this falls under Wheeler’s category of victories “of computation over common sense.”
I can understand if you want to gig yourself 1.5 sigma to make your process sigma align with the one in all the tables, accepting the Motorola shift. What I don’t understand is why you would then decide that, in the longer term, it’s going to shift another 1.5 sigma (this seems to be the logic behind the “benchmark Z” used in some software packages these days.) So…now six sigma is actually three sigma by default? What’s the point?
I recently taught a Six Sigma certification exam prep course for my local ASQ chapter, and the primer for that course—a popular reference used by a huge number of applicants for that ASQ certification—suggested that, given binomial data (a proportion defective), you should use a log transformation to force the data into an approximation of the Poisson. I don’t know why anyone would do this, unless you really have to be able to have that negative process sigma and a DPMO of more than one million. For years, I have been doing just the opposite; transforming Poisson data to Binomial using e-DPU to estimate DPMO.
This brings up another excellent point from your authors: we need to figure out how we are going to count units and opportunities. My own belief is that we should limit ourselves to definitions that end up providing an estimate of proportion defective. An opportunity, in that case, would be the most discrete thing we could count; in other words, there could be no more than one defect per opportunity. This would get rid of the “negative sigma” nonsense.
I strongly endorse their recommendation that we use DPMO. The procedure they outline for finding DPMO is straightforward and useful. Calculating DPMO this way would provide a reasonable estimate. If we want to err on the side of safety, we might continue to use the 1.5 sigma shift for high-volume processes, or no shift for lower-volume processes. DPMO is a more intuitive metric, and would keep people from having to go to the table to translate DPMO to process sigma, and then decode it again later for anyone who wants to know what it means. That’s unnecessary rework, something we’d all like to avoid.
Wednesday, October 14, 2009
Six Sigma Metrics...Why?
I have a client who is in a tizzy over whether they are recording short-term or long-term process sigmas. They are not high-speed, high-volume manufacturers, so I told them just use DPMO; don't worry about process sigma. It's not an intuitive metric anyway, nor is it anything like an accurate estimate of what to expect. And once you start putting "long-term" and "short-term" stuff in, you end up with really stupid non-intuitive discussions. I have yet to see an explanation of "short-term sigma using long-term data" that helps anyone understand anything. I'm not quite in the school of the purists who believe that, if you have a process in control, you don't have any undetected shifts. I can show you any number of real data sets to prove that...even using all four of the Western Electric Zone tests. In the interests of trying to maintain some standard metric with the rest of the Six Sigma world, I have had all my clients who were interested in using process sigmas use the standard Motorola tables or the calculator at isixsigma.com. This calculator takes your number of defects and opportunities and looks up a Sigma, applying the 1.5 Sigma shift. The assumptions for the calculator say that it assumes long-term data but provides a short-term sigma. Why should we care, and how does this terminology help anyone or make sense? Long-term data are supposed to be data that come from a process that has run long enough for some shifts to have taken place. If we're going to talk long-term, what we should use for an operational definition is "data that display a significantly different overall standard deviation than that of its local dispersion statistics." What all this is intended to provide is an estimate of what we can expect from a process in the future, given a stable process. Isn't the idea derived from a capability study? Essentially a cpk of 2 (process mean six sigma units from the nearest specification limit) equated to "Six Sigma Quality." If you buy the Motorola 1.5 sigma shift, then you gig yourself a sigma and a half, so it's really 4.5 sigma; instead of 2 ppb defective, you get 3.4 ppm. Now, there are a lot of people who object to predicting parts per million non-conforming from a capability study; I'm not one of those, as long as everyone involved realizes that we can't take any of those predictions too literally or assign too much precision. If I were claiming a process sigma of 6, and a count of defectives in the next million opportunities turned out to be 5 (or even 10 or 15), I wouldn't re-assess my sigma.Another thing I've seen lately is using a transformation to transform perfectly good data to poisson data, then deriving a DPMO and sigma from that. Now there's a great example of what Don Wheeler would call "a victory of computation over common sense." If I have 100 defective parts in a run of 1000, I have 10 percent defective. Assuming that 10% is stable over time, That equates very simply to a DPMO of 100,000. This is also assuming one opportunity per unit. It does make sense to me to do the opposite...if I'm getting more than one defect per unit (so I probably have Poisson data), it makes sense to me to transform the data using e^-dpu; that gives me an approximation to the binomial that lets me estimate DPMO.My biggest question, though, is why--other than to comply with a very short-lived tradition--should we use process sigma at all? It certainly doesn't provide any more information than DPMO, and we always have to translate it into DPMO anyway. If we must continue to process sigma, can we please just can all the short-term/long term stuff and (in a nod to standardization, even if controversial), just assume that shift happens and calculate DPMO from a stable process and use the sigma table? Maybe I'm wrong, and there is some real great reason to continue doing this, but I need an explanation and justification that makes sense. It seems to me that a lot of this is arbitrary and unnecessary.
Wednesday, September 2, 2009
During a recent discussion in the LinkedIn Deming HR group, one of the discussants posted the following link:
http://www.ted.com/talks/view/id/618
I can't recommend it too highly. It's a talk by Daniel Pink, about the science associated with rewards for performance. Anyone who was a follower of Deming, and anyone who has read Alfie Kohn, is already familiar with the concept that reward for performance can be harmful. Daniel's discussion, especially his piece about the candle problem, is an eye-opener. I would like to try that experiment at a conference or with a large class sometime.
My question or concern is the same as Daniel's: why do we continue to do, in business, what the science says is exactly the wrong thing to do? In its most public fashion, we do it on a grand scale with CEO compensation.
Daniel does a good job of pointing out that pay for performance, when it's linked to any job that requires thinking or problem solving, does more harm than good. What his talk doesn't cover, though, is the systems thinking aspect of this topic; the fact that you can't measure the performance of anyone in isolation. It's often the system that creates most of the performance we attribute to individuals.
This is certainly a worry in education these days, with many government officials pushing for performance pay for teachers. Stuck in the old carrot and stick paradigm, with nothing to go on for metrics but aggregate standardized test scores, these schemes will go a long way toward further suboptimizing our education system. Instead, put the money you might use for rewards into building systems like those built by Geoffrey Canada in the Harlem Children's Zone. Canada showed that by taking a systems approach you can improve the performance of even the most underpriveleged student populations, and put those students on an even playing field with the most priveleged.
http://www.ted.com/talks/view/id/618
I can't recommend it too highly. It's a talk by Daniel Pink, about the science associated with rewards for performance. Anyone who was a follower of Deming, and anyone who has read Alfie Kohn, is already familiar with the concept that reward for performance can be harmful. Daniel's discussion, especially his piece about the candle problem, is an eye-opener. I would like to try that experiment at a conference or with a large class sometime.
My question or concern is the same as Daniel's: why do we continue to do, in business, what the science says is exactly the wrong thing to do? In its most public fashion, we do it on a grand scale with CEO compensation.
Daniel does a good job of pointing out that pay for performance, when it's linked to any job that requires thinking or problem solving, does more harm than good. What his talk doesn't cover, though, is the systems thinking aspect of this topic; the fact that you can't measure the performance of anyone in isolation. It's often the system that creates most of the performance we attribute to individuals.
This is certainly a worry in education these days, with many government officials pushing for performance pay for teachers. Stuck in the old carrot and stick paradigm, with nothing to go on for metrics but aggregate standardized test scores, these schemes will go a long way toward further suboptimizing our education system. Instead, put the money you might use for rewards into building systems like those built by Geoffrey Canada in the Harlem Children's Zone. Canada showed that by taking a systems approach you can improve the performance of even the most underpriveleged student populations, and put those students on an even playing field with the most priveleged.
Monday, August 10, 2009
What Happened to the Deming Philosophy?
This is taken from part of a discussion on LinkedIn. Rafael Aguayo, a Consultant in Quality, Management and Strategy and Instructor at Stony Brook University, discussed this history and posed these questions today. See my response below.
Rafael's Post:
Two people in the same situation can have very different experiences. So let's consider some more objective measures of what has occurred. In the late 1980s and early 1990s Quality in the US was largely associated with Deming. Media articles regularly referred to Deming as the preeminent quality expert. Success at Ford, Harley Davidson and many other companies, that had adopted some or much of Deming's ideas, created interest and excitement in quality and other people and movements tried to position themselves as the next big thing. Specifically I would mention Reengineering and Six Sigma. At the time there were many successes accomplished under the banner of TQM and Six Sigma was a minor influence.
My assessment is that Deming-influenced quality represented 50% of the market. Yes, that is subjective, but I think anyone who was active at the time would have said that his influence and recognition was profound. Given the successes,the publicity and seeming strength of the quality movement I would have expected that by 2009 every hospital and US corporation to have been talking red beads and funnels in addition to statistical tools.
Instead, when I joined this group there were 171 members, while the Lean Six Sigma group had about 30,000. That translates to a market share of .57%. Even allowing an order of magnitude error this represents but 5.7% of the market. I have to ask what happened?
The logic of marketing is very different from formal logic. When GM discarded the Oldsmobile brand they expected those buyers to buy other GM cars. Instead they went elsewhere. If a significant amount of success were achieved under the banner of TQM and the brand is then disavowed then those successes are disavowed. At the time I observed the disappearance of some successful and respected consulting firms, such as Joiner Associates. And the interest in Deming shrank precipitously. It is possible that this outcome was inevitable. Once Jack Welch endorsed Six Sigma it may have been inevitable. Also the fiasco of Reengineering that some organizations, including ASQ, implicitly endorsed could not have helped. But whatever the causes the current reality is disappointing. While you say that has not been “your experience” your actions say something very different. By obtaining black belt certification and selling your services as such you implicitly acknowledged that Deming or SoPK is not a viable marketing brand.
It is not just that the market for a more profound understanding of quality has shrunk. Today young people who appreciate the importance of quality and process need not even once see a demonstration of the Red Beads or the Funnel. They can go out and do their best, blissfully unaware that their actions are tampering, on a massive scale. Luckily there are still many people laboring in schools and in firms with a deeper appreciation of the fundamentals.
Maybe I am hallucinating, but the reality of today is so far from what I would have expected that I must ask the question what happened? And what can be done to turn the situation around?
My response:
I'd like to add a comment to at least offer my observations in answer to Rafael's question: "What happened?"
It is, of course, not an easy thing to determine. Part of it was some admitted hubris on the part of those of us who were, or aspired to be, "Deming Disciples." One of the things we admired about Deming was his unyielding and unflinching ability to speak truth to power. He was often seen as curmudgeonly in his approach, but he never let anyone doubt that he didn't suffer fools gladly, and he was unabashed about putting anyone--including CEOs--into that category, if they offered any evidence that they belonged there. He was also very compassionate and thoughtful, and freely offered help and advice to anyone interested in learning. He just didn't have much patience for those who thought they had nothing left to learn. So he was a bitter pill for many CEOs to swallow. They did it, when they thought he could help; and, as Raphael pointed out, for a while Deming was the one person that almost everyone relied on for help.
Once he was gone, and the crisis of the 80's was over, Jack Welch and others were selling Six Sigma--not as a Quality initiative, but as a cost-cutting one--I think many of them jumped at Six Sigma because it seemed simpler, more prescriptive, more programmatic, less lofty and philosophical...maybe instant pudding. They certainly didn't have use for those Deming practitioners who (without Deming's extensive background or credibility) tried to act as Deming had. I have had Quality executives from major corporations tell me that "Deming was just a philosophy," implying that it was pie-in-the-sky, without any practical use for business. It's hard to get these people to listen to you after you explain how ignorant a statement that is...
Another thing that happened is that Six Sigma provides a roadmap that actually does work, when used well. Many companies had a lot of success with their Six Sigma projects. GE had some highly vaunted and publicized success...I will never know how much of it was real, because between making it mandatory and "firing the bottom 10%," who knows which GE numbers can be trusted? In any case, these projects can be very effective, when used as one component of an overall Quality Management System.
I think Rafael's insight about marketing is a good one. Many Deming practitioners were blindsided by Six Sigma, saw its statistical and other flaws, and concluded that it was the enemy, not worthy of consideration. We did get out-marketed, because we had no champion like Welch or Bossidy or Galvin touting huge success stories; most of the stories in Quality Progress and Quality Digest were about Six Sigma. Virtually all the mainstream business literature abandoned Quality; the only mention of it was the occasional Six Sigma story. Then Lean reared its head, and perversely became a competitor to Six Sigma.
When I joined Process Management International, they were working to develop a Deming-based Six Sigma methodology. We had people with a strong Deming foundation who had worked for Motorola and GE, and I think we were successful, with a sound methodology, presented as one set of tools in an overall tranformational approach, that took into consideration all the aspects of SoPK and the 14 points. At least we were able to continue to tell people about Deming, the SoPK and the 14 points, to show the Red Bead and the Funnel. Interestingly, during a conference that included a lot of the Deming and JUSE elite, a consensus position was developed that saw Six Sigma as a [marketing] "vehicle" for quality...a way to explain it and to act as a lever for change, a foot in the door.
Would I have been happier teaching and consuling in "pure" Deming? That's all I wanted to do when I first retired from the Navy. No one was hiring for that, though, because the jobs for consultants who did that were few and far between. In any case, Would I still do it? You bet...I do, as much as I can.
What are your thoughts?
Rafael's Post:
Two people in the same situation can have very different experiences. So let's consider some more objective measures of what has occurred. In the late 1980s and early 1990s Quality in the US was largely associated with Deming. Media articles regularly referred to Deming as the preeminent quality expert. Success at Ford, Harley Davidson and many other companies, that had adopted some or much of Deming's ideas, created interest and excitement in quality and other people and movements tried to position themselves as the next big thing. Specifically I would mention Reengineering and Six Sigma. At the time there were many successes accomplished under the banner of TQM and Six Sigma was a minor influence.
My assessment is that Deming-influenced quality represented 50% of the market. Yes, that is subjective, but I think anyone who was active at the time would have said that his influence and recognition was profound. Given the successes,the publicity and seeming strength of the quality movement I would have expected that by 2009 every hospital and US corporation to have been talking red beads and funnels in addition to statistical tools.
Instead, when I joined this group there were 171 members, while the Lean Six Sigma group had about 30,000. That translates to a market share of .57%. Even allowing an order of magnitude error this represents but 5.7% of the market. I have to ask what happened?
The logic of marketing is very different from formal logic. When GM discarded the Oldsmobile brand they expected those buyers to buy other GM cars. Instead they went elsewhere. If a significant amount of success were achieved under the banner of TQM and the brand is then disavowed then those successes are disavowed. At the time I observed the disappearance of some successful and respected consulting firms, such as Joiner Associates. And the interest in Deming shrank precipitously. It is possible that this outcome was inevitable. Once Jack Welch endorsed Six Sigma it may have been inevitable. Also the fiasco of Reengineering that some organizations, including ASQ, implicitly endorsed could not have helped. But whatever the causes the current reality is disappointing. While you say that has not been “your experience” your actions say something very different. By obtaining black belt certification and selling your services as such you implicitly acknowledged that Deming or SoPK is not a viable marketing brand.
It is not just that the market for a more profound understanding of quality has shrunk. Today young people who appreciate the importance of quality and process need not even once see a demonstration of the Red Beads or the Funnel. They can go out and do their best, blissfully unaware that their actions are tampering, on a massive scale. Luckily there are still many people laboring in schools and in firms with a deeper appreciation of the fundamentals.
Maybe I am hallucinating, but the reality of today is so far from what I would have expected that I must ask the question what happened? And what can be done to turn the situation around?
My response:
I'd like to add a comment to at least offer my observations in answer to Rafael's question: "What happened?"
It is, of course, not an easy thing to determine. Part of it was some admitted hubris on the part of those of us who were, or aspired to be, "Deming Disciples." One of the things we admired about Deming was his unyielding and unflinching ability to speak truth to power. He was often seen as curmudgeonly in his approach, but he never let anyone doubt that he didn't suffer fools gladly, and he was unabashed about putting anyone--including CEOs--into that category, if they offered any evidence that they belonged there. He was also very compassionate and thoughtful, and freely offered help and advice to anyone interested in learning. He just didn't have much patience for those who thought they had nothing left to learn. So he was a bitter pill for many CEOs to swallow. They did it, when they thought he could help; and, as Raphael pointed out, for a while Deming was the one person that almost everyone relied on for help.
Once he was gone, and the crisis of the 80's was over, Jack Welch and others were selling Six Sigma--not as a Quality initiative, but as a cost-cutting one--I think many of them jumped at Six Sigma because it seemed simpler, more prescriptive, more programmatic, less lofty and philosophical...maybe instant pudding. They certainly didn't have use for those Deming practitioners who (without Deming's extensive background or credibility) tried to act as Deming had. I have had Quality executives from major corporations tell me that "Deming was just a philosophy," implying that it was pie-in-the-sky, without any practical use for business. It's hard to get these people to listen to you after you explain how ignorant a statement that is...
Another thing that happened is that Six Sigma provides a roadmap that actually does work, when used well. Many companies had a lot of success with their Six Sigma projects. GE had some highly vaunted and publicized success...I will never know how much of it was real, because between making it mandatory and "firing the bottom 10%," who knows which GE numbers can be trusted? In any case, these projects can be very effective, when used as one component of an overall Quality Management System.
I think Rafael's insight about marketing is a good one. Many Deming practitioners were blindsided by Six Sigma, saw its statistical and other flaws, and concluded that it was the enemy, not worthy of consideration. We did get out-marketed, because we had no champion like Welch or Bossidy or Galvin touting huge success stories; most of the stories in Quality Progress and Quality Digest were about Six Sigma. Virtually all the mainstream business literature abandoned Quality; the only mention of it was the occasional Six Sigma story. Then Lean reared its head, and perversely became a competitor to Six Sigma.
When I joined Process Management International, they were working to develop a Deming-based Six Sigma methodology. We had people with a strong Deming foundation who had worked for Motorola and GE, and I think we were successful, with a sound methodology, presented as one set of tools in an overall tranformational approach, that took into consideration all the aspects of SoPK and the 14 points. At least we were able to continue to tell people about Deming, the SoPK and the 14 points, to show the Red Bead and the Funnel. Interestingly, during a conference that included a lot of the Deming and JUSE elite, a consensus position was developed that saw Six Sigma as a [marketing] "vehicle" for quality...a way to explain it and to act as a lever for change, a foot in the door.
Would I have been happier teaching and consuling in "pure" Deming? That's all I wanted to do when I first retired from the Navy. No one was hiring for that, though, because the jobs for consultants who did that were few and far between. In any case, Would I still do it? You bet...I do, as much as I can.
What are your thoughts?
Subscribe to:
Posts (Atom)
.jpg)
