WEF Discussion Forums
Laboratory Management and Technical Issues
I am devolping control charts for my BOD testing. I run a GGA standard with every batch of BOD testing, and I never have a problem getting the standard to be with +/- 15 % of 198. I would like to develop a control chart that sets Upper and Lower contol limits based on my historical standard results. Is it correct that an "up-to date" control chart has 20 data points on it? Or do you keep a control chart with all standard results on it? That would get to be a very large chart. Any suggestions, advice, explanation or references for how to create accurate control charts would be greatly appreciated.
I'm glad to see that you are emphasizing the BOD test when talking about control charts, LabLady! If there is a test that can go wrong in a heartbeat, it's BOD! Using a control chart to monitor performance for BOD testing is a good way to catch a potential problem before it takes you out of control.
I'll be glad to give you my recommendations along with my rationale for making them, but at the same time, I will tell you that some regulatory and accreditation agencies have different thoughts on the subject.
I recommend constructing your control chart based on 20 results where you think the lab has been doing a pretty good job. the National Environmental Lab Accreditation Program (NELAP or INELA) requires 25, but after 20, the statistics change very little unless there is a significant blunder in those 5 extra ones. And speaking of "blunders"...results that are WAY out of whack... I advise labs to take 22 consecutive results, and omit the highest and lowest as arbitrarily being "non-representative". Doing so will give you a more realistic base for constructing the control chart.
Using the mean and standard deviation from those 20 results, you construct the control chart. I recommend using upper and lower warning limits that are 2 standard deviations from the mean, and action limits that are 3 standard deviations. Some regulators/accreditors base action limits on 2.5 standard deviations, but doing that will result in enough seemingly "out-of-control" situations that can be attributed to random error, which you cannot eliminate, that you will be spending a lot of time for nothing. Using 3 standard deviations, only 3 in 1000 results will be "out-of-control" because of random error. So when one of your results is outside the 3s limit, it is statistically most likely that it was a blunder rather than random error that caused it, and blunders are what you want to find the cause of, and eliminate if possible.
OK...you have the control chart constructed and now you are going to use it to monitor performance. Some control charting programs (computerized) are written to calculate a new mean and standard deviation every time you record a new result. The main reason, IMHO, for using control charts is to monitor for trends that are leading toward an out-of-control situation (or perhaps indicate you are doing better!), and when the mean and standard deviation are changed with each result, it weakens the chart's ability to spot trends. It's like shooting at a target, finding you missed the bullseye by a foot, and going down range to change the location of the bullseye so the next shot will be closer to it! Sounds ridiculous, doesn't it, but in essence, that's what you would be doing.
Others recommend changing the statistics and building a new chart after a certain number of tests (maybe 30) or after a certain time period (maybe quarterly). I recommend not changing the statistics until the chart shows you are doing a better job...e.g., the that average results is getting closer to 198 mg/L, or that the results are getting closer together (indicating a smaller standard deviation). When you note such a good change, you build a new chart. Your first thought might be that you will eventually get to the point where you can't keep your results within the action limits. Statistically, that can't happen. And I can almost guarantee you that results will get better after you start using a control chart in such a manner...it's the nature of the human spirit to want to improve. The next thing you know, you will have a chart that shows you are doing such a good job that you will want to invite the mayor or CEO to the lab to take a look.
What if your chart shows a bad trend...for example, consecutive results that show a downward trend going away from 198 mg/L. In such a case, you don't change the statistics used to construct the chart...you find the cause of the downward trend and eliminate it. A classic, real case...a commercial lab was using a seed from a nearby WWTP, keeping it in the incubator and feeding it starch to keep the bacteria happy. After a few weeks, they would fall below what they used as a lower acceptance limit, go get some new seed, and, voila...their results would snap right back. But in the meantime, they had to (or SHOULD have!) disqualify the results for that batch where the GGA result tanked out. I constructed a control chart using their data and it showed an easily recognized downward trend in twelve or so results prior to each of their many periodic out-of-control situations. I convinced them to use control charts. Three years later, I inspected the same lab and it hadn't been out-for control on the low side since the initial visit.
If you would like an Excel control charting program that does all the dirty work based on the concepts above, send me an e-mail...firstname.lastname@example.org.
It must be the early morning coffee, but I'm going to be picky-picky with Perry's most excellent answer. (I'm going WAY out on a limb here, picking a "fight" with Perry...I'm well aware of the big difference between knowledge and wisdom...unfortunately, I come up a little short on the latter.)
I wouldn't automatically toss ANY result unless you have a good reason. What's a good reason? The probe spontaneously exploded, a gerbil fell into the BOD bottle, I accidentally read my coffee instead of the sample...you get the idea.
It's my opinion that you're biasing your statistics if you just toss data.
Okay, let the food fight begin. I'm hiding under my desk.
Throw #1 - An over-ripe tomato.
Missed! Dadgum it! Guess I'd better not continue this contest!
Scientifically, you are absolutely correct, Chuck. From a practical standpoint, however, I've seen many folks just getting started in control charting who have based their statistics on a group of 20 (or 25) results that obviously include one result this is a "flier". Or, if not technically a "flier", obviously not representative of their normal performance. Rather than embarking on a seminar on statistics, I simply advise throwing out the highest and lowest. You are correct that this will bias the data, but it will skew it toward tighter control limits, and that in my mind is a good thing...makes the analyst try harder to continue at that higher performance level.
Notice that I didn't recommend omitting ANY results once the chart is constructed. Even if a result is recognized as having been caused by that pet gerbil in the BOD bottle, the analyst should keep that glaring result on the control chart as a reminder to keep the gerbils in their cage.
OK...I'm out of over-ripe tomatoes and have no rotten eggs...
Your humble student,
There's nothing worse than winning a battle and losing the war. I concede the war before the fighting begins! Perry, call back the tomato launchers. Since I raised the issue in a "public" forum, I'll fall on my sword in public.
Perry's right on the money. I've been reading Dennis Helsel's "Statistical Methods In Water Resources," and it's clouded my mind. Excuses, excuses.
Thanks for such a thorough explanation Perry. When constructing my control chart, is it best to use my 22 most recent GGA standard results (throwing out the highest and lowest result)? Or just randomonly pick a date range for the 22 consecutive results? I will email you to get that excel program.
If there are 22 results, even if they are somewhat in the past, that you think are representative of you performance, I would say go ahead and use them to construct your initial chart. Remember that if they weren't your best work, you will soon see a "good" trend in the latest points plotted on the chart. When you notice that improvement, you use the latest 20 points to construct your new chart, even if those 20 include some you had used in calculating the mean and standard deviation for the original chart.
You can do that as often as you want to, but at some point, you will say "this is as good as I'm going to get" and then settle in with THE chart you will use for a longer period. At some point, you might again improve...maybe after purchasing an LDO probe/meter, for example. You'll know when it's time to change to a new chart.
Chuck, as I said before, scientifically you are correct, so there is no need to fall on your sword. And also, you need to rethink your salutation! It must be nice being both intelligent AND wise!!