AverNotes: What Should You Be Testing?

What is the primary goal of your testing? This simple question is not asked enough. I have tested a lot of products but unless I know the goal of one’s testing effort, I can’t recommend what tests should be run. The best advice I can give in these situations is; it depends.

No single test methodology that works best for all products, in all markets and in all phases of its development. It is pretty obvious you would use different tests with dissimilar products. For instance, you probably wouldn’t to perform acoustic quality tests on a device that has no speaker or microphone. What is less obvious to many is that the suite of tests for any given product should (with rare exception) change at different points in a product’s life cycle. If fact, I would be hard pressed to identify a product that requires an identical test suite to be used in both the design phase and in full scale production.

Validation vs Verification

Validation and verification are synonymous terms in some contexts but in the context of product life cycle test requirements I am going to make a semantic distinction.

• Validation – The process of determining if a product meets its intended design goals.
• Verification – The process of determining if a manufacturing sample is acceptable.

Validation of a design is clearly the more rigorous of these two processes. To validating a design, it must be tested over the entire range of possible use scenarios. Validation of audio products will often include listening tests because no audio analyzer can tell you how a device really sounds (a topic worth its own article). There is also a plethora of abusive tests such as drop tests, electrical shock tests, interference tests, water immersion tests and more that may be required. You wouldn’t do these tests in production because no consumer would want to buy a product after it has been soaked and dropped. All-in-all, a rigorous design validation can be a lengthy process that takes many months.

Verification of a product during the manufacturing process needs to be a quick process. As such, it is also a much simpler process than validation. The purpose of verification is simply to insure that the product was assembled correctly and functions within key specifications. This level of reassurance only requires a cursory view of some critical parameters and exercising certain product controls. The goal should be completed in seconds to minutes depending on the degree of product complexity and failure mechanisms, product price and market tolerance for quality issues.

Choosing your test suite

The first step in developing a test suite is to make a conscious decision as to whether you are validating a design or verifying its proper manufacture. I can’t tell you what validation tests you’ll need because that depends on your product type, product price and the markets you serve. However, I can say that your verification suite only needs to be a small subset of your validation suite. Of course, this subset has the same dependencies as the validation phase in addition to the production failures you are likely to encounter and what you will do about those failures. Simple, inexpensive products for which rework is impossible or impractical will warrant different test regimens than multi-part assemblies that sell at a premium.

Such logic seems pretty basic but I have found in practice, many production line tests are significantly more exhaustively than they need to be. Each time I have pointed this out, I get pretty much the same explanation; no one really thought about it. This creates a hidden cost that is measured on a per unit basis rather than a per project basis.

I can’t tell you how great your hidden cost is because like everything else, it depends. In this case it depends on total test time, product volume and the length of the production run. However, the calculation of the testing cost and any possible savings is pretty straightforward. And, if a small percentage reduction in test time can produce a significant change in profitability, it is worth exploring the cost of streamlining your testing. Bring this problem to your test equipment vendors and you will see them salivate to meet your goals.

Paul Messick

About "Paul Messick"

Paul has more than 40 years of engineering experience in scientific instrumentation, radar and RF design, analog and audio design, and microprocessor based solutions. He started Avermetrics in 2011 to find a better way to test electronic products, both day-to-day on the bench and in high-volume factory production. In his spare time he is a truly awful guitar player, and with enough practice hopes to become merely terrible.