Sequential Reliability Test Calculator

A Sequential Reliability Testing Calculator was recently added to the Reliability Analytics Toolkit. Sequential testing often provides a more efficient method to verify equipment reliability achievement. Really “good” equipment will be accepted much quicker and really “bad” equipment will be rejected much sooner, often resulting in fewer test hours needed than using a military handbook 781 fixed length reliability test. The tool provides the ability to plan a sequential reliability demonstration test for verification of equipment mean time between failure (MTBF) if it can be assumed that the equipment follows an exponential failure distribution (i.e., constant failure rate). Input parameters include the following:

  • Lower test MTBF (θ1). The test plan will reject an item whose true MTBF is θ1 with a probability of 1 – β.
  • Upper test MTBF (θ0). The test plan will accept an item whose true MTBF is θ0 with a probability of 1 – α.   θ0 = d * θ1.
  • Discrimination ratio (d). The discrimination ratio is one of the standard test plan parameters which establish the test plan envelope. d = θ01.
  • Consumer’s risk (β). The consumer’s risk is the probability of accepting equipment with a true MTBF equal to the lower test MTBF (θ1).
  • Producer’s risk (α). The producer’s risk is the probability of rejecting equipment with a true MTBF equal to the upper test MTBF (θ0).
  • True MTBF. The MTBF that would be observed if an infinite number of units were tested for an infinite amount of time.

Example 1
Suppose that a system had a specified lower test MTBF of 100 hours.  A test discrimination ratio of 1.5 is chosen along with a 5% consumers and producers risk:

The above inputs result in the test plan summarized below. The plan requires a minimum time to accept of 883 hours if no failures occur and a maximum time to accept of 6,800 hours if the test ends in truncation. The red line in the sequential test plot is the reject line while the green line is the accept line. Any failures occurring during testing are plotted on the graph and if this plot crosses the green line, the equipment accepted will have a true MTBF of greater than 100 hours with a 0.95 probability (i.e., 1 – β).

Simulating Testing
For planning purposes, the tool allows the user to simulate testing by assuming a true MTBF and generating simulated failure times using a random number generator. Simulated failure times are generated using the reliability function for the exponential failure distribution and the random number generator associated with the Python programming language. Modifying the inputs to simulate three tests and assuming a true system MTBF is 150 hours:

The resulting simulations are shown below in both plot and summary table form.  For a true MTBF of 150 hours, the average test time for the three simulations is 3,803 hours.

If the true MTBF is changed from 150 hours to 250 hours and the simulation is re-run, the average test time drops to 1,978 hours – a really good MTBF is accepted much quicker, on average:  

If the true MTBF is much less than the 100 hour lower test MTBF, say 50 hours, the simulated tests all reach reject decisions relatively quickly.  On average, it takes 779 hours to reach a reject decision:

The tool also outputs the proposed test plan in tabular form, showing all possible accept/reject points and allows the option is to plot specific failure times that a user has entered for input 4A.

Increase Decision Risk to Lower Test Time
In the first example above, the minimum time to accept was 883 hours if no failures occurred and a maximum time to accept was 6,800 hours if the test ended in truncation. How can the test be shortened? If higher decision risks can be accepted then test time can be significantly shortened.  In the above example, if an accept decision was reached, there was a 0.95 probability that the true MTBF was greater than 100 hours.  What would the test characteristics be if this probability was 0.70 instead of 0.95?  Change the decision risks to 30%, as shown below:

Now, instead of a minimum test time of 883 hours, only 254 hours will be required if no failures occur. The maximum test time is now only 680 hours, as compared to 6,800 hours for the 5% decision risk test.  If the equipment reaches an accept decision, there is a 0.7 probability that the true MTBF is 100 hours, or better.  Conversely, there is a 0.3 probability that the true MTBF is less than 100 hours. These decision risks are highlighted in the operating characteristic curve, as shown in the second picture below.  If the true MTBF is 100 hours, there is a 0.3 probability of acceptance, as highlighted in the figure. However, if the true MTBF is equal to the upper test MTBF (θ0 = 150 hours, which is the lower test MTBF(θ1) times the discrimination ratio), then there is a 0.7 probability of acceptance.

 

Impact of Discrimination Ratio on Test Length
The discrimination ratio d is the ratio of the upper test MTBF to lower test MTBF, d = θ01. The farther apart θ0 is from  θ1 (i.e., larger d) the faster a test decision is reached. For example, all other things being equal, the figure below shows testing for the same lower test MTBF of 100 hours with the test on the left having a d=1.5 (θ0 =150) and the test on the right having a d=3.0 (θ0 =300). Although not visually intuitive because the plot scales are different, the area of indecision for the test on the left is far greater than the test on the right, specifically 92,552/6,798 = 13.6 times greater.  The conclusion is that a smaller  discrimination ratio results in a greater the area of indecision and a longer test. It takes more test time to discriminate between an upper and lower test MTBF if they are close together.

 

References

  1. Bazovsky, Igor, Reliability Theory and Practice.
  2. Wald, A., Sequential Analysis.
  3. Wald, A. (1945). Sequential tests of statistical hypotheses. Annual Mathematica Statistics. 16, 117-186.
  4. MIL-HDBK-781A, Reliability Test Methods, Plans, and Environments for Engineering Development, Qualification, and Production.
  5. Epstein, B., & Sobel, M. (1955). Sequential Life Tests in the Exponential Case. The Institute of Mathematical Statistics.