Determining the optimum allocation of resources in a portfolio of tests for a Department of Defense (DoD) system is a challenging problem, primarily due to the lack of an accepted and easily obtained value measure for test results. Past attempts to quantify the value that a test provides to a program manager have focused on prioritization schemes or estimates of cost savings postulated to occur by finding and fixing problems as early as possible. These methods have not gained traction, largely due to the difficulty of obtaining cost estimates and historical data. In addition, the use of a cost metric does not capture the true value of DoD testing, which is to reduce technical uncertainty and programmatic risk. This research proposed a methodology to determine test value by estimating the amount of uncertainty reduction a particular test is expected to provide using Shannon's Information Entropy as a basis for the estimate. The methodology was applied to multiple case studies consisting of a small test portfolio based on real-world flight test examples and a simulated large test portfolio. Conclusions from the research are that using uncertainty reduction as a test value measure is easy to apply, produces results that are intuitively appealing, provides insights about the system during the test planning process, is robust to sensitivities in the input variables (test value and cost), and is scalable to a large portfolio of tests that is typical for a major DoD weapon system program or complex commercial system.
|In Administrative Set: