Testers of AI-based systems often face a test oracle problem – the challenge of generating expected results for their test cases. AI-based systems can suffer this problem for several reasons. First, many AI-based systems are poorly specified; it is common to fund AI projects with no clear objective other than to exploit the new AI technology. Second, AI-based systems are used where the problem complexity means that conventional systems are ineffective; this inherent complexity often means testers struggle to generate expected results. Third, much AI technology is self-learning and probabilistic, meaning that systems are naturally non-deterministic; so a result generated today will often differ from the result generated yesterday for the same test.
Together, these three factors can make the generation of test cases time-consuming and expensive. Even when domain experts are available, it can take an extremely long time to generate enough test cases to justify test automation. Similarly, for non-deterministic systems, a simple regression test framework is likely to fail due to the changing outputs, meaning that sophisticated test cases and frameworks are needed to cope with the uncertainty.
For some scenarios, there are test techniques that address these difficulties. Back-to-back testing, where we use another system with similar behaviour as our test oracle, is possible. A/B testing can be used if we only need to see improvements over an existing system. However, for many situations, testers of AI-based systems need a generic test technique that applies when a similar system is not available – this is where metamorphic testing becomes a must-have addition to a tester’s toolset of test techniques.
Metamorphic testing generates new test cases from an existing test case using a form of metamorphosis (a change of the form of a thing). We make changes to test inputs and expected results using metamorphic relations that are based on the known required behaviour of the test item. A simple example could be the testing of a program summing two positive integers. A suitable metamorphic relation might state that if we increase the value of one of the test inputs by a known amount, then the expected result should increase by a similar amount. To generate a second (follow-on) test case, we only need a valid source test case and the metamorphic relation, and, in this case, we can then generate a near infinite number of follow-on test cases by simply changing the amount we increase the test input.
In this presentation, I will explain the test oracle problem for AI-based systems and how metamorphic testing can be used to address it. I will show how metamorphic testing can be applied to the testing of a variety of AI-based systems and where test cases would be challenging to generate using conventional test techniques. I will also propose a test framework to support the automation of metamorphic testing and present data that show it can be used to effectively find defects in a cost-efficient manner.
307, Commerce House, Cunningham Road, Bangalore 560052
10:00 AM – 7.00 PM
Monday – Friday
Phone: +91 99101 05147
© STeP-IN Forum All Rights Reserved 2021