Testers of AI-based systems often face a test oracle problem – the challenge of generating expected results for their test cases. AI-based systems can suffer this problem for several reasons. First, many AI-based systems are poorly specified; it is common to fund AI projects with no clear objective other than to exploit the new AI technology. Second, AI-based systems are used where the problem complexity means that conventional systems are ineffective; this inherent complexity often means testers struggle to generate expected results. Third, much AI technology is self-learning and probabilistic, meaning that systems are naturally non-deterministic; so a result generated today will often differ from the result generated yesterday for the same test.

Together, these three factors can make the generation of test cases time-consuming and expensive. Even when domain experts are available, it can take an extremely long time to generate enough test cases to justify test automation. Similarly, for non-deterministic systems, a simple regression test framework is likely to fail due to the changing outputs, meaning that sophisticated test cases and frameworks are needed to cope with the uncertainty.

For some scenarios, there are test techniques that address these difficulties. Back-to-back testing, where we use another system with similar behaviour as our test oracle, is possible. A/B testing can be used if we only need to see improvements over an existing system. However, for many situations, testers of AI-based systems need a generic test technique that applies when a similar system is not available – this is where metamorphic testing becomes a must-have addition to a tester’s toolset of test techniques.

Metamorphic testing generates new test cases from an existing test case using a form of metamorphosis (a change of the form of a thing). We make changes to test inputs and expected results using metamorphic relations that are based on the known required behaviour of the test item. A simple example could be the testing of a program summing two positive integers. A suitable metamorphic relation might state that if we increase the value of one of the test inputs by a known amount, then the expected result should increase by a similar amount. To generate a second (follow-on) test case, we only need a valid source test case and the metamorphic relation, and, in this case, we can then generate a near infinite number of follow-on test cases by simply changing the amount we increase the test input.

In this presentation, I will explain the test oracle problem for AI-based systems and how metamorphic testing can be used to address it. I will show how metamorphic testing can be applied to the testing of a variety of AI-based systems and where test cases would be challenging to generate using conventional test techniques. I will also propose a test framework to support the automation of metamorphic testing and present data that show it can be used to effectively find defects in a cost-efficient manner.

Dr Stuart Reid is Chief Technology Officer at STA Consulting in Seoul with over 38 years’ experience in the IT industry, working in development, testing, and education. Application areas range from safety-critical to financial and media.

Stuart supports the worldwide testing community in a number of roles. He is convener of the ISO Software Testing Working Group, which has published the ISO/IEC/IEEE 29119 series of software testing standards and is editor for the ISO Technical Report on the testing of AI-based systems. He also founded the International Software Testing Qualifications Board (ISTQB) to promote software testing qualifications globally and is currently part of the task force working on a new certification on testing AI-based systems.