An artificial intelligence system has successfully solved a complex mathematical problem, originally designed 20 years ago to test the limits of AI capabilities.