Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

Martin Udengaard wants more for his son, and he doesn’t think Cambridge schools can deliver. Cambridge Public Schools no longer offers advanced math in middle school, something that could hinder his ...