Former Tesla AI Head: AI Discovers Crucial Tuning Combo Overlooked by Humans Overnight

On March 22, Andrej Karpathy, former head of Artificial Intelligence and Autopilot at Tesla, revealed that while training the GPT-2 model, after months of unsuccessful manual tuning, he handed over the optimization process to an autonomous agent. The AI found—overnight—a set of parameter combinations that humans had easily overlooked but which significantly impacted performance. These parameters exhibited complex interactions, demonstrating that systematic search outperformed human intuition. Karpathy urged researchers to deliberately "step back" in domains with objective evaluation metrics and leverage automated tools effectively. He noted that although AI has advanced rapidly in verifiable tasks like programming, its application remains challenging in areas that are difficult to quantify.