Risk of Recursive AI: Optimizing the Imperfect Objective
Recent discussions about slowing the development of artificial intelligence are often framed in dramatic terms: artificial general intelligence may become smarter than humans, recursively improve itself, and eventually escape human control. I find this framing somewhat misleading. Modern AI systems are not unconstrained intelligences. Transformer-based models operate within very real physical and computational limits: available compute, memory capacity and bandwidth, electrical power, training time, data quality, and the efficiency of optimization algorithms. Recursive self-improvement therefore does not imply that intelligence can increase without bound. The more interesting problem is different: What happens when an increasingly capable optimizer repeatedly optimizes an imperfect objective function without sufficiently independent external correction? We must consider AI as an optimization system. Consider a simplified learning system with parameters $\theta$ and objective function $...