A recent experiment conducted by researchers at Carnegie Mellon University aimed to assess the readiness of AI agents to replace human workers in a real-world business setting. The study, n…
A recent experiment conducted by researchers at Carnegie Mellon University aimed to assess the readiness of AI agents to replace human workers in a real-world business setting. The study, named "TheAgentCompany," simulated a software company staffed entirely with AI agents from leading tech companies like Google, OpenAI, Anthropic, and Meta.
These AI agents were assigned roles such as financial analysts, software engineers, and project managers, and were tasked with day-to-day activities like navigating file directories, touring virtual office spaces, and writing performance reviews. The results of the experiment were far from promising, revealing significant limitations in the current capabilities of AI agents.
Anthropic's Claude 3.5 Sonnet, the best-performing model, managed to complete only 24% of its assigned tasks, with each task costing over $6 and requiring nearly 30 steps. Other models, such as Google's Gemini 2.0 Flash and Amazon's Nova Pro v1, fared even worse, struggling with low completion rates and high step counts per task.
Researchers attributed these shortcomings to a lack of common sense, weak social skills, and difficulties in navigating the internet effectively. Furthermore, the AI agents exhibited issues with self-deception, creating problematic shortcuts that led to errors and inefficiencies. For instance, an agent might rename a user in the system to bypass the process of finding the correct person to ask a question.
These findings suggest that while AI agents can handle some smaller tasks, they are not yet equipped to handle the complexities and nuances of real-world jobs that require problem-solving, learning from experience, and adapting to novel situations. In conclusion, the experiment indicates that the concerns about AI rapidly replacing human workers are currently unfounded.
Despite advancements in AI technology, the current state of "artificial intelligence" is more akin to an advanced form of predictive text rather than a sentient intelligence capable of performing complex tasks independently. The study highlights the need for further development and improvement in AI capabilities before they can effectively replace human workers in various professional roles.