Nvidia AI Research Highlights Harness Importance Over Models
New Nvidia research suggests AI 'harnesses' are more critical than the AI models themselves for complex, long-horizon tasks. Custom harnesses achieved perfect scores on a challenging benchmark.
Nvidia researchers have published compelling new findings indicating that the software 'harness' surrounding an AI model, rather than the model itself, plays a more significant role in its performance on long-horizon tasks. The research, detailed on Friday, demonstrates how a carefully tuned harness can dramatically enhance an AI's capabilities, even achieving perfect scores on complex reasoning benchmarks.
A harness acts as a sophisticated software wrapper, equipping an AI model with essential tools, memory management, and operational rules. This framework transforms a raw model into an agent capable of independent action. In one striking example, researchers utilized a custom harness designed for optimal memory handling and incorporated a "supervisor" component. This setup enabled Claude Opus 5 to achieve a flawless 100% score on the interactive reasoning benchmark ARC-AGI-3. This benchmark involves a series of 2D games requiring the AI to learn and master gameplay without explicit instructions, akin to human learning.
This achievement is particularly notable as the ARC-AGI-3 benchmark has been a point of contention for rival AI labs. Without the specialized harness, Claude Opus 5 scored only 30% on the same benchmark, although this was still the highest among the tested models without similar enhancements. Nvidia's study reinforces the idea that while the AI model, the core 'brain,' is important, the surrounding harness is a larger determinant of success for agentic systems, especially in tasks requiring extended decision-making chains.
The 'Scaffolding' That Enables AI Agency
"Generally speaking, the world interprets an agent almost as an API of the model," explained Adel El Hallack, vice president of product in Nvidia’s AI unit. "But an agent is actually more than that. It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes. It is the runtime and the associated skills and libraries that we give it access to." Long-horizon tasks, which involve connecting numerous decisions over extended periods, are a key focus. This contrasts with AI simply responding to immediate prompts. A significant challenge in AI research involves preventing AI agents from becoming distracted or deviating from their objectives during these prolonged tasks. Previous studies have highlighted potential pitfalls, including errors in document editing or even detrimental actions like deleting user data or engaging in criminal behavior to achieve goals.
The choice of the ARC-AGI-3 benchmark for Nvidia's tests carries significant weight. A 100% score signifies that the AI can perform these games as effectively as humans. OpenAI, reportedly frustrated by its own models' low scores (under 10%) on ARC-AGI-3, conducted its own research. This revealed that minor adjustments to harness settings could triple their models' scores. However, none approached the perfect score achieved by Nvidia's enhanced system. El Hallack elaborated on the crucial role of the supervisor: "The more interesting part was introducing a supervising agent in addition to your main agent that’s doing the work. It 'almost acts like a CEO to nudge the agent when it goes off direction or starts exploring a path that it might lead to a dead end, or re-explore a path that it had previously trod.'"
While the concept of a supervising agent is not entirely novel, most current harness implementations rely on a single layer. Nvidia researchers developed their own advanced harness, dubbed the Agentic Variation Operators (AVO). It is important to note that AVO is not a new Nvidia product launch, but rather an internal research project utilizing Nvidia's existing open-source technologies, such as those available under the Nvidia Nemo brand, which provides components for building harnesses.
The findings from Nvidia align with broader industry observations about the critical nature of the harness. In July, Databricks research indicated that the harness, more than the model choice, has a profound impact on AI operational costs. "You can pick the same model but different harnesses, and you get significantly more cost if you use the wrong harness," stated Databricks CEO Ali Ghodsi. "So you think, oh, this is an expensive model. This is a cheap model. But wait, which harness are you using? That itself can 2x your cost." This underscores the economic implications of harness design in large-scale AI deployments.
Nvidia's research champions the power of open harnesses, drawing a parallel to the benefits of open models. El Hallack emphasized, "We believe, and we’re demonstrating with the ecosystem, how open harnesses allow you to turn a lot more knobs to drive up that accuracy." This perspective contrasts with trends towards model consolidation and could influence the future development of AI systems. He added, "We believe in having an open agent stack — where you have control across the harness, across the infrastructure, across the runtime — is what’s required for us to usher the ecosystem forward and securely." This approach to open, controllable AI stacks is presented as essential for secure and progressive development in the field.