Remote Labor Index: AI Agents Automated 2.5% of Real Freelance Projects at Launch

The Remote Labor Index (RLI), published in October 2025 by the Center for AI Safety and Scale AI, tested AI agents on 240 real freelance projects across 23 categories of work, representing over 6,000 hours of work valued at over $140,000. The best agent, Manus, reached an automation rate of 2.5%: its output was judged at least as acceptable as the human deliverable on 2.5% of projects.^[1][2] The other agents tested scored between 0.8% and 2.1%. The paper's four failure modes, which overlap, are poor-quality output (45.6% of the AI deliverables analysed), incomplete deliverables (35.7%), corrupted or unusable files (17.6%) and inconsistencies across a project (14.8%). For executives, the lesson is about sequencing: autonomous agents are far from replacing whole projects, while targeted augmentation of specific workflows is where returns are realistic today.

The leaderboard has since moved: CAIS reported 15.8% for Fable 5 on 1 July 2026, and the Scale leaderboard listed 21.25% for Opus 5.5 as of 18 October 2026.^[3][4]

For CEOs, CIOs, CFOs, and CHROs, this is the data to trade guesswork for evidence when making billion-dollar calls on automation, reskilling, and future-proofing operations.

Let's cut to the chase—AI isn't just a technology trend, it's a strategic uncertainty. Most board-level conversations on AI center around abstract metrics or internal pilot results that don't reflect cross-sector economic value. Executives are being asked to approve budgets, launch rebuild programs, or sign off on headcount shifts without hard evidence of what AI can actually deliver.

RLI brings a new level of clarity to the table. It's the first tool to move the conversation from "What's possible?" to "What's practical?". In an era where AI models can write code, summarize reports, or generate images in seconds, the temptation to believe in a workforce overhaul is immense. But the data says otherwise: AI is still struggling with the complexity, consistency, and accountability required to replace human labor—even in remote settings where AI should theoretically thrive.

The Remote Labor Index is a multi-sector benchmark designed to measure how well AI agents can perform end-to-end, real-world freelance tasks that represent actual remote labor.

Each project includes:

• A real job brief from a freelancing platform
• All input files originally provided to a freelancer
• The final human deliverable that met client expectations

Then, multiple AI agents are given the same material, and their outputs are judged manually—not by code or rubric, but by whether a reasonable client would accept it.

Key metrics:

• Automation Rate: % of projects where AI output is equal or better than human
• Elo Score: A relative ranking of AI agents across tasks
• Autoflation: Cost reduction from substituting human labor with AI

It's not theoretical. It's economic. The RLI covers 240 projects spanning 23 categories of work, representing over 6,000 hours of real work valued at over $140,000^[1].

This isn't just another AI research paper. The implications of RLI stretch across every function and leadership role in the enterprise:

• CEOs & Boards: You're shaping 3–5 year rebuild roadmaps. RLI tells you where AI will impact jobs and cost structures next.
• CIOs & CTOs: You're building platforms that need to be ready for human-AI hybrid workflows. RLI defines where software needs augmentation layers.
• CFOs & CROs: You're balancing compliance, cost pressures, and digital investments. RLI helps separate real deflation signals from premature AI bets.
• CHROs & Talent Leaders: You're steering org redesigns and change management. RLI tells you where to reskill, where to restructure, and what to ignore (for now).

This benchmark helps you time your moves—when to invest, when to wait, and when to accelerate.

What makes RLI different is its ground truth approach.

Each project in the index includes:

• A detailed brief written for a freelancer
• All necessary input files (design assets, data files, etc.)
• A gold-standard human deliverable

AI agents are evaluated side-by-side with these human outputs, and real evaluators decide if the AI version is "client-acceptable."

These are not toy tasks. They range from:

• Building interactive dashboards
• Creating 3D animations for marketing
• Formatting a scientific paper for an IEEE conference
• Designing architectural plans
• Coding full-featured games

This approach lets RLI track how much of the actual digital economy can be automated by AI today—not in theory, but under realistic business conditions.

RLI draws from projects with genuine commercial value—each originally completed by experienced professionals on platforms like Upwork. These aren't isolated prompts like "write a poem" or "generate a chart." They're complex, time-intensive, and multi-format deliverables.

The categories include, among others:

• Video and animation
• 3D modeling and CAD
• Graphic and product design
• Game development
• Architecture
• Data analysis

Each task reflects the kind of work that enterprises already outsource or support via internal digital teams—making this a mirror to your real labor exposure.

You'd think AI agents trained on billions of tokens would dominate these tasks, right?

Not even close.

The best automation rate in the October 2025 paper was 2.5%. That's it. The majority of agent outputs were rejected for being incomplete, technically broken, or professionally unacceptable.

Breakdown of AI performance in the October 2025 paper^[2]:

Elo is measured against a human baseline fixed at 1,000.

On the public leaderboard the top rate has since risen: 15.8% for Fable 5 (CAIS, 1 July 2026) and 21.25% for Opus 5.5 (as of 18 October 2026).

Despite advances, these AI agents are far from delivering production-grade outcomes at scale.

• Automation Rate = Can this AI do the job as well as a human?
• Elo Score = Is this model improving over others? Is it closer to human parity?
• Autoflation = How much cheaper is work if done via AI—when it works?

These metrics together provide early indicators of future disruption, much like inflation trackers or digital engagement indexes do in other sectors.

So why is AI—despite the hype—still underperforming on real remote work?

Let's break down the four main failure modes the paper reports across its evaluations (they overlap, so the shares add up to more than 100%):

1. Technical and File Integrity Issues (17.6%)
   • Corrupted or unreadable files
   • Missing components
   • Wrong file formats (e.g., sending raster graphics when vector formats were required)
2. Incomplete or Malformed Deliverables (35.7%)
   • Projects that were only partially completed
   • Truncated videos, incomplete source code, or missing elements like voiceovers or datasets
3. Low-Quality Outputs (45.6%)
   • AI outputs that technically completed the task but were amateurish, unpolished, or unfit for commercial use
   • Examples: child-like design, inconsistent layouts, robotic voiceovers
4. Inconsistencies in Multimodal Projects (14.8%)
   • For example, in a 3D architectural render, windows might move between views or lighting inconsistently change

These failures reflect that AI agents lack critical professional capabilities—judgment, consistency, aesthetics, and the ability to self-correct. Unlike specialized tools, AI models still struggle with cross-file coherence and project-level thinking.

, incomplete deliverables (35.7%), low-quality outputs (45.6%), and multimodal inconsistencies (14.8%)")

Think of RLI like a seismograph for AI labor disruption.

It doesn't just show what AI can automate today—it sets the stage for monitoring progress over time. This is critical because automation won't happen all at once. It will arrive in waves, starting with narrow categories, then spreading wider as capabilities improve.

At launch in October 2025, performance was near the floor, though the paper's Elo scores already showed models steadily improving. The automation rate has since passed 5%, then 10%, then 20%: CAIS reported 15.8% for Fable 5 on 1 July 2026, and the public leaderboard listed 21.25% for Opus 5.5 as of 18 October 2026.

For the enterprise, that means:

• Current AI won't replace your workforce wholesale
• But the automation tipping point could hit sooner than you expect
• RLI can act as a "leading indicator" for when and where that tipping point starts

RLI introduces a powerful concept: Autoflation.

As an illustration, imagine replacing a $1,000 freelance job with a $3 API call: that would be a 99.7% cost reduction for that task, if the AI delivered acceptable quality.

In the October 2025 paper, autoflation was close to zero because agents completed only a handful of projects. The paper does not name a tipping point. As an AIDOLS illustration only: once automation rates reach something like 10–20% of projects, the labor pricing model for digital work could come under heavy pressure, and the best agent on the Scale leaderboard has since reached 21.25% (checked 18 October 2026). The leaderboard reports automation rate, not autoflation, so it does not yet show how far costs have actually fallen.

Implications:

• Enterprises may shift to AI-augmented models, where a single human directs multiple AI agents
• Talent models shift, not due to layoffs, but through redesign—moving human workers to review, QA, or prompt design roles
• Vendors and freelance platforms may see massive pricing compression

And this is just the beginning.

Let's move from insight to action. Here's how different leaders can apply the RLI framework today:

For CEOs and Corporate Strategy Teams

• Refine your automation roadmap: Use RLI to identify which categories of work are safe from disruption—and which are vulnerable
• Pressure-test AI implementation investments: Is your investment ahead of actual AI capability?

For CIOs and CTOs

• Adjust platform architecture: RLI shows where AI agents still fail—this helps prioritize human-in-the-loop features
• Evaluate AI agent scaffolding and workflows for creative, visual, and multi-modal tasks

For CFOs and Risk Committees

• Forecast cost deflation scenarios: Plan for what happens when AI can deliver 10%, 30%, or 50% of project value
• Redesign control frameworks: As humans hand over more tasks to AI, controls must follow

For CHROs and Workforce Strategy Leaders

• Revamp skill taxonomies: RLI highlights areas where AI might replace jobs vs. augment them
• Invest in human-AI teaming: Prepare employees to become AI reviewers, orchestrators, and quality leads

Rebuild programs need reality checks. RLI provides a powerful ground-truthing mechanism. Here's how to use it to de-risk rebuild:

• Augment before you automate: Focus first on tools that support human workers, not replace them
• Pilot where RLI's successes cluster: Start experiments in the kinds of work where AI deliverables matched or beat the human ones in the RLI paper (e.g., audio editing and production, image generation such as ads and logos, report writing, data retrieval/web scraping)
• Build change programs with time horizons: Treat 2025–2027 as augmentation years, and post-2028 as potential inflection points

By integrating RLI insights into rebuild strategies, companies avoid premature bets, disruptive false starts, and unrealistic AI expectations.

There are two types of risk when it comes to AI and automation:

1. Overreaction

• Laying off staff too early based on AI hype
• Replacing freelancers with AI models that fail quality checks
• Investing in fully autonomous AI workflows when even the best agent on RLI delivers client-acceptable work on only about one project in five (21.25% on the Scale leaderboard, checked 18 October 2026; 2.5% in the October 2025 paper)

2. Inaction

• Failing to pilot AI in augmentable domains
• Not training teams in AI-first tools and workflows
• Missing early signals of competitor productivity gains

RLI helps you thread the needle: act where it matters, wait where it doesn't.

The Remote Labor Index is not a crystal ball—it's a compass.

It helps leaders:

• Set priorities around AI investment
• Design pilot programs where ROI is most likely
• Monitor automation risks by domain
• Equip teams with real-world expectations

RLI's creators plan to update the benchmark over time, giving organizations a rolling view of AI's changing capability landscape. Forward-thinking firms will embed RLI tracking into quarterly AI steering committees, workforce planning, and vendor risk analysis.

Unless dated otherwise, the figures above are from the original paper, published in October 2025. As of 18 October 2026 the top rate on the public leaderboard was 21.25% (Opus 5.5); check remotelabor.ai or the Scale leaderboard before quoting an automation rate in a board paper.

What does not depend on the latest leaderboard is the practical lesson: deploy AI first on high-volume workflows where a person stays in the loop and the result can be measured against a baseline, and defer bets on fully autonomous delivery of complex, multimodal projects. AIDOLS maps a workflow first and gives a written verdict on whether each bottleneck needs software, a process change or AI; the free workflow map is the low-risk way to start.

You don't need to panic about AI—but you do need a plan.

Here's a Monday morning checklist:

1. Read your org's AI roadmap—does it align with what RLI shows is possible?
2. Ask your tech and HR leaders where augmentation can start now
3. Review current freelance/vendor categories—are they in high-risk RLI zones?
4. Bring RLI data into your next exec strategy offsite—and build scenarios around it
5. Educate your board on the real state of AI automation—grounded in economic reality, not hype

Let RLI be your flashlight in the fog. Because the future of work isn't just being written in code—it's being tested, project by project, on the front lines of remote labor.

For organizations ready to act on these insights with a working system rather than an advisory report, see the AIDOLS methodology: a first measurable result in 2–3 weeks, a production deploy in 90 days with a 40%+ efficiency target, and, for a Build, the fee refunded in full if the deployed system does not generate ROI exceeding it within 90 days.