The story is that AI isn’t paying off, that the ROI just isn’t there, that the org charts and the earnings calls are still waiting for the productivity spike.
Harsh Savergaonkar and I are pretty extreme vibe coders. Since January, I personally push at least 1K commits a month…
Stop right there, I already know what you’re thinking.
I know, I know… lol. Commits do not mean a thousand units of careful thought and productive work. Blah, blah, blah. Yes, I agree. The metric isn't, by itself, a great measure of valuable output. But it is a directional indicator. A good enough proxy.
The fact is, though, that whatever you think of commits, companies are creating valuable products and services today with AI that were impossible just 6 months or a year ago.
A couple of examples at my company:
Rocketboard: Like every other company, I created a custom project management tool that kicks ass over existing tools, and we used it to replace ClickUp. Saved over $5K a year.
Synthetical Research: We will soon be launching our internal market research product for synthetic consumer research and fake app store testing that we have been using internally. This enables synthetic consumer testing for pennies on the dollar and app store testing at dramatically lower costs than current vendors.
And by the way, I’m not the only one pushing lots of commits/PRs; I’m not saying I’m super unique or special; many others have also publicly shown similar output using AI.
In fact, plenty of others are dramatically outproducing me… I’m not exceptional at all.
But take that as a directional indication that AI-driven output could change how builders abstract development and generate increasingly valuable, well-considered outputs. And output, with the recent shift to orchestration, increasingly more competent models, and the chief of staff operating paradigm, only increases valuable output.
Harsh started years before me and got me started back in December.
We’re also pretty good at what we do. In fact, we won first place at an AI hackathon sponsored by Peak XV (India’s most prestigious venture firm) and Apps for Bharat (a famous Indian startup).
All that to say, we know a thing or two about working with AI… from experience. And from using it better than the average vibe coder. We are operators who have been doing this at a relatively extreme level.
And we’re certainly not the same loud mouths who opine on things they have only ever dabbled in, if that. No executive hand-wavey nonsense here.
Today, we explain the paradox of AI ROI.
Why do we hear about or personally see some examples of extreme productivity by some AI vibe coders, but when we look at teams, companies, and our economy, the story seems to be that AI ROI is just not there?
My claim: the productivity and ROI are here. The problem with AI ROI isn’t AI; it’s the people using AI and the nature of our organizations.
Check out our full discussion here:
Based on our discussion, the answer came in three parts. None of them is the model. It’s not about the ability of AI to drive productivity; instead, all three are organizational and about people.
Part 1: It’s a skill issue
Start with the uncomfortable one. Most people cannot do this.
My rough estimate, and it's mine, not a study: at least 95 of 100 people cannot use AI to build software to production quality. Take the number as directional, not precise. The point is the gap is wide, and most people are on the wrong side of it.
There’s a ladder people don’t see. Level one is a prototype. Moving from prototype to something generally usable is at least a 10X jump in complexity, time, and effort. From usable to production is at minimum another 10X, maybe 40X or 100X increase. So the world fills up with prototypes and brittle software that breaks all the time, and everyone concludes AI doesn’t work.
Harsh put the skill point more cleanly: “Everybody has access to TikTok and Instagram; everyone has a phone. If they were given the same task, like record a 20-second video, only a few people will come up with something that’s fun to watch.” His conclusion: “Tools don’t matter; it’s that your skills matter.”
The skill breaks down into four things.
Subtraction. The model over-builds by default, so half the job is telling it what not to do. Ask for a sign-in page, and it will bolt on forgot-password, social login, ten generic things you never asked for, all of them sloppy. Harsh’s rule: “You need to be able to tell it: these are the four things we need, these are the four things we don’t need.” And his frame for finding those four: “you need to act like a QA tester for your own software.” Attack it from every kind of user behavior, and name the cases you’re not designing for.
Discernment. Even frontier models make logic mistakes and lack-of-context mistakes all the time. They fill missing context with an assumption and then build on it. From misdiagnosis to relying on training data patterns that aren't appropriate for your situation, you must safeguard your deliverables from the mistakes the model you employ will undoubtedly make at some point. My test: if you’re not catching two or three major problems a day in what the model hands you, your judgment is missing, and you’re the one being driven. Harsh goes further than reviewing output. He reads the model’s reasoning: “I literally expand its thinking, and I read through it.” That’s how he caught it, mid-task, adding IAM security roles to a multiplayer feature with zero users. He stopped it: “Bro, we don’t have any users using multiplayer. Just make it as simple as you can. Let’s get 10 people using it. Then let’s start adding security.” The architecture got much simpler and the whole thing shipped in a day.
Best practices. Vibe coding is probabilistic, so you engineer both sides of the probability. On the input side: your context setup, your prompt scaffolding, what files you inject, what’s in your Claude.md or agents file, and most importantly, adversarially reviewing plans or architecture by decorrelated models from fresh context. You need to understand what the LLM has in context, because it will confidently fill any gap based on that context. Harsh runs an adversarial room on the input side: “It becomes like a group project: I have Codex, Claude Code, and three of my own personalities talking and coming to a conclusion. People don’t go to that depth to build something. They just expect that they’ll type something and it’s done.” On the output side, you do the same thing: evals, adversarial reviews by multiple de-correlated LLMs. And you need deterministic gates. Harsh: “If you don’t have those gates for input and output, there’s no way you can say the LLM produced slop.” No test, no verdict. “You didn’t give it parameters, so how the hell is it supposed to know?” The progression I tell people: prompts, then reviews and evals, then loops, then agentic orchestration.
Naive usage. Most people never get past step one of that progression. I’ve interviewed a lot of tech leads in gaming who tell me “I’ve been using AI for a year and a half, I’m an expert.” I say, show me your GitHub repo. There’s nothing. I ask basic questions about injecting context or prompt scaffolding, and they just don’t know. They type “make this for me” and expect it to work. That’s not skill, that’s hope. More expert usage would have a deep understanding of the tools you use and a framework for experimentation.
Part 2: Defending the status quo
In the games industry and at my own studio, I’ve observed that the employees who have the most difficulty leveraging AI are engineers. Why?
First, the correction to the popular take. The best vibe coders are not non-engineers. They’re elite engineers. The absolute best are probably the S-tier guys at Anthropic and the other frontier model companies; I’ve met some of them, and their output is ridiculous. Harsh’s version: people from Anthropic and OpenAI will tell you “I haven’t seen a line of code in the last six months, and everything was shipped using AI.” Skill has a technical floor, and the ceiling belongs to people who cleared it.
The surprise is at the median, not the top. Across the AI-native studios I talked to at GDC in March, mostly web games and Godot shops, the highest-output vibe coder was usually a PM, not an engineer. At my own company, some artists outproduce engineers. Not because engineering knowledge hurts. Because the median engineer defends an old workflow, and the PM has nothing to defend.
Harsh named the mechanism. Engineers have been in the depths of the code so long that they still check every small edit, every diff, every step. So the AI runs at human speed: “Let’s say the engineer has gone for lunch, or they have just stepped out for the day, then the work is stopping. So even though AI is the one doing it, it’s still going at a human speed.” The speed limit is still how much one person can do in a day. Hand them a new way of working and the reflex is defensiveness, then what Harsh calls “this laziness of learning a new way.” And underneath that, something quieter: “they have heard the no so many times that they are still skeptical.” Stack Overflow said no. Their seniors said no. Now it’s suddenly yes, and they don’t believe it. Harsh again: “For non-developers, they have never heard that this is not possible; they’ve never heard a no, so from the get-go they’re happy to try it out, and then it just works for them.”
Scope this correctly. It’s the median engineer with a defended workflow, not the ceiling. That’s not a skill gap. That’s a status quo problem, and it sits one level up from the individual.
Part 3: Incentives, and the people who beat them
Fix the skill, and you hit the wall that actually explains the paradox. Harsh gave me the whole thesis in one breath:
“Most people aren’t incentivized to get more output. That’s why they’re not getting it. That’s why overall it looks like output hasn’t supercharged. But look at incentivized teams — small teams, solo founders, people super incentivized to reach the end of the line — they’re showing how crazy good the output can get.”
The gains are real. They accrue to whoever is structurally paid to capture them. Founders. Solo builders. Small teams with equity. Everyone else is paid to clear an expected bar and stop.
A salaried engineer who becomes 50× more productive does not ship 50× the code. They ship the same code in less time and go home. That’s the comp plan working exactly as designed. Nobody’s being lazy about the work, but they are being lazy about the change, and that’s different.
Many employees also have the trade backward. Harsh: “A lot of people get it wrong, ‘give me more incentive, I’ll give more output.’ It’s the opposite. I’ll give more output, and that makes you give me more incentive.” Most people never figure that out, and so they never push AI to the level it can actually go.
The healthy version is the one I landed on: “The output should not be constrained by the person. It should be constrained by your token budget.”
Here’s the part that matters if you can’t hand out equity to everyone: without the incentive, the whole thing rests on the person. So the real question becomes who, exactly, drives that output on their own, and that is a hiring question before it is a comp question. I screen for four traits, and they map directly onto the skills above.
High IQ. Catching the model’s logic jumps is raw reasoning. You’re spotting the context it didn’t have and the leap it made without it, dozens of times a day. This is the discernment from Part 1, turned into a hiring bar.
High agency. Another way I think about agency is how much you care. That you give a shit. This is the incentive problem seen from the inside: if your instinct is to show your expected output plus a little more and then check out, your output is fundamentally limited, and you’re the one limiting it. Agency is what makes “constrained by token budget, not the person” actually true.
Attention to detail. Harsh’s example is the warehouse scene in The Intern, where Anne Hathaway repacks a box herself: “it shows how much tiny attention to detail people who care have.” The small stuff is where production quality lives — and it’s the difference between subtraction done right and ten sloppy features nobody asked for.
Technical foundation. At least today, you still have to understand infrastructure, databases, and the technology choices you’re making with the LLM. And you need what Harsh calls the human side of scale: “one thing that’s important for you as a human is to have a large context window yourself.” One feature now touches many parts of your product, and you have to hold the whole system in your head to know what to test.
Find those four traits, and you’ve found the person who has the characteristics of someone who can leverage AI well and is the type of person to try to do so.
Part 4: Building “crap” has zero ROI
Skill answers: can I build it? For web, my answer now is: I can build anything. But that’s increasingly becoming the easy half. The question the model can’t answer is what to build.
Harsh sorts everything you make into three buckets: 1. crap, 2. a tool for yourself, or 3. a real product you’ll launch.
And watch the trap he named — the dopamine hasn’t caught up to the economics: “If something was hard to build, our brains still tag it as meaningful. But it’s not hard anymore. So you ship things that were hard, that look amazing, that nobody will ever care about.” Impressive and pointless is the default failure mode now.
In a world where development costs go to zero, the limitation shifts from “can we build it” to “WTF should we actually build?” Product-market fit, unfortunately, still remains hard.
Product velocity is magnitude AND direction. So where do you direct your newfound development capabilities? Another 100 random dashboards and tools no one will ever use?
For a games audience, that’s the whole warning. Everyone can ship a slick prototype this weekend. Just search for “AI Games” on X. You’ll see 100s of posts of slick-looking nonsense.
And remember, the prototype is a small fraction of the battle. Prototype → Usable → Production. Now try getting to production.
Impressive visual fidelity with nothing underneath is just another piece of crap with little to zero ROI.
So where’s the ROI?
It was never missing.
It’s sitting with the handful of people in your building who have the skill, aren’t defending an old workflow, are incentivized to run, and have room to run. Everywhere those four conditions fail, it’s throttled.
So do something about it. Three moves, in order.
Measure retained value, not just commits. I opened this piece with a commit count and half of you have been mad about it ever since. Fair. Commits were only ever a proxy, and a noisy one.
Here's what I'd actually measure: how many of your AI initiatives are being used or monetized, not just built. Adoption. Time saved. Whether the thing survives in production. Track tokens per completed task too, but as a diagnostic, not a target. Make it a KPI and people will optimize it: pick easier tasks, skip the iteration that made the work good. Measure it to spot who's under-using the tool. Don't reward people for spending less.
No proxy is perfect. But a good-enough one you track beats a perfect one you don’t. If you’re measuring nothing, you won't know where your ROI went.
Give one operator a bounded problem. Find the person with the four traits: high IQ, high agency, attention to detail, technical foundation. Often it’s not the engineer. Hand them one real problem with room to run and watch what comes back. That’s your ratio test, run live.
Rewrite the remit so building isn't gated to engineering. Your highest-output builder is probably not in your engineering org. It's the PM who shipped the tool your backlog sat on, the artist who stopped waiting for a ticket. Right now they're building on nights and weekends, off the side of their desk, with no cover when it breaks. Most orgs quietly forbid this and then wonder where the ROI went. Fix it in the org chart: give non-developers explicit permission to build, and make building part of their actual remit, not a hobby you tolerate. That's the structural change, and it's yours to make.
Here’s the part I want you to consider more carefully: AI gave you magnitude. It did not give you direction. It will build almost anything you point it at, which means the whole game is now what you point it at.
The studios that win the next few years won’t be the ones shipping the most. They’ll be the ones who know what's worth shipping.
