There is a line of questioning right now about whether AI is making people more productive. This reminds me of some questions we get at Intelligems, and, to put it bluntly, it absolutely blows my mind.
Of course we are more productive with AI!
If I were to ask engineers at Intelligems to stop using AI, I’d be met with a “what’s wrong with you!?” line of questioning from them. The gains they see are so obvious to them (and me) that we’d never actually entertain the idea. Which makes the skepticism around productivity that much harder for me to understand.
Where I think it comes from, though, is people judging outputs via outcomes. That’s faulty, especially when outcomes have so many different inputs (not just the one you’ve changed).
Instead, you have to consider what would have happened had you not made the change.
Economists call this a counterfactual.
Example: A team is spending $100,000 per month on tokens and is growing 6%. In October 2024, it was also growing 6%, but was spending nothing on tokens. So, what did the $100,000 buy?
The skepticism I hear is that maybe the tokens bought nothing. But that’s comparing two different worlds!
The counterfactual is that, without the token spend, the team may only be growing at 1% now. And, perhaps more positively, had you given that team in October 2024 $100,000 in token spend, they might have grown 12% because it was such a competitive advantage.
I’ve seen this same mistake at Intelligems over the years. A brand runs a test that reports a 5% lift in revenue. Three months later, the business isn’t up 5%, and the brand asks where the 5% lift went.
The answer to the question is more about externalities than anything else. Maybe ad spend fell. Maybe that previous time period had a sale and this one didn’t. Maybe the lift sent Facebook new signals and the algorithm found a different audience.
Maybe, without the test, revenue might have dropped 3%.
Counterfactuals are unsatisfying if you’re looking for a tight, tidy answer. But they’re revealing if you’re trying to understand more. It just requires a good amount of questions about all the things unrelated to the output you were measuring in the first place.