There isn’t a business or information technology leader today who doesn’t have artificial intelligence at the top of their priority list. The challenge for most organizations is deciding what to measure and how to determine whether an AI initiative was successful.

To better understand how customers should approach AI metrics, I spoke with Colleen Aubrey from Amazon Web Services Inc,, who has some deep experience in this area.

As senior vice president of Applied AI Solutions at AWS and a member of Amazon’s S-team, she builds on that foundation with lessons from 20-plus years at Amazon, where she built customer-facing products for millions of sellers and advertisers. The playbook she drew from that experience gives enterprises a practical path to move out of AI pilot purgatory, and it starts with throwing out the AI return-on-investment slide.

She argues that understanding AI ROI requires thinking in terms of business metrics. “I worry that people start trying to come up with new metrics to measure the impact of AI, but it’s the same metrics,” she explained. “It’s your business metrics that matter.” AWS has spent 20 years building cloud infrastructure. AI business applications are a much newer opportunity.

When Aubrey joined AWS, she inherited 35 application-layer programs. “I would say Connect for the contact center was probably our most successful example,” she told me in a recent interview ahead of AWS re:Invent Nov. 30-Dec. 4.

She studied what made it work: The team had deep operational experience running Amazon’s customer service function; it chose a sharp wedge (outstanding voice quality on the AWS backbone rather than chasing feature parity; and the team knew its customers. The lessons from that success are now the blueprint for the expanded Amazon Connect family: Connect Customer, Connect Talent, Connect Health and Connect Decisions.

The playbook Aubrey built for the Connect family closely mirrors what IT leaders need to move their own AI programs out of pilot mode. Here are my five takeaways that should matter to IT leaders:

Feature parity is a trap

The first lesson Aubrey drew from her assessment was about where to start. Nobody runs a better supply chain than Amazon, which is exactly why the early version of AWS Supply Chain was such a useful learning experience. When Aubrey got close to the product, she saw a team working to catch up rather than lead. They had been building five modules that “largely were about getting to feature parity” with entrenched enterprise resource planning vendors that had a decades-long head start.

Connect had taken the opposite path, and it’s instructive. “They understood they weren’t going to achieve feature parity,” Aubrey said of the Connect team. Instead, they led with outstanding voice quality on the AWS backbone, started narrow and built from there.

Aubrey applied that same lesson to the supply chain. She rebooted it as Connect Decisions by asking a different question: where is the pain most acute, and where is the current tooling weakest? The answer was planners. “They’re firefighting the whole time,” she said. “They’re overwhelmed by a whole bunch of signals. By the time they get to decide, the moment has passed.”

The same applies to enterprises. AI projects that stall often try to replicate an entire workflow. The ones that succeed identify a painful decision point and make it dramatically better.

The business metrics haven’t changed

Aubrey’s point about measurement deserves unpacking. “Companies are stuck on what’s the ROI of AI, and they’re looking at return on tokens and this sort of stuff,” she said. “The question is, can you move your business metrics by working in a different way?”

Her proof point is Wells Vehicle Electronics, which, according to AWS, cut inventory by 7% in three months while expanding the SKUs they actively monitor from 20 to 2,000. Nobody at Wells is talking about tokens. They’re talking about working capital. If a project can’t be tied to a metric the chief financial officer already tracks, it’s an experiment, not an investment.

AI doesn’t remove bottlenecks; it moves them

Amazon started where most companies start: with software development, making AI tools available across the company. Aubrey admits there’s a bit of chaos at first. “Everyone’s just experimenting with a bunch of things,” Aubrey said.

She sees that as a feature, not a bug, because that’s where the real learning happens. “You go through this learning period of, you know what, our test coverage sucks,” she said. Fix test coverage, and you find “our pipelines suck,” so you fix pipeline health, eliminate manual intervention and deploy faster. The result in her group this year: a twofold improvement in the velocity of pushes to production.

The lesson is that the velocity didn’t come from the coding assistant. It came from fixing everything downstream that the assistant exposed. Enterprises that buy the tool and skip the plumbing will see activity rise while output stays flat.

Own the outcome, not the task

I asked Aubrey whether Amazon has adjusted its goals to reflect what employees can now do with AI. Her answer was an honest “yes and no.” In her user experience team, roles have collapsed: designers are being pushed to ship code to an alpha environment, and Amazon has created a new “UX tech” role.

The “no” is that org design and goal-setting are still catching up. Her message to her team is blunt: “The only choice you can’t make is to own the task. You have to choose to own the outcome.” Every IT leader should borrow that line. Task-based job descriptions will age poorly.

Good AI with bad UX will fail

AWS’s design philosophy for the Connect family is what she calls “humorphism,” the idea that AI should fit the way people already work, not the other way around. Her biggest concern is the lag between internal prototypes and production.

“Shipping good AI with a bad user experience will fail, in my opinion,” she said. Her team has invested heavily in closing that gap, moving beyond traditional command-based interfaces toward AI that picks up cues, remembers decisions, and carries work forward. The goal, she said, is to make AI so usable that people change how they work, not because they’re told to, but because it makes sense.

She was equally candid about maturity. TE Connectivity has scaled Connect Decisions to 2,000 planners, but Connect Talent reached general availability only in September and has early customers in deployment. Aubrey said the team is focused on getting the product right before aggressively scaling, reflecting the same “pick a wedge” discipline she applied across the Connect solutions.

What this means for IT leaders

• Pick a wedge, not a platform. The temptation with AI is to reimagine an entire function at once. Instead, identify the decision point where people are drowning in signals and acting too late, such as a planner buried in exceptions or an agent handling a surge of rebookings. A narrow, painful problem delivers a quick win, clear metrics and a credible foundation to build on.
• Kill the “AI ROI” slide. Return on tokens, prompts per user, and adoption dashboards measure activity, not value. Before any project is funded, tie it to a business metric the company already tracks, such as inventory, resolution rate, time to hire, or cycle time, and set a baseline. If the metric doesn’t move within a defined window, the project shouldn’t scale, no matter how impressive the demo was.
• Budget for the downstream fix. AI rarely removes a bottleneck; it relocates it. Amazon’s twofold gain in shipping velocity came from fixing test coverage and pipelines, not from the coding assistant. Assume your deployment will expose the next constraint, whether that’s data quality, integration, approvals, or testing. Earmark budget and people for that work up front rather than discovering it six months later.
• Rewrite roles around outcomes. Job descriptions built around tasks will age poorly as AI takes on more of those tasks. Work with HR now to redefine roles and goals around the outcomes people own, and update performance expectations to reflect the tools they have. Be honest with employees that some teams will shrink while others will have new gaps to fill; ambiguity breeds more anxiety than candor.
• Make usability a buying criterion. As Aubrey put it, good AI with a poor user experience will fail. In vendor evaluations, weigh setup, configuration, testing, analytics and day-to-day experience as heavily as model capability, because those factors determine whether business users adopt the product. Ask vendors how business users, not just technical teams, interact with the product day to day.

Final thoughts

What made this conversation valuable was Aubrey’s willingness to share how she thinks about building AI products that work in production, from studying what made Connect successful to applying those lessons across supply chain, hiring, health care, and customer engagement. That candor is a useful model for IT leaders, because most enterprise AI programs are stuck for the same reasons AWS’s applications once were: too much breadth, too little focus and success measured by the wrong yardstick.

The AI era won’t reward companies with the most pilots. It will reward those that pick a sharp problem, fix the plumbing it exposes, redesign work around outcomes and move the metrics they already have.

Zeus Kerravala is a principal analyst at ZK Research, a division of Kerravala Consulting. He wrote this article for SiliconANGLE. 

Photo: Robert Hof/SiliconANGLE