Amazon.com Inc. today introduced a new artificial intelligence model for the technology behind the online retail giant’s “Just Walk Out” checkout-free service for brick-and-mortar retailers.

The new model further increases the accuracy of the previous machine learning capabilities of the previous technology and applies them to physical stores by applying them to cameras, sensors in shelves, and other information. For retailers the system works a lot faster and easier to install; for shoppers, it means that they can just supply their payment information, credit card or mobile wallet, on entry and shop. Upon exit, the payment for items taken is produced along with a receipt.

Just Walk Out technology was piloted by Amazon in 2018 and used generative AI and machine learning technology at the time to examine shopper behavior at the time to examine shopper behavior with a series of models, Jon Jenkins, vice president of Just Walk Out technology in AWS applications, explained during a press briefing.

The technology has been used in Amazon Fresh and smaller Go-branded stores for years. Now Just Walk Out is also provided as a solution for third-party retailers to install in their shopping venues for cashier-less shopping. According to Jenkins, the technology is available in over 170 locations globally including airports, stadiums, universities and hospitals in the U.S., the U.K. and Canada.

Jenkins compared the new Just Walk Out model to large language models such as OpenAI’s GPT-4o, which is multimodal, meaning it can take text, audio and video, and output any of those.

“Instead of generating text or images, we generate receipts,” Jenkins said. “And instead of the input information being snippets of text or questions, our input is video from cameras, weight signals from sensors on shelves and pictures of what comes out in someone’s hand. It can also work with RFID chips embedded in products.”

This replaced the previous system which was a series of models, each of which did a sub-task of this process where one model had to wait for the previous signal to resolve. One model had to track a person moving around the store, another had to wait to see someone reach into a shelf, and yet another had to wait to see what was withdrawn. That was slow, tedious and needlessly complicated.

In the new model, all of this happens at once as it is capable of combining all of these signals from sensors, cameras and radio-frequency chips with a 3D map of the store into one action. This means it can readily handle strange issues that shoppers create in a chaotic retail environment.

For example, a shopper might pick up and put down multiple varieties of soda at a time. It’s not uncommon for shoppers to pick up an item, walk to another part of the store and then put something down on a shelf where it doesn’t belong – much to the chagrin of store employees. A freezer could fog up, blocking the camera’s view of the items within. All of these complex situations can be handled more accurately by the new model, which can learn quickly from the sensors and cameras by watching the hands and items using computer vision and self-teaching algorithms.

Jenkins was keen to point out that Just Walk Out technology also changes the nature of shopping because it could reduce shrink or shoplifting. “When you put Just Walk Out technology in a store what would have been theft – me taking a Snickers bar and jamming it in my pocket – that becomes a purchase.”

He also emphasized that although the technology makes heavy use of cameras to track the movement of customers around the store, it does not use or collect any biometric information to complete its tasks.

“We don’t use any biometrics at all. We don’t even track what clothing you’re wearing or anything like that,” Jenkins said.

For example, it’s not uncommon for Just Walk Out to be deployed in stadiums where everyone is wearing the same jersey or ball caps, which makes it difficult to identify people based on their outfits. “So, if you were using the visual appearance of what people looked like to determine who they are or where they were in the store, that would be a bad way to go about it.”

Instead, the system tracks people by their movements and the movement of their hands, he explained. The most important thing to the system is that that pair of hands “signed in” with a payment method at entry and that is mapped to them upon entry. In fact, the same person could leave the store – which would trigger a receipt and a payout – then re-enter the store with a different payment method, causing the system to believe this was a whole new person.

Images: Amazon.com