Amazon.com today showcased the multiple ways in which artificial intelligence-based machine learning and computer vision algorithms are being combined with synthetic data to improve key retail automation technologies such as Just Walk Out, Amazon One and Amazon Dash Cart.

In a keynote at Amazon’s re:MARS event today in Las Vegas, Dilip Kumar, Amazon’s vice president of physical retail and technology, explained how computer vision plays a critical role in enabling these technologies. For instance, Just Walk Out, a technology that enables shoppers to skip the checkout line, has benefited from ongoing innovation in areas such as sensors, optics and machine vision algorithms.

These advances have allowed Amazon to reduce the number of cameras required at Just Walk Out-enabled stores, Kumar said, making them more cost-effective and capable of running their algorithms locally.

“Our sensors and algorithms have evolved to detect a broad range of products and differences in shopping behavior in full-sized grocery stores, all while ensuring an effortless customer experience,” Kumar said. “We also increased the diversity of environments our algorithms have to account for as we deploy Just Walk Out technology to third-party retailers.”

Meanwhile, the company is using computer vision and sensor fusion algorithms to scale its Dash Cart service, which allows customers to skip the checkout at Amazon Fresh stores in the U.S. Most important, Kumar said, is that the company has developed more robust algorithms that can detect items while in motion and capture both their weight and their quantity.

“The machine vision algorithms also have strict latency budgets, as we keep track of a customer’s receipt in real time,” he added.

AI also helps to provide better customer recommendations. For instance, shoppers at Amazon Style, the company’s physical apparel store, will be treated to a diverse list of recommended items based on the products they scan as they peruse the shop floor.

“The system also generates complementary selections, such as a shirt to match a pair of jeans to create a complete outfit,” Kumar said. “We went to great lengths to keep the fun in shopping while elevating the experience through machine learning algorithms.”

None of this would be possible without the use of synthetic data, though. As Kumar explained, Amazon was challenged by the lack of diverse training data needed to train these algorithms. To compensate, Amazon’s researchers set about building massive sets of synthetic, or machine-generated photorealistic data, that could be used to perfect its algorithms.

In the case of Just Walk Out, Amazon had to build synthetic datasets to mimic realistic shopping scenarios, Kumar said. For instance, synthetic data was used to create variations in lighting conditions to account for the differences in sunlight at various stores. In addition, Amazon created crowds of synthetic shoppers so as to teach its algorithms to handle numerous customers at the same time.

Synthetic data was even used to create palm prints to train Amazon One, which is a service that enables people to use their palm to pay at a store or enter physical locations. Genuine palm prints are hard to come by, Kumar explained, but Amazon needed to train Amazon One’s algorithms to recognize different demographics, age groups, temperatures and even variations such as calluses and wrinkles. So it opted to build a huge volume of diverse and realistic synthetic palm images instead.

One final challenge Amazon has faced is that shoppers’ behavior tends to change the more the company scales up its retail technology. Just Walk Out was first deployed in smaller Amazon Go stores that typically cover just 1,800 square feet but has since been rolled out to much bigger grocery stores of 40,000 square feet or more, Kumar explained.

Amazon faced an unexpected challenge because it quickly realized shoppers behave differently at these stores. At an Amazon Fresh grocery store, for example, people tend to rummage around for items such as the freshest produce, whereas at an Amazon Go they’re more likely to just grab a sandwich, for example. Similarly, the expansion of Just Walk Out to businesses such as travel retailers and sports venues presented unique challenges.

“All these scenarios increase the complexity for our algorithms, and my team continues to innovate to meet the requirements of our customers and retailers.” Kumar said.

Photos: Amazon