Over the course of the past decade or so, grocery shoppers at outlets large and small have become accustomed to automated self-checkout stations that reduce the number of human cashiers required to scan and weigh items. Shoppers, in turn, are tasked with scanning, measuring, and bagging their groceries. And though usage of self-checkout technology has grown, so too have the number of critics who argue the setups displace human workers despite largely not living up to their promise of shorter lines.
This automated model for grocery shopping still requires a human at the end of the loop to actually place the scanned items in bags, but that could change. Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a new AI-enabled, soft-handed robot system they say can accurately identify and bag common grocery store staples like fruit, chips, and cans of soup. A specifically designed algorithm also helps the robot separate out delicate items in order to avoid crushing them. The end result: a prototype robot capable of safely packing a bag of standard groceries. Whether or not such a system will ever actually improve on the traditional human bagger still seems unlikely.
Computer vision and gentle fingers help the robot sort delicate groceries
The researchers detailed their multi-modal, gripping robot system, appropriately called “RoboGrocery” in a recent paper published in IEEE Explore this week. An RGB-D camera equipped with computer vision technology allows the robot to identify items approaching it on a conveyor belt and estimate their overall sizes. That initial estimate helped the robot determine where to grab each item.
Once the items make their way to the robot, it then uses finger-like soft grippers to pick them up. Pressure sensors loaded on the grippers lets the robot feel the item and make a determination to whether it’s delicate or not. Finally, an onboard AI algorithm helps it assign a “delicacy score” to each item. Items with a high score, like fruit or chips, are placed to the side while less delicate items like soup cans or cereal are bagged right away. Once the more rigid items are packed, the robot goes back and places the delicate items on top to prevent them from getting crushed or otherwise damaged.
The video above shows the robot in action gripping a bundle of grapes and a container of Trader Joe’s baking soda. When presented with grapes trundling down a conveyor belt, the robot’s grippers quickly swiped the fruit up without appearing to pop or bruise any of them. After holding the grapes for a moment, the robot placed them aside in a safe area meant for delicates. A can of baking soda, by contrast, was placed immediately in the grocery bag.
Tactile sensors on the grippers put pressure on all of the objects to determine their stiffness. Objects that appear to deform slightly when grapes are determined to be more delicate. This means, in theory at least, the robot shouldn’t clutch onto a mushy tomato with the same grabby as a box of Lucky Charms. Researchers say the use of these multiple sensing modalities (vision and pressure) compliment one another and “ensure an accurate and timely understanding of the properties of an object’s material.”
“Imagine a team of helpful robots working side by side with you to help you pick up and pack all the groceries you need for the day,” CSAIL Director Daniela Rus said. “This is the kind of future that we are enabling here.”
This isn’t the first attempt at creating a robot packer. Previous demonstrations, like this one from the firm Covariant, have shown machines can use computer vision and a suction like arm to identify and place items into different binning areas. Other robots, including those used by Amazon, already excel at boxing packages, but they are typically focused primarily on speed rather than determining the delicacy of objects. The MIT researchers believe their RoboGrocery unit’s specific combination of soft handed-robotics and AI decision-making should set it apart. In a test of the system, researchers say their robot had nine times less instances of damaging items than another robot without the same tactile sensors.
“It’s [RoboGrocery’s] ability to assess items, determine their delicacy, and pack efficiently without causing damage sets it apart from conventional robotic packers,” CSAIL Director Daniela Rus said in a statement.
Self-checking tech hasn’t lived up to its promise so far
Though the bagging robot is still in the research phase and not ready for commercial deployment, the researchers envision a path forward where it could be used alongside current self-checkout or “cashierless” tech to create a more automated shopping experience. Self-checkout systems that task consumers with pricing their own groceries date back to the late 1980s but have only gained major traction in the US over the past decade. In 2021, according to a report from The Food Industry Association, almost 30% of all grocery store transactions came from self-checkout lanes. That’s more than double the amount from just three years prior. Today, various forms of self-checkout systems can be found in Whole Foods, Kroger, Walmart, Target, CVS, and many other common retailers.
These automated systems were pitched as a win-win for consumers and grocery store operators. For shoppers, self-checkout promised to speed up lengthy wait times. Grocery store owners, meanwhile, could and often did, radically reduce labor costs by replacing multiple human scanners with one or two tasked with overseeing the machines. But neither of those visions have really played out exactly as intended.
Grocery stores are still riddled with lengthy lines due to some customers needing help working the machines. Restricted items like alcohol similarly require a human staffer to manually intervene with a purchase. And even when consumers do want to use the systems, they are often faced with a frustrating situation where multiple lanes are closed due to technical issues with the kiosks that can take time to address. These complaints aren’t rare. A 2021 survey of US grocers by research firm Raydiant claims 67% of shoppers report having a self-checkout system fail on them.
For grocers, the savings on labor cost are partly undercut by constant need for self-checkout upkeep and repairs. The lack of human oversight has also reportedly led to a sizable increase in casual shoplifting. Recent reports suggest automated checkout systems may make shoplifting easier and more commonplace. Some companies are starting to take notice. Last year, Walmart reportedly removed self-checkout systems in several New Mexico. Costco, meanwhile, announced it would add more staff to manage the automated machines. Booths, a supermarket chain in the United Kingdom went a step further and removed self-checkout options in 26 of its 28 stores.
“Our customers have told us this over time, that the self-scan machines that we’ve got in our stores they can be slow, they can be unreliable, they’re obviously impersonal,” Booths managing director Nigel Murray said in a previous interview with the BBC.
Bagging robots will need efficiency and low failure rates to ever feel like a viable option
Growing frustrations and resistance to automated checkout systems could make it even more difficult for bagging robots to ever truly gain a meaningful foothold. It’s not difficult to imagine grocery shoppers already fuming with anger over malfunctioning or out of service self-checkout lines applying that same anger to a machine attempting to pack bags. To ever succeed, these robots would likely need to pack bags faster and more efficiently than a human but with an extremely low failure rate. Shoppers unpacking their groceries at home only to realize a robot cracked their eggs or turned a tomato into sauce are unlikely to trust the machine with future purchases.
Still, just like self-checkout before hit, bagging robots of the future could appear enticing to grocery store chains as yet one more avenue to reduce costs on already dwindling human jobs.