Lerrel Pinto, a computer science researcher at New York University and one of MIT Technology Review’s 2023 Innovators Under 35, is working to bring robots into our homes in a more meaningful way. He envisions robots that can perform tasks beyond just vacuuming, such as chores, elder care, and rehabilitation.
The challenge lies in training these multiskilled robots, which requires a significant amount of data. Pinto proposes a solution: self-supervised learning, where robots collect data as they learn. This approach, supported by his colleague Yann LeCun and Meta’s chief AI scientist, involves finding novel ways to gather data and furthering the fusion of machine learning and robotics.
Pieter Abbeel, director of the robot learning lab at the University of California, Berkeley, praises Pinto’s work, calling it a major milestone in combining machine learning and robotics. He believes that Pinto’s research will be remembered as the foundation of future developments in robot learning.
Unlike AI models that leverage large language datasets gathered from the internet, robots require specific data to train. Self-driving-car companies spend countless hours on the road collecting valuable data for training their vehicle models. Similarly, household robots need hours of robot’s-eye footage performing various tasks in different environments.
Pinto made significant progress in 2016 by creating the world’s largest robotics dataset at the time. He accomplished this by having robots generate and label their own training data, running them continuously without human supervision. Since then, Pinto and his team have developed learning algorithms that enable robots to learn from failures. By training models with data collected from unsuccessful attempts, they can improve the robot’s performance.
Another approach Pinto explores is mimicking human behavior. Robots observe humans performing tasks, such as opening doors, and attempt to replicate them. With each new observation, the robot strengthens its ability to accomplish tasks it hasn’t seen before.
In his latest project, Pinto enlists the help of volunteers to record videos of themselves using simple tools to grasp objects around their homes. This low-tech approach, coupled with efficient learning algorithms, allows robots to learn with minimal data. Pinto’s team has demonstrated that dexterous behaviors, like opening a bottle with one hand or flipping a pancake, can be achieved with just one hour of training.
By enabling robots to learn from themselves and their human counterparts, Pinto hopes to usher in a new era of AI. He believes that movement is fundamental to intelligence, and only physical creatures like robots can bring about meaningful changes in the world.
Sources: MIT Technology Review, University of California, Berkeley.