With an eye toward putting their research muscle behind solving broad societal issues, Toyota Research Institute (TRI) opened its laboratory doors to journalists to showcase their latest advances in robotics aimed at assisting humans in their homes. TRI’s focus is on home-based assistance to address the vexing issue of our aging population. According to the United Nations, over the next three decades the global population over the age of 65 is projected to more than double. That means over 1.5 billion people around the world will be 65 or older by the year 2050. This graying of the population will have profound effects on society, the workforce and the economy. TRI believes their research can provide options to address these challenges with advanced human-assist robots to help people age in place with dignity.

TRI’s robotics research is guided by the Japanese notion of Ikigai, a powerful idea that every person’s life should have meaning and purpose. TRI’s human centered approach turns typical AI philosophy on its head: instead of replacing human beings, TRI’s research uses AI to amplify human ability.  This approach is known as Intelligence Amplification (IA), where machines and humans work in synergy to do something better than neither could do alone. “Studies of Ikigai teach us that we feel most fulfilled when our lives incorporate work that we love and that helps society,” said Gill Pratt, CEO of TRI and Chief Scientist for Toyota Motor Corporation (TMC). “To enable more people to achieve their Ikigai, TRI is pursuing new forms of ‘automation with a human touch’ (known as “Jidoka” in the Toyota Production System) to develop capabilities that amplify, rather than replace, human ability with the goal of bringing deep happiness and fulfilment to all people.”

THE POTENTIALS AND CHALLENGES OF HUMAN AMPLIFICATION ROBOTS

As societies age, there will be huge demand for increased caregiving, systems that enable us to live independently longer, and assistance for an increasingly aging workforce. Robots and automation can play a key role in freeing up people to spend more time with family, assisting people with tasks they enjoy, or helping them perform work for their jobs.

TRI believes that robots are not seen in these roles today because roboticists haven't yet figured out how to make robots reliably operate in the complex, unstructured environments that people function in every day. Unlike factories, where the environment is structured and programmable, natural human environments, like someone’s home, are unstructured and diverse. For example, every home is unique, with a different combination of objects in distinct configurations that are constantly changing.

“TRI robotics research is focused on the home because it is in that environment that robots can provide the greatest assistance in achieving human fulfillment. It is also one of the most complex environments for robots to master,” said Max Bajracharya, VP of Robotics at TRI. “Our work is focused on two key challenges: teaching robots from human behavior and using simulation to both train and validate robot behaviors.”  Collectively, we think of this idea as fleet learning, where when one machine learns something, they all learn something. “We believe this is going to be the key to making robots in human environments practical,” said Bajracharya.

TEACHING ROBOTS

To address the diversity a robot faces in a home environment, TRI teaches the robot to perform arbitrary tasks with a variety of objects, rather than program the robot to perform specific predefined tasks with specific objects. In this way, the robot learns to link what it sees with the actions it is taught. When the robot sees a specific object or scenario again, even if the scene has changed slightly, it knows what actions it can take with respect to what it sees.

Leveraging humans as teachers is a key way TRI is enabling robots to learn and acquire real-world skills. Using virtual reality, a human trainer sees what the robot sees in real time then commands the robot to perform a variety of different actions.  The robotics team at TRI is working on generalizing this type of learning, so if one robot learns a wiping task in the mock home kitchen, it could do the same task in any kitchen. TRI’s teaching process and approach to fleet learning is explained in more detail in this story.

USE OF SIMULATION FOR MANIPULATION RESEARCH

TRI has made a significant investment in simulation for both engineering and validating robot behaviors. The mechanics of the way a robot hand interacts with objects is very complicated to simulate, so simulation has traditionally not been used for robotic manipulation research.

In order to teach a robot a new behavior, or refine one that it already knows, TRI’s simulation software provides a way to understand the robot's performance without having to physically perform all the tasks every time a change is made. The results in simulation are tested on real kitchen sets in the TRI Cambridge lab. TRI researchers’ use of simulation tools can also advance robotic development even during a period of limited access to hardware and testing facilities.

“We’ve used our dish loading robot and clutter-clearing experiments to automatically improve our behaviors in simulation and have that result in improved performance on the real robots,” said Russ Tedrake, TRI Vice President of Robotics Research. The TRI robotic manipulation team based in Cambridge, Massachusetts, is working on furthering prior research into those challenges. Their progress on taking on hard problems in manipulation was previously shared in this feature.

NEW ROBOTIC HARDWARE AND SOFTWARE

In considering robotic solutions in the home, the TRI team is also looking at some more radical ideas.

One innovative concept is a “gantry robot” that would descend from an overhead framework to perform tasks such as loading the dishwasher, wiping surfaces, and clearing clutter. By traveling on the ceiling, the robot avoids the problems of navigating household floor clutter and navigating cramped spaces. When not in use, the robot would tuck itself up out of the way. To further investigate this idea, the team has built a laboratory prototype robot that can do all the same tasks as a floor-based mobile robot but with the innovative overhead mobility system.

In another innovation, to address the need for soft contact as robots interact with indoor environments, TRI researchers have developed novel soft grippers that have high-density tactile sensing capability. “The soft grippers are made of a soft bubble that complies when it is in contact with objects or tools, and we can control that compliance by changing the pressure in the bubble,” explained Tedrake. “Inside is a depth camera that senses the shape of the bubble skin, and tracks motion to estimate the shear forces on the surface.” More details on the soft bubble gripper can be found here.

“MOCK HOME” SPEEDS UP DESIGN AND ITERATION CAPABILITIES

At the TRI headquarters in Los Altos, California, a research team led by Jeremy Ma and Dan Helmick work in a new “mock home” robotics testing facility. This research environment, in which TRI can rearrange floor layouts and move objects, allows TRI to develop fundamental robot capabilities before testing them in a real home. The mock home features a kitchen, dining area, bathroom, and living space similar to those found in an actual home.

“Even though the mock home is a relatively new addition to our office, it truly gives our team the ability to test concepts quickly and efficiently,” said Jeremy Ma, Sr. Manager, TRI Robotics.  “We plan to finish the mock home construction by the end of this calendar year and expect the lab to be a vital building block in establishing the future capabilities of robotics at Toyota.” 

MEETING REAL USER NEEDS

For robotics to be successful in the home, TRI believes it is important to discover and account for individual human tastes, needs and means of fulfilment. TRI takes a “fail fast” approach to technology development so that its work can more quickly make a positive impact on actual lives and improve social good. Rather than simply developing capabilities that researchers find interesting or think will push the field forward, TRI has a Robotics User Experience and Industrial Design group to uncover and probe real user needs.

“We rely heavily on observational research techniques such as contextual inquiries,” said Steffi Paepcke, a TRI User Experience Team Lead. “Before COVID-19, we went to Japan to work with our research partners to visit the homes of older adults and observe them going about their daily lives, making note of friction points, challenges, and opportunities.”

“We observed that cooking is a beloved activity for many, though it can get more strenuous over time. Sharing meals and feeding loved ones also can serve as a focal point for social connection…, so giving elderly people a fully automated cooking robot or pre-cooked meals might be physically beneficial but emotionally detrimental.”

The challenge is to truly understand how to develop human-centered robotics when each individual user represents a unique case. The goal is not to just give people gadgets, but instead find ways to truly enable people to achieve fulfillment and help society, each in their own unique way.

TOYOTA COLLABORATION

TRI’s robotics charter is to develop new robotics capabilities for Toyota that can contribute to solving real world problems and aiding global societies.

To create real-world impact, TRI is working closely with other groups inside of Toyota.

One such group is Toyota AI Ventures, Toyota’s first corporate venture capital firm that strategically invests in early stage startups.

Another group is TRI-AD, soon to be the Woven Planet Holdings Group, which focuses on taking new capabilities from TRI and other parts of Toyota and, in collaboration with them, developing them into product concepts.