Helping robots to understand complex spoken commands—and act accordingly

The leftmost illustration represents the kind of language that Thomas Howard and his students have enabled robots to understand through past projects in his Robotics and Artificial Intelligence Laboratory, which refer to objects in their current state.  The other two illustrations represent the kinds of language this research will pursue, which refers to the past and/or future state of objects.  (Illustrations by Michael Osadciw of University Communications)

Thomas Howard will use his CAREER award to improve human-robot communication

“Collaborative robots that can seamlessly interact through language are poised to have a transformational impact on society,” says Thomas Howard, a robotics expert at the University of Rochester. 

However, for that transformation to occur, robots need better models of how people use language to refer to things in their environment.  This means that robots must not only reason about the present, but also the past and/or future state of objects so that they can communicate with their human partners on a wider variety of tasks, says the assistant professor of electrical and computer engineering.

With support of a National Science Foundation Faculty Early Career Development (NSF CAREER) Award, Howard will create algorithms, using machine learning techniques, to endow robots with the capabilities they need.

Current approaches to language understanding and generation for collaborative robots either directly consume sensor readings or use perception algorithms to construct a model of the environment.  Howard’s research has focused on the latter approach, with an emphasis on novel models that generate “minimal but sufficient” world models by leveraging language inputs to guide model construction.  This has allowed him and ECE PhD students Siddharth Patki and Jacob Arkin to develop robots that are capable of efficiently interpreting instructions like “Retrieve the ball inside the box” or “Pick up the middle block in the row of five blocks on the right”, which may require hypothesizing the location of unobserved objects based on spatial references from the language input or approximating inference in large symbolic representations. 

The improvements in grounded language communication Howard envisions would allow robots to respond to instructions that require knowledge beyond the present, such as “Hand me the wrench I was using five minutes ago,” or “Grab the cup that is about to fall off the table.” This would allow humans to use less constrained language to interact with robots on collaborative tasks without the oversight of robotics engineers.   It would also allow robots to use language with spatiotemporal references to provide humans with concise answers to questions they may ask.  

The challenge

The problem with language that refers to the past and/or future state of objects is a matter of efficiency.  Current robots are not capable of observing and tracking every object they see, nor are they able to project infinitely the state of these objects into the future.  “Imagine trying to track the motion of every object you interact with on a given day, from every picture frame on the wall, every fork in a drawer, every plug in a wall, to every pencil on a table,” Howard says.

And that could be overwhelming.

“Even if a robot was capable of doing this, it would be wildly inefficient because most of that knowledge would never be used.”   

Instead, Howard will develop a new iterative approach that selectively interprets the motion of selected objects from robot’s sensor data using clues from language.  “This research will use machine learning algorithms in several different ways, not only to associate language with symbols and environment features but also to understand how much data to use and whether simulating into the future is necessary,” he explains.  “This project will develop a unified approach for solving joint problems of language, perception, and simulation that are becoming increasingly important for human-robot teams.” 

Broadening the impact

CAREER awards are the National Science Foundation’s most prestigious recognition for early-career faculty members, recognizing individuals “who have the potential to serve as academic role models in research and education and to lead advances in the mission of their department or organization.” CAREER awards provide recipients with five years of funding to help lay the foundation for their future research.

Recipients must demonstrate how their research can have a broader educational impact.

Howard is revamping the three robotics courses he teaches at the University and will integrate aspects of his CAREER research in them. For example, students will have an opportunity to develop the hardware for an integrative robotic platform in one class, then “make it smart” by developing the software in the next class. “You build an artifact in the first one, but you keep on developing that artifact in the advanced classes. So, you’re not starting from scratch,” Howard says. The award will provide $14,000 for the purchase of components for students to build, then program 20 prototype robots.

The grant will allow him to support graduate and undergraduate students in each of the five years.  He will continue his work with the University's Kearns Center to host high school students as summer interns and contribute modules on robotics for the Upward Bound Math/Science program. Howard and his students will also continue to provide demonstrations of robotics at the Rochester Museum and Science Center.

About Howard

Howard started his professional preparation for a career in robotics as an undergraduate at the University of Rochester, where he completed a double major in electrical and computer engineering and mechanical engineering in 2004.

Prior to returning to the University in 2015, he earned a PhD in robotics from The Robotics Institute at Carnegie Mellon University and then held appointments as a research scientist and a postdoctoral associate at MIT's Computer Science and Artificial Intelligence Laboratory in the Robust Robotics Group, as a research technologist at NASA Jet Propulsion Laboratory in the Mobility and Robotic Systems section, and as a lecturer in mechanical engineering at Caltech.

This is Howard’s second early career award; he was one of only nine recipients to receive a NASA Early Career Faculty Award in 2019.  His other awards include Best Paper Awards at Robotics: Science and Systems (2016) and IEEE SMC (2017), two NASA Group Achievement Awards (2012, 2014), and a Wilmot Assistant Professorship (2019-2021).  Howard was also a finalist for the ICRA Best Manipulation Paper Award (2012) and the Kate Gleason Young Engineer of the Year Award (2021) from the Rochester Engineering Society.