Though robots can execute a wide range of vital actions, and their importance has notably increased during the pandemic, the human-robot discussion is still in its infancy because of the machine’s lack of feeling. With this specific issue in mind, Cornell University researchers created a way for soft robots to identify human touch by examining a person’s shadows using a camera.
Currently, several groups are developing electronic skin for robots. But the Cornell University researchers are applying a more straightforward approach, utilizing shadow-imaging cameras to provide robots with the sense of touch.
The method, known as “ShadowSense,” is low-cost as it can detect various physical human interactions by using an off-the-shelf USB camera. The device “captures” the shadows produced by hand gestures near and in direct contact with the robot’s skin. Machine-learning software analyzes the movement and categorizes it.
Guy Hoffman, the paper’s senior author, and a Mills Family Faculty Fellow and associate professor at Cornell University, said:
Touch is such an important mode of communication for most organisms, but it has been virtually absent from human-robot interaction. One of the reasons is that full-body touch required many sensors and was therefore not practical to implement. This research offers a low-cost alternative.
Roboticists have deemed full-body touch for robots impractical thus far because the feat requires so many contact sensors. The new method by Hoffman and his colleagues provides an efficient alternative. Hoffman explains in the paper that the technology enables connection with robots without the need for large, costly sensor arrays. Furthermore, removing the need for contact sensors reduce the weight and wiring required.
Doctoral student Yuhan Hu, the paper’s lead author, said:
By placing a camera inside the robot, we can infer how the person is touching it and what the person’s intent is just by looking at the shadow images. We think there is interesting potential there because there are lots of social robots that are not able to detect touch gestures.
The technology originated as part of a collaborative project to develop inflatable robots that may guide people to safety during crisis evacuations – like through a smoke-filled building where the robot could identify the touch of the hand and lead the person to an exit.
The researchers’ prototype robot includes a soft inflatable bladder of nylon skin stretched around a cylindrical skeleton, approximately four feet tall, installed on a mobile base. Under the skin is the robot’s USB camera, which connects to a laptop computer.
The researchers create a neural network-based algorithm that uses formerly recorded training data to differentiate between six touch gestures – not touching at all, touching with a palm, touching with two hands, punching, hugging, and pointing. They trained their physical interaction classification algorithm with shadows relating to those six gestures. People can even program the system to respond to specific gestures and touches.
Their robot can successfully distinguish the different gestures with an accuracy of 87.5% to 96%. Lighting conditions impact the outcome.
And manufacturers can potentially convert the robot’s skin into an interactive display screen. Actually, the device doesn’t have to be a robot at all! The ShadowSense technology’s applications aren’t limited to robotics. Someone can incorporate it into other materials, such as balloons, turning them into touch-sensitive products.
Hoffman said:
While the technology has specific limitations, for example, requiring a type of sight from the camera to the robot’s epidermis, these constraints could spark an approach that is new social robot design that could support a visual touch sensor like the one we proposed. In the future, you want to experiment with using optical devices such as, for example, lenses and mirrors to enable the additional form.
Not only does ShadowSense make robots more user-friendly, but it also provides privacy – a comfort that is becoming rare in these high-tech times. Using shadows and touch as the method of interaction addresses some of the privacy concerns around voice and facial recognition.
He said:
Touch interaction is the same channel that is binding terms of human-human interaction. It’s a modality that is intimate. And that’s not easily replaceable.
If the robot can only see you in the shape of your shadow, it can identify what you’re doing without taking fidelity that is most of your look. That provides you a physical filter and protection and offers psychological comfort.
The team said mobile guide robots could use the technology to respond to taps and pokes from nearby humans. Likewise, home droids that provide the elderly with some company and help with simple predetermined tasks could use the technology.
