Robot Sensors

Cameras and depth sensing

Cameras are the richest sense most robots have for the money. Colour cameras feed object recognition and the vision-language models that interpret instructions, while depth cameras add a distance to each pixel using stereo pairs, projected light patterns or time-of-flight timing. Head-mounted cameras give a humanoid its view of the room, and wrist cameras show the hand and object up close during a grasp. Cameras struggle with glare, darkness, glass and blank walls, so most robots back them up with other sensors. Specialist imaging goes further still, with industrial machine vision from Cognex or Keyence and thermal cameras from Teledyne FLIR built for inspection work.

LiDAR and laser eye safety

LiDAR measures distance by timing pulses of laser light and builds a point cloud of the surroundings, which mobile robots use for mapping, localisation and obstacle detection. It works in the dark and gives accurate geometry, but rain, dust, mirrors and very dark surfaces can confuse it. Ouster is one maker of these sensors. Because LiDAR relies on lasers, eye safety matters wherever people work nearby. IEC 60825-1:2014 sets out how laser products are classified according to their optical radiation hazard and what labels and instructions manufacturers must supply, so request the laser classification of any LiDAR fitted to a robot that will share space with staff or the public.

Inertial and joint sensing

An inertial measurement unit, or IMU, combines accelerometers and gyroscopes to report how the body is tilting and accelerating, which a legged robot needs to stay upright. Joint encoders report each joint's angle, and motor current or dedicated torque sensors estimate the load at each joint. Together these give proprioception, the robot's sense of its own body. Researchers at the University of Michigan showed the value of fusing them: their contact-aided invariant extended Kalman filter combined IMU readings, leg kinematics and foot contacts to estimate the pose and velocity of a Cassie-series biped, and outperformed a commonly used quaternion-based filter. Good state estimation is what lets a walking controller react to a slip in time.

Force and touch sensing

Force-torque sensors at the wrist measure the push and twist on the hand, letting a robot insert a peg, wipe a surface or pass an object to a person without crushing it. Tactile sensors go further and report where and how hard contact happens on a fingertip or palm. MIT's GelSight sensor, developed in Edward Adelson's group at CSAIL, places a camera behind a soft rubber block with a painted face to capture the shape of whatever presses into it; MIT reported robots using it to estimate the hardness of objects and to keep track of small tools such as screwdrivers when the gripper hid them from view. Touch data remains harder to use than vision, but it covers the moment of contact.

Sources and further reading

• IEC 60825-1:2014 Safety of laser products - Part 1: Equipment classification and requirements — International Electrotechnical Commission (IEC).
  How laser products, LiDAR included, are classed by optical radiation hazard and what labels and user information they need.
• Contact-Aided Invariant Extended Kalman Filtering for Robot State Estimation — University of Michigan (arXiv).
  Fuses IMU, leg kinematics and foot contact to estimate a walking robot's pose and velocity, tested on the Cassie biped.
• Giving robots a sense of touch — MIT News.
  How MIT's camera-based GelSight sensor lets robots judge hardness and handle small tools the gripper hides from view.

Common questions

Do humanoid robots need LiDAR?

Not always. Some humanoids rely mainly on cameras and depth sensors, while others add LiDAR for mapping and obstacle detection. The right mix depends on lighting, the size of the area and how precisely the robot must know where it is, so ask makers why they chose their sensor set.

How often do robot sensors need calibration?

It depends on the sensor and how hard the robot works. Cameras and LiDAR can drift after knocks or temperature swings, and force sensors may need zeroing between tasks. Ask the maker for the recommended calibration routine, the tools it needs and whether the robot can check itself.