Pose-and-shear-based tactile servoing

• Published in: The International journal of robotics research Vol. 43; no. 7; pp. 1024 - 1055
• Main Authors: Lloyd, John, Lepora, Nathan F.
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01-06-2024; SAGE PUBLICATIONS, INC
• Subjects: Artificial neural networks; Bayesian analysis; End effectors; Group theory; Kinematics; Lie groups; Mathematical models; Neural networks; Pose estimation; Pushing; Robot dynamics; Robots; Servocontrol; Shear; Tactile sensors (robotics); Three dimensional motion; deep learning; tactile servoing; Tactile sensing; feedback control; object manipulation; Bayesian filtering; robotic pushing
• ISSN: 0278-3649; 1741-3176
• DOI: 10.1177/02783649231225811

Tactile servoing is an important technique because it enables robots to manipulate objects with precision and accuracy while adapting to changes in their environments in real-time. One approach for tactile servo control with high-resolution soft tactile sensors is to estimate the contact pose relative to an object surface using a convolutional neural network (CNN) for use as a feedback signal. In this paper, we investigate how the surface pose estimation model can be extended to include shear, and utilise these combined pose-and-shear models to develop a tactile robotic system that can be programmed for diverse non-prehensile manipulation tasks, such as object tracking, surface-following, single-arm object pushing and dual-arm object pushing. In doing this, two technical challenges had to be overcome. Firstly, the use of tactile data that includes shear-induced slippage can lead to error-prone estimates unsuitable for accurate control, and so we modified the CNN into a Gaussian-density neural network and used a discriminative Bayesian filter to improve the predictions with a state dynamics model that utilises the robot kinematics. Secondly, to achieve smooth robot motion in 3D space while interacting with objects, we used SE(3) velocity-based servo control, which required re-deriving the Bayesian filter update equations using Lie group theory, as many standard assumptions do not hold for state variables defined on non-Euclidean manifolds. In future, we believe that pose-and-shear-based tactile servoing will enable many object manipulation tasks and the fully-dexterous utilisation of multi-fingered tactile robot hands.