Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer

• Main Authors: Lu, Jingpei, Richter, Florian, Yip, Michael
• Format: Journal Article
• Language: English
• Published: 08-02-2022
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Robotics
• DOI: 10.48550/arxiv.2010.08054

in IEEE Robotics and Automation Letters, vol. 7, no. 2, pp.
4622-4629, April 2022 Keypoint detection is an essential building block for many robotic
applications like motion capture and pose estimation. Historically, keypoints
are detected using uniquely engineered markers such as checkerboards or
fiducials. More recently, deep learning methods have been explored as they have
the ability to detect user-defined keypoints in a marker-less manner. However,
different manually selected keypoints can have uneven performance when it comes
to detection and localization. An example of this can be found on symmetric
robotic tools where DNN detectors cannot solve the correspondence problem
correctly. In this work, we propose a new and autonomous way to define the
keypoint locations that overcomes these challenges. The approach involves
finding the optimal set of keypoints on robotic manipulators for robust visual
detection and localization. Using a robotic simulator as a medium, our
algorithm utilizes synthetic data for DNN training, and the proposed algorithm
is used to optimize the selection of keypoints through an iterative approach.
The results show that when using the optimized keypoints, the detection
performance of the DNNs improved significantly. We further use the optimized
keypoints for real robotic applications by using domain randomization to bridge
the reality gap between the simulator and the physical world. The physical
world experiments show how the proposed method can be applied to the
wide-breadth of robotic applications that require visual feedback, such as
camera-to-robot calibration, robotic tool tracking, and end-effector pose
estimation.