Back close

Monocular Tracking of Human Hand on a Smart Phone Camera using MediaPipe and its Application in Robotics

Publication Type : Conference Paper

Publisher : IEEE

Source : 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC)

Url :

Keywords : Hand-tracking,Depth Estimation,Perspective projection,Google MediaPipe, Robot Position Control,Robot Kinematics

Campus : Bengaluru

School : School of Engineering

Department : Mechanical

Year : 2021

Abstract : With Industry 4.0, robots are finding their way into the production lines of multiple industries. The conventional methods to program these robots require special training, help from a robot technician, or have a basic understanding of robot kinematics. Integration of machine learning into robotics can make the programming process shorter, cost-effective and user friendly. In this paper, we propose to use a customizable machine learning solution of ‘Google MediaPipe Hands' to track human hands from a monocular camera of a smartphone and use it for robotic applications. MediaPipe Hands provides only a 2.5D pose estimation; we propose to use a simple calibration and the concept of perspective projection to get the 3D position of the hands relative to the smartphone. By conducting multiple experiments, we found that the hand tracking solution has satisfactory accuracy rates. We developed robot simulations in Python to check the viability of the hand tracker for robotic applications. We could accurately control the end-effector movement using the hand tracker.

Cite this Research Publication : Sreehari Sreenath (2019-21 MTech), D. Ivan Daniels (2016-20), A. S. D. Ganesh (2016-20), Yashaswi S. Kuruganti (2016-20) and Rajeevlochana G. Chittawadigi has presented a paper titled, “Monocular Tracking of Human Hand on a Smart Phone Camera using MediaPipe and its Application in Robotics” 9th IEEE Region 10 Humanitarian Technology Conference (R10 HTC) during Sept. 30 to Oct 2, 2021, held in hybrid mode (online and physical at Bengaluru). Conference proceedings in IEEE Xplore.

Admissions Apply Now