Learning-Finding-Giving: A Natural Vision-Speech-based Approach for Robots to Assist Humans in Human-Robot Collaborative Manufacturing Contexts

Emilio Herrera, Maxim Lyons, Jesse Parron, Rui Li, Michelle Zhu, Weitian Wang

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Human-robot collaboration can improve and enhance current manufacturing processes, in which robots are able to provide collaborative assistance to humans, allowing for increased productivity and minimal time waste. The use of everyday mechanical appliances and tools is unavoidable, making these handover tasks common in human-robot collaborative manufacturing contexts. A typical handover task can be performed in three general steps: object identification, object grasping, and object handover. In this work, we propose a learning-finding-giving framework based on computer vision and speech recognition approaches for robots to dynamically identify and deliver tools for human partners in collaborative tasks. The YOLOv5 object detection algorithm is utilized for the identification of common mechanical tools. To teach robots to understand the target objects, a custom dataset is created from over 2000 images of the mechanical tools, followed by the implementation in real-world human-robot collaborative tasks. Experimental results and evaluations show that the proposed solution allows robots to dynamically understand and grasp tools with high accuracy, effectively assisting in handover tasks for human teammates.

Original languageEnglish
Title of host publication2024 IEEE 4th International Conference on Human-Machine Systems, ICHMS 2024
EditorsMing Hou, Tiago H. Falk, Arash Mohammadi, Antonio Guerrieri, David Kaber
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9798350315790
DOIs
StatePublished - 2024
Event4th IEEE International Conference on Human-Machine Systems, ICHMS 2024 - Hybrid, Toronto, Canada
Duration: 15 May 202417 May 2024

Publication series

Name2024 IEEE 4th International Conference on Human-Machine Systems, ICHMS 2024

Conference

Conference4th IEEE International Conference on Human-Machine Systems, ICHMS 2024
Country/TerritoryCanada
CityHybrid, Toronto
Period15/05/2417/05/24

Keywords

  • handover
  • human-robot collaboration
  • object detection
  • robotics
  • smart manufacturing

Fingerprint

Dive into the research topics of 'Learning-Finding-Giving: A Natural Vision-Speech-based Approach for Robots to Assist Humans in Human-Robot Collaborative Manufacturing Contexts'. Together they form a unique fingerprint.

Cite this