Emotion-based Robotic Action Optimization System for Human-Robot Collaboration

Although collaborative robots aim to boost productivity in manufacturing, misalignment between robot's actions and the human's intentions of the collaboration can cause discomfort or frustration, potentially discouraging future collaborations. Inspired by human-to-human interactions, this paper aims to help solve this problem by enabling a collaborative robot to adjust how it moves and acts based on human emotions to improve the overall collaboration process. To achieve this goal, an emotion-based robotic action optimization system was developed and integrated into a collaborative robot. The system utilizes hierarchical reinforcement learning (HRL) to train and guide the robot to adjust its actions according to detected human emotions. Specifically, this paper introduces (1) a HRL model that leverages a vision-audio-based emotion recognition model to determine and adjust robot actions (movement speed, drop-off distance, reaction time, and rate of success) according to human emotions. The goal of this model is to avoid negative emotions of the human user that are triggered by the robot actions. (2) A robot motion control method driven by recognized human intentions and actions from the HRL model, guiding the robot arm and gripper to adjust movements and deliver parts as desired. (3) objective and subjective evaluation experiments to evaluate the effectiveness of the developed system. The results and analysis of the experiments demonstrated the effectiveness of our developed system in a human-robot collaboration setting.