Background Teleoperation has emerged as a crucial field during the Covid-19 period, playing a significant role in enabling humans to carry out complex remote jobs effectively. Robotic arms, as one of the human-friendly tools, provide convience for the users to perform simple tasks like grasping and placing. However, this advancement has also brought some challenges including but not limited to Perception, Non-haptic Feedback and Motion Sickness, which has sparked the interest of researchers, leading them to explore effective solutions to provide robust control for the robot arms in remote scenarios.

Overview A VR-based teleoperation system has been established in this project to help facilitate human experience while performing remote work. The entire framework is set up under a simulation environment (Unity3D) with two Kinova arms -- One arm called Manipulator Arm is operated by users to execute remote task; Another called Camera Arm is designed to provide real-time views for the users using a single camera as their perception input, which is attached to the end-effector. Shared autonomous control is achieved by applying IBVS (Image-based Visual Servoing) algorithm to the Camera Arm. Also, interfaces for VR device (Oculus) have been integrated, allowing users to get a view presence in the headset and enabling them to control the Manipulator Arm using the controller. This setup enables operators to focus solely on controlling the Manipulator Arm to perform tasks, while the Camera Arm autonomously adjusts its view in real-time, significantly reducing the user's workload.