Semester Project: Learning Motion in Manipulation Tasks from Demonstration
Published:
Implementation
1. Virtual Reality Based Robot Teleoperation
This section presents a virtual reality (VR)-based teleoperation system for controlling two Franka Emika Panda robots. The system utilizes an HTC Vive Pro Kit as the VR hardware, interfaced with Robot Operating System (ROS). The setup enables intuitive real-time control and demonstration of robotic manipulation tasks.
- VR Interface: Use HTC Vive controllers to track hand movements.
- Control System: The controller data is mapped to robot end-effectors through SteamVR and OpenVR.
- Safety Features: Includes workspace boundary checking and impedance control for safe interactions.
2. Haptic Feedback Teleoperation
To enhance control accuracy and user experience, a haptic device sigma.7 was introduced for bilateral teleoperation. Key components include:
- Haptic Feedback: Provides real-time force feedback using a Bota Systems SensOne 6D force-torque sensor.
- Velocity Control: Implements relative velocity scaling to map operator movements to the robot end-effector.
- Clutch Mechanism: A foot pedal mechanism enables smooth transitions between tasks.
3. Object Pose Estimation using Vision-Based Methods
An object recognition system was developed using Grounded-SAM, combining 2D segmentation and depth information. The steps include:
- Object Segmentation: Uses Grounding DINO for object detection with text prompts.
- Depth Processing: Intel RealSense D435 captures depth information for precise pose estimation.
- Data Filtering: Implements noise reduction using PCA and statistical outlier removal.
4. Vision-Based Tactile Sensing
To improve performance in contact-rich tasks, the GelSight Mini sensor was employed for high-resolution tactile sensing. This enables the robot to measure:
- Contact forces and object surface geometry
- Texture recognition for precise grasping
Semester Project Report
Below, you can view the Semester Project Report directly or download it for local viewing: