Project Overview
Rosie is a home assistant robot designed for object detection, grasping, and navigation. This project explores:
- Simulation in Isaac Sim for testing movements and object interactions.
- Object detection using YOLOv8 for recognizing and picking objects.
Currently, Rosie consists of:
- ✅ A humanoid robot STL file sourced from cults3d Rosie STL
- ✅ A mecanum-wheeled car base (used in a previous project)
- ✅ Components sourced from AliExpress
Technical Specifications
- Programming Languages: Python, C++
- Frameworks & Libraries: ROS2, PyTorch, OpenCV, Micro-ROS, MoveIt2
- Hardware Components:
- Computing: NVIDIA Jetson Nano, Raspberry Pi, ESP32
- Sensors: LiDAR, IMU, Depth Cameras
- Core Algorithms:
- SLAM & Nav2 (Navigation2 Framework) – Used for autonomous mapping and path planning
- MoveIt2 for motion planning and manipulation
1. Simulation in Isaac Sim
Challenges in Simulating Mecanum Wheels
One of the biggest issues in Isaac Sim was properly simulating mecanum wheels. The physics engine did not handle their omnidirectional movement well, leading to challenges in achieving smooth motion.
Current Approach:
- Used manual control for movement while working on better simulation.
- In real-world tests, SLAM successfully mapped the environment, and Nav2 handled navigation, but replicating this in Isaac Sim remains a challenge.
Next Step:
- Continue improving wheel simulation for better accuracy.
Screenshot of Isaac Sim Simulation
2. 3D Printing and Hardware Testing
I was able to test Rosie on real hardware! The robot was mostly 3D printed, which made it somewhat unstable, but it successfully demonstrated the core functionalities including movement and basic object interaction.
Hardware Testing Outcome:
- ✅ Successfully tested on real hardware
- ✅ Robot worked despite being mostly 3D printed
- ⚠️ Stability issues due to 3D printed components, but overall functional
Mobile Base Integration: Mounting the compute module and electronics to the chassis for navigation testing.
Manipulator Assembly: Organizing batches of custom 3D-printed components for the robotic arm (featuring my assistant 🐶).
3. How Rosie Moves and Picks Objects
This demo showcases Rosie’s movement and object picking in Isaac Sim.
Key Features:
- Mecanum wheel movement in simulation.
- Object selection via a UI system—clicking an object assigns its coordinates for picking.
- Bounding box calculation determines how the robot should grasp the object.
Watch the Isaac Sim Demo Video Here: Rosie Demo Video Screenshot of Object Selection UI in Isaac Sim
4. Object Picking: AAB vs. OBB Bounding Boxes
Initially, I used Axis-Aligned Bounding Boxes (AAB), but they were inaccurate for irregular objects.
Why AAB Didn’t Work:
- ❌ Misalignment when objects were rotated
- ❌ Inaccurate grasping due to fixed orientation
Why OBB is Better:
- ✅ Adjusts for object rotation, ensuring better grasping
- ✅ Allows the robot’s gripper to align properly
5. Object Detection Using YOLOv8
Rosie currently uses a pre-trained YOLOv8 model for object detection.
Future Plans:
- Train YOLO on household objects for better recognition.
- The goal is to identify and grasp objects commonly found in homes.
6. Transition from Gazebo to Isaac Sim
Initially, Rosie was simulated in Gazebo running inside VMware Ubuntu, but it had performance issues.
Problems with Gazebo in VMware:
- ❌ Slow simulation speed due to limited NVIDIA GPU utilization.
- ❌ Inefficient reinforcement learning training.
Switch to Isaac Sim:
- ✅ Better GPU acceleration for improved performance.
- ✅ More realistic physics simulation for reinforcement learning.
7. Conclusion
Rosie successfully demonstrated home assistance capabilities through both simulation and real hardware testing. Despite stability challenges from 3D printed components, the robot proved the viability of the design. I’m now exploring PPO techniques for other robotics projects!
🔹 Links & References
- Demo Video: Rosie Demo Video
- Code: https://github.com/SamsonAjadalu/Rosie-Robot
- STL File Used for Rosie: Rosie STL