Project Overview

Rosie is a home assistant robot designed for object detection, grasping, and navigation. This project explores:

  • Simulation in Isaac Sim for testing movements and object interactions.
  • Object detection using YOLOv8 for recognizing and picking objects.
  • Reinforcement learning (PPO) to improve grasping accuracy.

Currently, Rosie consists of:

  • ✅ A humanoid robot STL file sourced from cults3d Rosie STL
  • ✅ A mecanum-wheeled car base (used in a previous project)
  • ✅ Components sourced from AliExpress

Technical Specifications

  • Programming Languages: Python, C++
  • Frameworks & Libraries: ROS2, PyTorch, OpenCV, Micro-ROS, MoveIt2
  • Hardware Components:
    • Computing: NVIDIA Jetson Nano, Raspberry Pi, ESP32
    • Sensors: LiDAR, IMU, Depth Cameras
  • Core Algorithms:
    • SLAM & Nav2 (Navigation2 Framework) – Used for autonomous mapping and path planning
    • MoveIt2 for motion planning and manipulation
    • Reinforcement Learning (PPO) for grasping optimization

1. Simulation in Isaac Sim

Challenges in Simulating Mecanum Wheels

One of the biggest issues in Isaac Sim was properly simulating mecanum wheels. The physics engine did not handle their omnidirectional movement well, leading to challenges in achieving smooth motion.

Current Approach:

  • Used manual control for movement while working on better simulation.
  • In real-world tests, SLAM successfully mapped the environment, and Nav2 handled navigation, but replicating this in Isaac Sim remains a challenge.

Next Step:

  • Revisit wheel simulation after NYSC to improve accuracy.

Screenshot of Isaac Sim Simulation Screenshot of Isaac Sim Simulation

2. 3D Printing and Assembly

Rosie is currently in the hardware assembly phase, but due to time constraints, it is not fully built yet.

Current Status:

  • ✅ Most parts have been 3D printed, but full assembly is pending.
  • ✅ Printing and assembly will resume after my NYSC.
  • ✅ Once complete, a full showcase and technical breakdown will be published.

Photo of 3D Printed Parts

3. How Rosie Moves and Picks Objects

This demo showcases Rosie’s movement and object picking in Isaac Sim.

Key Features:

  • Mecanum wheel movement in simulation.
  • Object selection via a UI system—clicking an object assigns its coordinates for picking.
  • Bounding box calculation determines how the robot should grasp the object.

Watch the Isaac Sim Demo Video Here: Rosie Demo Video Screenshot of Object Selection UI in Isaac Sim Screenshot of Object Selection UI in Isaac Sim

4. Object Picking: AAB vs. OBB Bounding Boxes

Initially, I used Axis-Aligned Bounding Boxes (AAB), but they were inaccurate for irregular objects.

Why AAB Didn’t Work:

  • ❌ Misalignment when objects were rotated
  • ❌ Inaccurate grasping due to fixed orientation

Why OBB is Better:

  • ✅ Adjusts for object rotation, ensuring better grasping
  • ✅ Allows the robot’s gripper to align properly

5. Object Detection Using YOLOv8

Rosie currently uses a pre-trained YOLOv8 model for object detection.

Future Plans:

  • Train YOLO on household objects for better recognition.
  • The goal is to identify and grasp objects commonly found in homes.

6. Transition from Gazebo to Isaac Sim

Initially, Rosie was simulated in Gazebo running inside VMware Ubuntu, but it had performance issues.

Problems with Gazebo in VMware:

  • ❌ Slow simulation speed due to limited NVIDIA GPU utilization.
  • ❌ Inefficient reinforcement learning training.

Switch to Isaac Sim:

  • ✅ Better GPU acceleration for improved performance.
  • ✅ More realistic physics simulation for reinforcement learning.

7. Reinforcement Learning with PPO

Rosie will use PPO (Proximal Policy Optimization) to improve grasping accuracy. It is stable for continuous control and ideal for robotic grasping.

8. Next Steps

Once I complete my NYSC, I will continue with the following tasks:

  • ✅ Complete 3D printing and assembly of Rosie.
  • ✅ Train a custom YOLO model for object detection.
  • ✅ Improve wheel simulation in Isaac Sim.
  • ✅ Finalize reinforcement learning for grasping.
  • ✅ Compare real-world vs. simulated performance.
  • ✅ Publish the full code and documentation on GitHub.

9. Conclusion

Rosie is a work-in-progress home assistant robot. With my NYSC ending soon, I will have time to complete the project and publish the full code and documentation.

🔹 Links & References