Jetson Nano Robot - Realsense, RPLidar, Joysticks

Jetson Nano control and vision with 4-Wheel Steering, ROS2 RealSense2, RPlidar, BNO055, Python3, Pygame and ModBus

Public Chat
Similar projects worth following
Nvidia Jetson Nano control and vision with 4-Wheel Steering, ROS2 RealSense2, RPlidar, BNO055, Python3, Pygame and ModBus to drive the Jetson Nano using modbus joystick commands

Nvidia Jetson Nano control and vision with 4-Wheel Steering, ROS2 RealSense2, RPlidar, BNO055, Python3, Pygame and ModBus  to drive the Jetson Nano using modbus joystick commands

Using: PC - Ubuntu 20.04 and ROS2 FOXY

  • 2 Joysticks, Thrustmaster HOTAS Joystick and Throttle

Jetson Nano - Ubuntu 18.04 and ROS2 Eloquent - Adafruit PCA9685 16 Channel PWM driver board - BNO055 gyro - USB wifi card - RPLidar - Realsense D435 - 4-Wheel Steering

Joystick Control - Pygame and Modbus: PyModbus

sudo pip3 install adafruit-servokit

sudo pip3 install pymodbus

sudo pip3 install pygame

Build the ROS2 packages if needed - realsense2, rplidar, bno055 or use your own camera solution

colcon build --symlink-install

start ROS2 Realsense And Lidar Nodes on the Jetson 

ros2 launch realsense_examples

Lidar node - Lidar works well in ROS2 but it also has python code unimplemented now

sudo chmod 666 /dev/ttyUSB0

ros2 launch rplidar_ros

Visualization - For the cameras, I keep the subscriptions to a minimum for a faster frame rate for just driving. Nav2 and slamtoolbox create maps and can create pathways but with no control node yet, I'm using Python.


Python code

I actually need to get all this running in ROS2, but that becomes difficult when I want my PC to run Ubuntu 20.04 which is fine and can run ROS2 Foxy but... The Jetson Nano image only runs Ubuntu 18.04 and that only allows it to run ROS2 Eloquent which has less features.

Also running all the nodes in ROS2 gets crazy with 8 or more terminals open and although it's well written and distributed, there are many working parts all connected so super easy to break something when coding.

Also would have to build my own ROS2 PCA9658 package as one does not exist.

There is a Jetson Nano 20.04 image out there which would be great but i couldn't get it to work. There are several revision versions of the Nano so that makes it even harder.

Pygame is the easiest way to get Joystick data from the PC but pygame doesn't like to run remotely because it uses a display and can't find one through SSH. There are ways around this through Pygame but also around Pygame using ModBus.

I had never heard of ModBus but apparently it's ancient and works great.

I run these scripts in different terminal windows - next step, one PC run script and one Robot run script

sudo python3 

Run on the PC. Starts the ModBus server - Set for IP transfer. You will need to change the IP addresses to the machine running the server. can just use localhost addresses. Must change the settings to use UDP, binary, ASCII or serial transfers - there are async server examples in the documentation at: The server can be configured to only receive a set number of commands, but is currently configured to the maximum number  I don't know what that is, hundreds or thousands I'd guess even though I'm only sending 5 right now.

sudo python3 

Run on the PC. Starts the Pygame Joystick modules and Modbus client -You will need to set your IP address to the machine running the sync_server. can use localhost addresses if its running on the same machine. Currently set for the Thrustmaster dual joystick setup described above. Need to set Joystick numbers, axis and buttons so your rig will work if you don't have the Thrustmaster setup. Only 5 axis/buttons transferring to the server now, you could set up all axis and all buttons if you wanted but I'm keeping it light and only need the 5 to make it work. I'm looking forward to having most of the buttons and hats do something cool like move the camera with servos, make sounds from a speaker or enable the lidar for obstacle avoidance. The PCA9658 can control up to 16 servos or esc's per board and you can daisy-chain 99 boards so that's a lot of servos!

sudo python3

Joystick Detection Helper...

Read more »

View all 13 components

View all 2 project logs

Enjoy this project?



HACKRC wrote 12/04/2020 at 22:17 point

Yes it's fast, I have the speed limited in the ros2_pca9685 package. It can also be limited in the Nav2 package parameters. You can have a different scale set up for the joysticks so you can run faster while joystick-controlled if needed. But to answer your question I have not done any speed tests yet; its a great question I'm guessing the maximum speed would be too fast for the controllers to keep up but I'd like to find out too 

  Are you sure? yes | no

Humpelstilzchen wrote 12/04/2020 at 08:35 point


Your chassis looks fast, so I wonder: What speed can you drive autonomously?

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates