Multi-Camera Depth Sensing
I built a real-time depth perception pipeline for a small marine vessel using four wide-angle cameras mounted at the bow, port, starboard, and transom. The goal of this is to create a training environment where I can create and develop a machine learning system for navigating small boats in tight marinas. Each feed is run through a one-shot depth estimation model and fused with virtual lidar sensor data to create a scaled point cloud. The camera perspectives are fused together and mapped to physical space using a virtual IMU and GPS inside the simulation. This creates the 3D point cloud on the right of the dashboard below. Additionally, I used a static mask to remove point cloud data that hits the hull. Without this fix, the one-shot depth estimation map was stretched by lidar and frame data around the hull of the boat.
The camera layout is designed to cover medium- to small-sized vessels for close-quarters maneuvering: forward and lateral views cover docking with an aft camera to watch wake and trailing traffic. I tested a number of one-shot depth estimation models and found Depth Anything V2 to be the best quality at the lowest compute cost. For reference, I achieved around 12 FPS on the computer vision plotting pipeline and simulation while running on an NVIDIA 4070 laptop (8 GB card). This was generally pretty poor performance because the simulation was being run with graphics and the computer vision system is 3D rendering its point cloud. The non-rendering environment will be more performant.
Simulation Environment
I created a modular boat simulation environment that uses voxel-based buoyancy, wind, current, and waves. The simulation can model any boat with any number of bow thrusters, rudders, inboard drive units, or fixed shaft systems. For a test boat, I used a tugboat with a single rudder, fixed-shaft propeller, and one forward bow thruster. The simulation passes data through a TCP simulation bridge. The simulation can be controlled manually by hand or from a networked ROS controller.
While there are a number of other simulation environment software packages such as MARUS and GRI, I decided to make my own for a couple of reasons.
First, existing simulations did not seem to have realistic buoyancy physics or account for the complexities of water surface simulation. In my simulation the waves themselves divert the boat's path based on hull geometry and orientation. The hull design and shape change performance and handling of the boat, so a simple yaw-axis drag or four-point buoyancy estimation would not create a realistic environment to train in.
Second, my custom simulation can have easily configurable virtual sensors without needing to fit another framework. For example, I simulated a solid-state lidar chip with "cone" cast beams replicating the sensing geometries of the beams from the documentation. Other simulations have simple downward depth sensors that use a straight ray cast. Most of the other sim environments I found had their own layers of abstraction on sensors, environment, and physics, etc., most of which I don't need. Creating my own framework in Unity, I can easily create virtual sensors to capture anything I want.
Environment Generation
Now that I have a base RL environment with sensor data, I started work on a way to create unique training scenarios. I created a system using Google Maps 3D tiles that lets me generate a rough 3D mesh of a location based on center coordinates. I took a list of every US recreational marina and its coordinates so that I can now dynamically generate a model of each. The problem is the mesh is low resolution and would probably not be a good analog for RL vision pipeline. I set out to simplify some recognizable elements in the environment into clearer 3D models. I used OSM Street Map data to get vector paths of docks in the marina shown in red. I quickly realized the accuracy of this data was poor. Many docks are missing and they are offset by 1-2 meters.
Future Plans
The main challenge is creating a realistic training data set across multiple marinas for an end-to-end depth-vision action model. While a YOLO model can extract dock positions more accurately, boats are harder: their generated 3D models are blocky, and without accurate boat geometry the vision-action pipeline cannot learn reliable collision avoidance from appearance alone. Training the model to respond to specific boat images would compound the training needed to achieve positive results as well as the complexity of the RL policy. To simplify, I plan to use the depth-maps and cameras to extract simplified spatial representations of docks, boats, and environment and feed them to an RL policy. This way I can use satellite data to generate training environments from satellite maps instead of relying on camera vision pipeline in training. There is a significant risk that my camera vision system needs to be able to reliably identify features like docks and boat perimeters in the real world in a way similar to the simulation. I can help this by adding noise to my satellite-data-generated environment so that the model is trained for the uncertainty and "fuzzy" nature of the physical sensors.
My next steps will be to create a program to detect 3D orientation and mesh of docks and boats from the onboard cameras and depth sensors. After that I can begin development of RL policy, reward structure, and training.