DeepX has developed an autonomous operation system for hydraulic excavators, and in April 2023, in collaboration with Fujita Corporation, it successfully demonstrated the autonomous operation of a continuous cycle of excavation and dump loading as a single operation at a construction site.

Stabilizing complex systems is essential for achieving autonomous operation at construction sites

The autonomous operation of systems that adapt to the surrounding environment and objects based on various conditions is complex, and there have been challenges in achieving continuous and stable operation of such systems, including autonomous driving systems.
For example, in actual construction sites, it is difficult to fix the conditions for dump trucks to hydraulic excavators to a single setting. During the loading process, the hydraulic excavator needs to flexibly adjust its movements based on complex conditions such as recognizing the surroundings of the hydraulic excavator, the position and orientation of the dump truck, and the quantity and shape of the soil inside the vessel. Additionally, the shape and hardness of the excavated soil vary depending on the situation, requiring real-time adaptation to these diverse conditions during excavation. However, generally, as the system needs to handle more complex conditions, the stability of the system tends to be compromised.
There have been challenges in achieving systems that can handle complex conditions while maintaining the essential element of stability, which is crucial for practical implementation.

The autonomous operation of repetitive and continuous excavation and loading tasks was demonstrated on-site.

In this demonstration, we achieved the autonomous operation of a continuous cycle of excavation and dump loading at Fujita’s construction site. The demonstration confirmed the following capabilities: the hydraulic excavator’s ability to autonomously adjust its movements based on the dump truck’s specifications (position, orientation, soil volume inside the vessel, etc.), and its ability to autonomously adjust its movements based on the shape of the specified excavation area and the remaining soil volume within a certain range. The stability of the system was also confirmed, as it operated continuously for approximately 8 hours, enabling more than 200 cycles of excavation and loading in a single day.

DeepX built an autonomous hydraulic excavator system based on ROS2.

DeepX’s main role involved software development, provision, and on-site integration services. Regarding the software, DeepX developed not only the underlying robotic system but also perception and control algorithms, as well as simulators for development and verification purposes. Fujita, on the other hand, handled arrangements for the demonstration site and hardware required for the verification experiments.

Developed software

Robotic system

DeepX developed a robotic system based on ROS2. In this demonstration, DeepX focused on the robustness and scalability of the system, with the aim of considering practical operation after the demonstration. Therefore, DeepX conducted development with a particular emphasis on the system’s robustness and extensibility.

Dump truck detection

DeepX developed a system that processes point cloud data from LiDAR to provide real-time perception of the relative position, orientation of the dump truck from the hydraulic excavator’s perspective, the shape of the vessel, and the loading status. In particular, to enable adaptation to various conditions such as different positions, orientations, and soil shapes, DeepX utilized deep neural networks for algorithm development.

Digging path generation

DeepX developed a system that automatically generates excavation trajectories based on the shape of the soil. In this demonstration, DeepX confirmed that the system is capable of dynamically adjusting the excavation depth and location according to the shape of the soil, allowing for appropriate variations in excavation movements.

Development team

In this demonstration, a solution team closely collaborated with customers to facilitate requirements definition and verification, while a software engineering team developed the autonomous driving system. The software development and verification were carried out through the collaboration of these teams. Notably, the software engineering team comprised members from various nationalities, forming a global team.

Foreign engineer members engaged in field demonstration experiments.

 

Future outlook

Based on the software developed in this project, we plan to move towards the verification of autonomous operation in large-scale construction sites. We will continue to pioneer the frontier of construction machinery autonomous driving for practical implementation in the future.