Since 2017, DeepX has been engaged in a hydraulic shovel automation AI project in collaboration with the Fujita Corporation (hereafter, Fujita). Fujita is a semi-major general contractor that provides services ranging from design through to construction, and that possesses extensive knowledge and technologies in areas such as operations issues, worksite needs, remote operation of heavy machinery, and hardware development.

There has been demand in recent years for employing AI in order to automate work in a wide range of industries, owing to the labor shortages resulting from a declining and aging population. In the construction industry, in particular, there is a significant labor shortage that is only exacerbated by the eventual retirement of skilled operators. In addition, there has long been a need for the development of technologies that would permit the unmanned operation and automation of machinery, due to the ever-present risk of major worksite accidents, such as collapsing structures and landslides.

With these concerns in mind, 1991 saw Fujita begin the development of remote operation systems that would make it possible to perform unmanned operations at hazardous worksites. These systems allow construction equipment to be operated from a remote site by an operator who is receiving images via a screen. In 2017, Fujita and DeepX began the joint development of artificial intelligence for the automated control of unmanned hydraulic shovels, with the hope of providing a solution to the emerging problem of labor shortages.

In this article, the past results and future prospects of this project are discussed by Katsuhiko Kawakami from the Machinery Division of the Fujita Civil Engineering Center, Hikaru Fushimi from the Advanced Systems Development Division of the Fujita Technology Center, and DeepX AI engineers Joji Toyama and Kohei Nishimura.

A declining number of engineers and harsh worksite environments: the urgent need for solutions via automation

Toyama (DeepX): This project aims to make use of AI in order to achieve the automated operation of hydraulic shovels. Even before our involvement, however, Fujita had already developed systems of its own, allowing for the remote operation of such machinery.

Kawakami (Fujita): That’s right. Fujita already had the knowledge required to allow the operation of construction machinery via remote, but systems of that nature still require an operator. That’s why we wanted to employ AI toward achieving fully unmanned machine operation, and so began collaborating with DeepX.

Fushimi (Fujita): Achieving unmanned operation is a pressing need in the construction industry. One reason for this is the shrinking workforce. At present, one in three persons working in the construction industry is aged 50 or more, and there is a growing shortfall in younger workers as well. In addition, there is an immense time investment required in training, with it generally taking 3-to-4 years to master the operation of a hydraulic shovel, and as much as 10 years where larger construction machinery is concerned. In the future, the lack of sufficiently experienced engineers, who have mastered the use of such heavy machinery, will become an increasingly severe problem.

Another reason is the inherent danger that on-site work involves. The machinery used in civil engineering construction is gigantic in scale, which can easily result in serious accidents. There are also many hazardous worksites, such as volcanic regions where pyroclastic flows may occur, and disaster sites where there is a risk of structures collapsing. For these reasons too, we anticipated the demand for remote operation to grow.

“Control of hydraulic shovels is a particularly challenging area.”

Fujita (DeepX): Even compared to other forms of construction machinery, the control of hydraulic shovels is a particularly challenging area. The movement of the machine will change, depending on the state of the oil flow in the hydraulic control mechanisms, while the conditions of the earth itself also vary greatly from case to case. As such, it is necessary to address a wide range of factors in order to achieve stable control. On the other hand, if we are able to successfully automate hydraulic shovels, it should be comparatively simple to apply the resulting technologies to other forms of construction machinery.

Kawakami (Fujita): That’s right. Moreover, we have voluntarily limited development in this project to include only sensor devices that can be installed on the outside of the machinery. This means that when heavy machinery is required following a sudden large-scale disaster, we can quickly install the equipment and achieve automation of any machinery that already happens to be present in the area. AI will be able to step into the role that human operators once played in performing remote operation.

Employing AI in hydraulic shovel posture recognition and control

Toyama (DeepX): I’d like to provide a simple explanation of how the AI for this project was developed. The AI that we have built is used to both recognize the posture of the hydraulic shovel and to control its movement.

For the former, the angle of each joint is estimated based on real-time images of the shovel arm itself. For the latter, operation signals are transmitted, based on the estimated joint angles, in order to realize the intended movement, such as digging. The development of both types of AI requires large amounts of data collection and physical testing at the worksite, and the members of Fujita have cooperated extensively with us in this.

Kawakami (Fujita): We primarily provided support relating to the real-world worksites, such as preparing test sites and machinery, assigning professional operators, and acquiring field data. In the early stages of the project, in particular, we allocated considerable resources to the collection of image data. One method we used involved capturing continuous footage of a hydraulic shovel from synchronized cameras placed in the operator’s seat and outside, adjacent to the machine while performing a variety of operations.

Nishimura (DeepX): And we really have to thank you for that. With the image data we received we were able to conduct manual annotation of the three joint articulations. All up, the annotated data consisted of around 2.5 million frames, with roughly 1 million frames of image data captured in profile view, and another 1.5 million frames from the operator’s seat. That image data was instrumental in training a model to recognize the posture of the hydraulic shovel.

Toyama (DeepX): Once it was possible to estimate the posture of the hydraulic shovel, we proceeded to develop a control system that would decide how the hydraulic shovel should operate. In the early stages of development we attempted to create a control model that learnt from feedback control and the operation data acquired from professional operators. However, we faced the problem of not being able to adapt correctly, depending on whether the ground was harder or softer. As such, we reconsidered our approach and used a simulator to virtually replicate various types of soil conditions. We then conducted reinforcement learning for the control model within the simulator until it was finally possible to perform in all kinds of soil types, then transitioned that back into the real world.

Fushimi (Fujita): We then examined the AI digging that was delivered by DeepX and provided feedback from a worksite perspective. At times, we visited the test site as often as once a week, in order to discuss what wasn’t working and ensure continued improvements. Someone also had to fill in the hole that had already been dug before we could perform the next round of testing, which ended up being primarily my responsibility [laughs].

Nishimura (DeepX): With our approach making use of reinforcement learning that was conducted within a simulator, the ongoing cycle of worksite verification and improvements proved to be quite arduous. There will always be some discrepancies between any simulation and the real world, and we had to devise ways of reducing those discrepancies while examining the control behavior at the test site. The design of the reward given during learning also has a large impact on the control policy performance. A minimum of one day was required in order to learn a single control policy. Our company’s servers were operating at full capacity throughout the constant cycle of on-site verification, improvement, and learning.

Kawakami (Fujita): I think there were considerable difficulties in reproducing the techniques of a skilled operator, however they stuck at it well. Because even an expert operator becomes tired after operating a machine for three or four hours, there is a particularly large value in replacing humans with tireless AI.

Designing suitable intermediate goals and finding the shortest route in surmounting the obstacles

Toyama (DeepX): When we first started this project the request from Fujita was extremely broad: use AI to automate a hydraulic shovel. At that time we had never even touched a hydraulic shovel or anything even resembling one, and knew almost nothing about the actual conditions on a construction site. There were also no precedents in terms of using AI to automate a hydraulic shovel that we could follow. We had absolutely no idea which route would allow us to surmount such obstacles.

Fushimi (Fujita): In the construction industry it is expected that the completed work process will be formulated in detail at the beginning of a project. However, when DeepX presented the project schedule to us it was only a rough outline, with the basic units measured in months, and completely blank beyond the six-month mark. This came as quite a shock to us.

Toyama (DeepX): Yeah, sorry about that [laughs]. Ordinarily, with machine learning, and particularly with the sort of deep learning that has generated large breakthroughs in recent years, full performance cannot be achieved unless a certain amount of data has been accumulated. However in nearly all cases we do not know in advance how much data will be necessary. Combined with the characteristics of AI development, it was nearly impossible to create a more detailed schedule in advance.

With these conditions in mind, the first operation that we set out to automate was horizontal towing, which has the shovel raised. We chose this operation because there is no contact with the earth, and we therefore expected it to be the simplest of the pure control problems.

Nishimura (DeepX): Owing to the fact that most of the sensor device connections and the interface between the backhoe and PC had been completed in advance, I think this intermediate goal was also good in terms of rooting out the bugs in these areas.

Fushimi (Fujita): It took a little under a year to automate this operation, however when it was done DeepX had begun to gain a considerable understanding of heavy machinery mechanisms and characteristics. This also made communication between the two sides easier.

Nishimura (DeepX): By this time we had mostly figured out how we should proceed with the hydraulic shovel AI development. By having a professional operator demonstrate the hydraulic shovel operations, and by actually operating it ourselves, we learned more about the issues that needed to be addressed. Recently we have accumulated even more data and gained more knowledge on heavy machinery, and are able visualize the complete process to some degree.

Fushimi (Fujita): Mr. Nishimura actually got a heavy equipment operator license. I was surprised at how quickly both of them absorbed knowledge.

Toyama (DeepX): We designated an immediate goal and then scaled that mountain. From there, we took stock of the situation around us, designated the next intermediate goal, and started climbing again. We have been repeating this process, over and over, as we carry out the project. However, the knowledge that we require can be vastly different depending on the mountain we choose to face, so we have to be able to quickly absorb new knowledge each time. It is also important that we explain to the people involved why each intermediate goal is the best choice, and gain their full agreement.

Kawakami (Fujita): AI is a hot trend nowadays, however the uses I see are nearly all abstract, for example “identifying cats in photos”. In fact, there are few people who understand, in detail, how AI can be put to use in their own industry.

DeepX, by contrast, explained very well how AI will affect the construction industry and how it can be of benefit. The fact that we have gained some degree of knowledge is one large reward of this project. Recently we’ve grown emboldened enough to make somewhat more difficult requests of them [laughs], such as asking for small adjustments to particular movements.

Actively incorporating knowledge beyond AI in order to solve problems

Kawakami (Fujita): There was something else that made an impression on me during this collaboration: when installing AI onto hardware, it can be very difficult to identify the exact causes of any bugs, right?

Toyama (DeepX): Yes. For example, when the actual behavior of the shovel deviates slightly from the movement we wanted, there are a variety of possible causes, including insufficient or imbalanced image data used during training, or other problems on the AI side, as well as conversion errors when sending signals, machine control failure, or any number of other hardware-side problems.

Fushimi (Fujita): When Mr. Toyama and Mr. Nishimura struggled for a while with a problem and had not found an answer, they would go bouldering or do some other activity to take their minds off of it for a while. After doing so, they frequently came up with the answer the following day.

Nishimura (DeepX): That’s right. Whenever we get stuck on something we try to shift our thinking and gain a perspective on things from an elevated position. We are also aware of the need to be continually absorbing new knowledge. In addition to the latest research in the AI field, we can increase our options by incorporating knowledge from other industries as well.

For example, during this project we encountered an issue in which the estimated joint angle was oscillating even though no signals were being sent to the hydraulic shovel at all. After we analyzed the relationships between the signals that were sent to the machine and the angular velocity, we solved the problem by incorporating these relationships into our estimations of the joint angle.

Kawakami (Fujita): The idea of analyzing the signals rather than focusing on the AI comes from the field of robotics. Even though their area of expertise lies in a different field, these two were able to identify and quickly apply such approaches based on worksite issues. I think it is precisely because they are not limited by the boundaries of their field that DeepX has proven to be capable of producing these kinds of ideas.

An ability to overcome “thorny challenges” based on a deep understanding of both AI technologies and robotics

Toyama (DeepX): We really appreciate that. The project is at a stage where it has successfully automated excavation work on level terrain, and from here on we will conduct repeated tests in more complex worksite environments as we strive to improve accuracy, so that the machine can eventually be used in all kinds of terrain.

Fushimi (Fujita): That’s right. We have come to the point where we can reproduce the remote operation techniques of a professional as long as the ground is flat. So we expect that the AI is certain to achieve superiority in performance sometime in the near future.

This project has brought about a variety of changes in us as well. The largest of these is that we have increased our understanding of AI. The long-term goal of this project, for Fujita, is to determine the optimal means of utilizing AI. If we can deepen the knowledge of AI among our employees through this project, then we should be able to adapt to the rapid changes in the times. I intend to reflect on how I can make use of the knowledge that this project has given me.

Kawakami (Fujita): Recently there are many companies that are getting involved in AI, but the majority of them are merely using software applications, such as chatbots. At the same time, while existing industrial machinery manufacturers are showing interest in AI, I feel that many of their designs are too conservative and do not go beyond the limitations of conventional robotics.

It is because DeepX has an understanding of both cutting-edge AI technologies and robotics that it is able to work to combine them. I think the reason that this project has been successful is that DeepX continues to accept thorny challenges that extend beyond the boundaries of their industry.

Profiles

Joji Toyama
Chief Development Officer, DeepX, Inc.
While undertaking a master’s degree program at the University of Tokyo Graduate School, Mr. Toyama received the Dean’s Award in the Faculty of Engineering, before joining DeepX after his graduation. During his time at DeepX, he has conducted research at the University of Tokyo Matsuo Laboratory covering a broad range of fields, from deep learning engineering logic through to applications for real-world problem solving. At DeepX he is in charge of project management, in addition to individual software development projects, and also undertakes fieldwork, including weekly visits to the Fujita outdoor machinery test site.

Kohei Nishimura
Engineer, DeepX, Inc.
Mr. Nishimura joined DeepX after graduating from the Faculty of Engineering at the University of Tokyo. In addition to development of deep learning programs, he also develops GPU machines for actual machine control and performs selection and design work regarding cameras and sensors. He also engages in a wide range of activities relating to data management, including development of the annotation tools necessary for the creation of deep learning teaching data and management of other annotation personnel.

Katsuhiko Kawakami
Senior chief consultant, Machinery Division, Civil Engineering Center, Fujita Corporation
Mr. Kawakami joined Fujita in 1987, where he is in charge of the planning and design of construction equipment, as well as the development and on-site implementation of new technologies required for worksites. In recent years he has also worked on developing disaster-response technologies and carried out work at actual disaster sites.

Hikaru Fushimi
Chief Researcher, Information Technology Solutions for Development Department, Technology Development Division, Fujita Corporation
After working at an IT company, Mr. Fushimi joined Fujita in 2016. He works in R&D concerning utilizing VR/AR and other cutting-edge ICT technologies, in order to improve productivity at construction sites. As part of this, he acquired a heavy machinery license (for vehicle type construction machines) during his participation in this collaborative project, and is also responsible for ground leveling and other hands-on heavy machinery operations during the tests.