Life is filled with unexpected turns and science is no exception. The same goes for my research field of artificial intelligence—such as algorithms that measure vehicle speed with CCTV data or detect a robot’s motions through a camera. While these topics do not seem to have much in common, all of this has been made possible through rapid advances in deep learning. In the past, it took a considerable amount of time for research that could address a problem in one field to expand to another. Nothing could be further from the truth these days. Current AI research seems to be full of undiscovered breakthroughs, one of which is ’translation.’ But don’t understand this as ‘the translation of languages,’ as you would normally interpret. It instead refers to the astonishing paradigm shift of ‘translating the world.’
Machine Translation Resembles Living Organisms
To help understand what translating the world could mean, let’s take a look at the history of machine translation. Solving machine translation was one of the first tasks AI researchers tried to tackle when the first generation of computers appeared almost 70 years ago. Despite being initially confident, it was not that long before researchers had to admit that their work had been largely unsuccessful.
Looking back now, the reason for this failure was quite simple—what’s easy for humans is sometimes extremely challenging for machines. In the case of translation, understanding the intent of the speaker is critical and it was difficult to incorporate this in the AI algorithms at the time. Another unexpected problem was the length of the sentences. There’s nothing strange about the change in length of a sentence when translated because each language has its own unique characteristics. However, in the world of AI, this change is far from natural and AI has notorious problems in dealing with input and output of variable lengths. Relatively simple issues like this were major obstacles and remained so up until the advent of deep learning, the superstar of AI research.
Deep learning has changed everything. The availability of GPUs, which provide tremendous computing power compared to traditional CPUs, and breakthroughs in deep neural network techniques have revolutionized every computer-related research field in the last few years including machine translation. Among the many innovations, the recently introduced “Transformer” architecture has achieved overwhelming success and could be the key to translating the world. If I had to describe the novelty of Transformer models in one word, it would be “attention.” Transformers are embedded with an attention mechanism, which allows computers to identify the relationship between certain pieces of information and pay more attention to them.
Being able to focus on a particular element comes rather naturally to living organisms. We may be under the impression that we consistently observe and receive information from the world, whereas we actually only pay attention to a few select pieces of information to quickly grasp the context. Think about how we listen to a story. We don’t interpret the meaning of each and every word. Perhaps we deliberately ignore what’s not so important to be able to dedicate all our attention to what we consider is. Likewise, the Transformer model allows the AI agent to simulate what we do by isolating some parts of the signals in order to focus more on them.
The results were incredible. Focusing attention on specific pieces of information has become the new paradigm in computational linguistics. Consider the following sentence: “The little girl always carries her doll with her.” The attention mechanism in the Transformer model associates the words “girl” and “her” denoting them as the same concept, regardless of their distance in the sentence. This may not seem like much, but when trained on large datasets, the performance is so stunning that it’s now reached a level often indistinguishable from human translation.
Broadening the Scope of Translation to the World
Based on the success of the Transformer architecture, researchers raised a rather sensible question: If Transformers can solve a task as difficult as machine translation, could they accomplish other tasks just as well? In fact, this question led researchers to think again about the fundamental workings of AI. They began to integrate the attention mechanism into other existing neural architectures and the number of scientific papers inspired by the Transformer architecture grew exponentially. Not only did this integration go beyond machine translation to the broader field of natural language processing, it was also being applied to other AI research fields such as computer vision.
Having shown that Transformers are superior to other models in terms of their ability to generalize, and considering they were originally introduced for machine translation, why not then express each and every other kind of problem as a ‘generalized translation’ problem?
In the case of language, we need to apply pre-processing for a computer algorithm to understand it. In particular, input sentences are broken down into smaller pieces in a process called tokenization. Without tokenization, it would be impossible for the computer to comprehend sentences in deep learning. The single unit of word or morpheme is called a token, which we use to represent semantic concepts or mathematical quantities. Researchers have found that, by using Transformers and expressing tasks as generalized translation tasks consisting of several tokens, we can solve all sorts of problems.
For example, object detection in images can be expressed and solved as a translation task. It’s also possible to do the opposite, whereby an image can be generated based on a sentence describing the image. Similarly, a research group is currently working on an algorithm that enables robots to play video games, using sensor information as input. The visual information received through the sensors (e.g. cameras) is converted into a sentence consisting of tokens, then translated to the output which is a sentence explaining an action the robot should take. This process can be repeated, allowing the robot AI to successfully play video games.
Another example from my own research is about building a visual radar for traffic using Transformers. The goal here was to measure the speed of vehicles observed from CCTV cameras. Traffic speed is usually measured using dedicated sensors like roadside radars or inductive loops, but this equipment is quite costly. I wondered if it would be possible to use cheaper CCTV cameras to measure speed or even improve existing smart transportation systems by detecting accidents or so on. The answer to both of these questions was “yes.” The key idea was to, once again, turn the visual radar task into some kind of generalized translation task expressed in tokens and solve this with Transformer models.
Let me elaborate further. To measure vehicle speed, we first need to determine the input and output sentences. For the input sentence, tokens are taken from a dictionary of car shapes. For the output sentence, we use a dictionary of “scale tokens”, or “ruler tokens” if you prefer. So, in summary, the key idea is to translate car shapes from the CCTV footage into scales, and from the scales, we measure distances on the road. Since speed is just distance divided by time, knowing the road scale means we know the speed of the car.
Of course, this is not quite as simple as it seems. In reality, our Transformer has to be trained appropriately to give correct translations. The basic procedure for training is similar to that of living organisms. We present examples of input sentences with the corresponding output sentences that it has to translate. Once the Transformer has seen enough of these input/output sentence pairs, it can do its job properly. In our case, we generated a virtual road scene and trained the network on a 3D simulation of traffic, resulting in faster and more accurate translation.
Our results were surprising. The AI radar for CCTV videos we had developed proved to be as accurate as applying complex methods on high quality footage from expensive sensors. It was also much more robust with 4 times less median speed error and at least 1,000 times faster overall! AI had done it again.
A Creative Approach to How AI Understands the World
Progress is often made in unanticipated ways with the solution to one problem becoming the key to solving many others. At the core of the Transformer architecture, the attention mechanism has allowed AI to perform extremely complex tasks such as abstraction and generalization. This is not unlike how mankind has brought countless problems into the realm of language, enabling metaphysical thought and problem solving.
The idea of expanding “language to language” translation to the translation of “everything to everything” is incredibly powerful. We can target virtually any task or previously unsolved problem. From computer vision for everything we see and touch, to voice or sound for everything we hear, the potential is endless. By interpreting the essence of translation as conveying a meaning so that it can be understood, researchers are now translating on so many fronts that AI will ultimately understand the world.
▶︎Jérome Revaud, a computer vision researcher at NAVER LABS Europe, located in Grenoble, France, is currently leading the Geometric Deep Learning team to enable AI to understand the world better based on massive amounts of images and videos.
▶︎ Hyung Taek Yoon, the creator of Forward Thinking series’ illustrations, is an illustrator and spatial storyteller. He has designed concepts and illustrations of various spatial projects, with works such as ‘Space Begins from Stories.’
▶︎Forward Thinking is a recurring online publication focusing on today’s major technological trends, such as AI, robotics, autonomous driving, and metaverses, containing stories of outstanding researchers collaborating with NAVER LABS. www.naverlabs.com/forwardthinking