As neural networks and machine learning continue to take on new challenges from analyzing images posted on social media to driving cars, there’s increasing interest in creating hardware that are tailored for AI computation.
The limits of current computer hardware has triggered a quasi-arms race, enlisting a growing array of small and large tech companies that want to create specialized hardware for artificial intelligence computation. But one startup aims to create the next breakthrough in AI acceleration by changing the most fundamental technology that has been powering computers in the past several decades.
A team of scientists at Boston-based Lightelligence are ditching electronics for photonic computing to achieve orders-of-magnitude improvement in the speed, latency and power consumption of computing AI models. Photonic or optical computing, the science of using laser and light to store, transfer and process information, has been around for decades. But until now, it has been mostly limited to optical fiber cables used in networking.
The folks at Lightelligence believe that optical computing will solve AI’s current hardware hurdles. And they have an optical AI chip to prove it.
The limits of electronic AI hardware
Artificial intelligence is one of the fastest growing sectors of the tech industry. After undergoing several “winters,” AI has now become the bread and butter of many of applications in the digital and physical world. The AI industry is projected to be worth more than $150 billion by 2024.
“There has been a huge explosion of artificial intelligence innovation in the past five years,” says Dr. Yichen Shen, co-founder and CEO of Lightelligence. “And what we think will happen in the next ten years is that there will be an explosion of use cases and application scenarios for machine learning and artificial neural networks.”
Deep learning and neural networks, the current dominating subset of AI, relies on analyzing large sets of data and performing expensive computations at fast speeds. But current hardware structures are struggling to keep up with the growing demands of this expanding sector of the AI industry. Chips and processors aren’t getting faster at the same pace that AI models and algorithms are progressing.
Lightelligence is one of many companies developing AI accelerator hardware. But as Shen says, other companies working in the field are basing their work on electronics, which is bound by Moore’s Law.
“This means their performance still relies on Boolean algebra transistors to do AI computation. We think that in the long run, it will still be bound by Moore’s Law, and it’s not the best of solutions,” Shen says.
Established by Intel co-founder Gordon Moore, Moore’s Law maintains that technological advances will continuously enable us to reduce the size and price of transistors, the main component of computing chips, at every 1.5-2 years. This basically means that you can pack more computing power in the same space at a lower price. This is the principle that has making phones and laptops stronger and faster in the past few decades.
But Moore’s Law is hitting a wall. “Now it takes five years to get to a smaller node and down the road it can take longer—ten or twenty years—to get to three nanometers or smaller nanometers,” Shen says. “We want to change the way we do computing by replacing electronics by photonics. In principle we can do the calculations much faster and in a much more power efficient way.”
The optical AI accelarator
In 2017, Shen, then a PhD student doing research on nano-photonics and artificial intelligence under MIT professor Marin Soljacic, co-authored a paper that introduced the concept neural networks that ran fully on optical computing hardware. The proposition promised to enhance the speed of AI models.
A few months later, Shen founded Lightelligence with the help of Soljacic. The prototype of the optical AI accelerator, which Lightelligence released earlier this year, is the size of a printed circuit board (PCB).
“Instead of using digital electronics, we use optical signals to do AI computation. Our main purpose is to accelerate AI computing by orders of magnitude in terms of latency, throughput and power efficiency,” Shen says.
The company has designed the device to be compatible with current hardware and AI software. The accelerator can be installed on servers and devices that support the PCI-e interface and supports popular AI software frameworks, including Google’s TensorFlow and Facebook’s PyTorch.
There are still no plans to create fully fledged optical computers, but the technology is certainly suitable for specific types of computation. “One of the algorithms that photonics is very good at implementing is matrix multiplication,” says Soljacic.
Matrix multiplication is one of the key calculations involved in neural networks, and being able to speed it up will help create faster AI models. According to Shen, the optical AI accelerator can perform any matrix multiplication, regardless of the size, in one CPU clock, while electronic chips take at least a few hundred clocks to perform the same operation.
“With neural networks, depending on the algorithm, there might be other components and operations involved. It depends on the AI algorithm and application, but we’ll be able to improve the performance from one to two orders of magnitude, ten to hundred times faster,” Shen says.
The company tested the optical AI accelerator on MNIST, a dataset of handwritten digits used to benchmark the performance of machine learning algorithms. The hardware performed much faster than other state-of-the art AI accelerator chips.
Soljacic explains that the proposition of optical neural networks was set forth decades ago, but at the time it didn’t get traction. However, the past years have seen two important changes.
“First, neural networks became hugely important. That provides a large motivation to people to develop specialized hardware for neural networks which didn’t exist thirty years ago. And more importantly now finally we have the same fabrication that made electronics manufacturing so successful, the CMOS processing, also available for photonics,” Soljacic says.
This means that you can integrate thousands of optical devices on the same chip at the same cost that it would take to integrate electronic devices. “That is something we didn’t have five to ten years ago, and that is what is enabling the technology for all of this, the ability to mass produce at a very low cost,” Soljacic says.
Manufacturing optical chips is also less expensive than electronic devices, Shen adds. “For photonics, you don’t need a 7nm or a 3nm node to do it. We can use older and cheaper nodes to manufacture it,” he says.
Solving the latency problem of AI hardware
Why is it important to improve the speed of AI calculations? In many settings, neural networks must perform their tasks in a time-critical fashion. This is especially true at the edge, where AI models must respond to real-time changes to their environment.
One of the use cases where low-latency AI computation is critical is self-driving cars. Autonomous vehicles rely on neural networks to make sense of their environment, detect objects, find their way on roads and streets and avoid collisions.
“We humans can drive a car only using only our eyes as guide, purely vision-based, and we can drive easily at 90-100 mph. At this point for autonomous vehicles, they’re not able to drive the car at that speed if they only rely on cameras. One of the main reasons is that the AI models that process video information are not as fast as needed,” Shen says. “Since we can decrease the time it takes to process data and run AI computations on the videos, we think we’ll be able to allow the cars to drive at a much faster speed. Potentially, we’ll be able to catch up with human performance.”
To be clear, it takes more than super-fast neural networks to solve the challenges of self-driving cars in open environments. Namely, unlike human drivers, neural networks are very limited in their ability to deal with situations they haven’t been trained for. But being able to improve the speed of processing neural networks will certainly put the industry at a better position than it currently is.
There are plenty of other areas where low-latency neural network computations can become a game changer, especially where AI models are deployed in the physical world. Examples include drone operations and robotic surgery, both of which involve safety issues that need real-time response from the AI models that control the hardware.
“Low latency AI is critical at the edge, but also has applications in the cloud. Our earlier models are more likely to be deployed in the cloud, not only hyperscale data centers, but also the local data centers or local server rooms in buildings,” Shen says.
Servers that can run AI models with super-low latency will become more prominent as our network infrastructure evolves. Namely, the development of 5G networks will pave the way for plenty of real-time AI applications that perform their tasks at the edge but run their neural network computations in local servers. Augmented reality apps, transportation and medicine are some of the areas that stand to gain from super-fast AI servers.
Making AI calculations power-efficient
Current AI technologies are very electricity-hungry, a problem that is manifesting itself both in the cloud and at the edge. Cloud servers and data centers currently account for around 2 percent of power consumption in the U.S. According to some forecasts, data centers will consume one fifth of the world’s electricity by 2025.
A substantial part of the cloud’s power goes into neural network computation, giving the AI industry an environmental problem.
“People already upload billions of photos per day to the cloud, where AI algorithms scan these photos for pornography, violence and similar content. Today, it’s one billion photos. Tomorrow it’s going to be one billion movies. That part of the energy consumption and the cost of systems running on artificial intelligence is going to be growing very rapidly, and that’s what we’re going to help with,” Soljacic says.
Switching from electronic to optical hardware can reduce the energy consumption of AI models considerably. “The reason that electronic chips generate heat is that the electronic signals go through copper wires and cause heat loss. That’s where the major power costs are. In contrast, light doesn’t heat up things like electronics do,” Shen says.
“In addition to the heat loss through the interconnect, just thinking about traditional electronic circuits, a portion of the power consumption is just leakage. It produces no work other than heat. That could easily be a third of a chip’s power budget. There’s really no benefit from that. The photonic circuit has none of that,” says Maurice Steinman, VP of engineering at Lightelligence.
At the edge, optical AI computing can help take some of the burden off devices where weight is a constraint.
“For instance, if you want to have substantial artificial intelligence to a drone, then you’ll have to add a GPU that consumes 1 kWh and requires a huge and heavy battery,” Soljacic says.
“Right now, the power costs of the AI chips account for about 20 percent of the electricity consumption in self-driving cars. This increases the size of the batteries, which in increases the power consumption for the cars,” Shen adds.
Using optical AI accelerators will help reduce the power consumption and the weight of these devices.
The future of AI hardware accelerators
Optical computing is not the only technique that can possibly address the hardware painpoints of current AI models. Other technologies that might help improve the speed and efficiency of AI models are quantum computing and neuromorphic chips.
Quantum computers are still years away, but once they become a reality, they will change not only the AI industry but many other aspect of digital life, including finance and cybersecurity.
As for neuromorphic chips, they are computing devices that try to imitate the structure of the brain to specialize for AI tasks. Neuromorphic chips are slowly gaining traction.
It will be interesting to see which one of these trends will manage to clinch the main spot for the future of AI computing.