Neural networks on steroids
Researchers have created a new circuit that could boost the performance of training neural networks.
Machine Learning twice as hungry every 100 days
According to an analysis by OpenAI last year, the computing resources used in the largest Machine Learning training runs increased on average tenfold per year in 2013-2017. That’s about five times faster than Moore’s Law.
As an example, in 2017 DeepMind’s AlphaGo Zero required 300,000 times the computing resources used by AlexNET, the then much admired neural network that competed in the ImageNet Large Scale Visual Recognition Challenge 2012.
As the OpenAI researchers rightly noted, three factors drive the advancement of AI: algorithmic innovation, the quantity and quality of data, and the amount of compute available for training.
Is this third factor being thought about enough? Turns out it is, and researchers have recently obtained a promising result.
Enter the analog computer
On last February a team has demonstrated the feasibility of analog, in-memory computing circuitry that could be employed to solve certain classes of problems in one single step, with no need to decompose the problem at hand in a routine of binary operations like those carried out today by digital computers.
And guess what? This new circuit’s preferred types of problems are those that can be modeled by linear-algebra equations, like multiplying and manipulating matrices. Therefore, this new analog computer could be used to boost gaming consoles or train neural networks, that is machine learning systems.
In fact, the authors project it (when complemented with CMOS-based digital circuits) to be more effective, or convenient, than quantum computers, which require special environmental conditions.
The result is made possibile by in-memory computing and by directly mapping the variables and the parameters of an algebraic model (such as the multiplication of a matrix and a vector) to physical aspects of an eletrical circuit, like voltage and current.
This means building an analog, instead of a digital, computer.
An experimental proof of concept
After falling in the Eighties, the idea of mixing analog and digital computing architectures has had a revamp about a decade ago, when it has become apparent that such a computing hybrid could be ideal for tackling application domains like gaming or machine learning.
Analog computers, like these that model the terms of an abstract problem with voltages and currents in a (complex) electric circuit, are superfast; digital computers are more versatile and accurate. When one needs twice as much computing resources every hundred days, hybridization may be considered, mixing the best of both worlds.
The circuit built by the authors has been shown capable of multiplying a vector by a squared matrix, to invert a matrix and to calculate eigenvectors, all in one cycle: they even built a small quantum mechanics showcase experiment, and the circuit solved Schroedinger’s equation in one step.
As the paper clearly points out, scalability is currently limited by electronic noise, by the yet insufficient precision of the memory’s electric resistance, and by the fact that the circuit needs to be reprogrammed (changing matrix cofficients) before every run. And integration with digital CMOS circuitry will not be trivial.
Furthermore, the algebraic problems tackled in the lab were somewhat simplified versions of real world ones.
We can say that we have a new drug that works well on guinea pigs. Now investors, manufacturers and researchers will try to take it on a human scale. I think it may be worth it.
This post reflects my opinion. No one paid for this post.