There was a time when semiconductor technology developed in such a way that the number of transistors doubled every year. This is known as Moore’s Law. This empirical law was readjusted in 1975, when the number of transistors in our microprocessors doubled every two years or so. After that, it held true for many years, until it became obsolete today. Since then, NVIDIA has established Huang’s Law by observing performance doubling every two years!
Huang’s Law: performance doubled every two years, x1000 in 10 years!
So, to promote this concept, the company doesn’t hesitate to highlight the leap made by their hardware over the last ten years. In 2012, for example, we found the K20X boasting Int8 performance of 3.94 TOPS. Then, with the M40, performance reached 6.85 TOPS, rising to 21.20 TOPS with the P100. With the V100, performance reached 125 TOPS, then 261 with the Q800, 1248 with the A100 in 2020 and 4000 TOPS with the H100!
Of course, this is an empirical law: based on experience and observation. Moreover, if we want to tease a little, we could say that between the K20X and the M40, performance isn’t quite x2, but that would be nitpicking.
All this to say that the company is not looking to reduce the engraving finesse or increase the number of transistors at all costs. No, performance gains are to be found in optimizing the various architectures. Nevertheless, let’s not overlook the role of etching either, since it allows us to fit more transistors onto a given surface.
Now, we’re curious to see how long this law will hold true, and when a new trend can be established.
