A mathematical approach towards quantization of floating point weights in low power neural networks

Neural networks are both compute and memory intensive, and consume significant power while inferencing. Bit reduction of weights is one of the key techniques used to make them power and area efficient without degrading performance. In this paper, we show that inferencing accuracy changes insignificantly even when floating-point weights are represented using 10-bits (lower for certain other neural networks), instead of 32-bits. We have considered a set of 8 neural networks. Further, we propose a mathematical formula for finding the optimum number of bits required to represent the exponent of floating point weights, below which the accuracy drops drastically. We also show that mantissa is highly dependent on the number of layers of a neural network and propose a mathematical proof for the same. Our simulation results show that bit reduction gives better throughput, power efficiency, and area efficiency as compared to those of the models with full precision weights.