AI workloads feature matrix multiplications in a variety of number formats, such as INT8, FP4, FP8, FP16 and BF16. Lower precision entails higher levels of acceleration. While memory bandwidth savings are immediately realised, scaling the arithmetic performance remains a challenge at lower bitwidths.
Imagination presents a novel approach to scaling arithmetic performance across number formats. The multiprecision dot product unit featured in the new E-Series GPUs accelerates matrix multiplications in all the above number formats. Accuracy is maintained and a third of the circuit area is saved compared to traditional design approaches using separate pipelines for each number format.
So how does it work?
The basic operation required to implement matrix multiplication is an accumulating dot product, of the form a0*b0 + … + a15*b15 + c = d, where a, b vectors are in lower precision and c, d are in higher precision such as FP16 or FP32. The number of terms is architecture-specific, with higher number of terms often used to reach higher performance densities. The operation is fused, such that it skips some intermediate normalisation and rounding steps. By specialising the arithmetic to a given number format combination and depth of operation, higher efficiency and accuracy can be reached.
The explosion of number formats is one of the modern trends in floating-point and challenges this approach. It would lead to matrix multiplication accelerators requiring an ever-growing list of dot product hardware operators, increasing the circuit area beyond the available silicon die.
Format conversions may be used to share the same path for numbers of the same width, such as FP16 and BF16. However, this approach does not enable the performance to scale at lower bitwidths.
The need to support multiple precisions at different rates is not new. It was previously solved by SIMD vector unit implementations. In this context, a typical operation is the fused multiply-add and a 256-bits wide datapath may compute four FP64, eight FP32, or sixteen FP16 such operations in parallel. The microarchitecture of a vector unit exhibits a high level of logic reuse.
Vector units are however not well suited to mixed-precision machine learning applications, where the operations must be made more accurate by using a wider accumulation format. They also lack the performance density attainable by a fused dot product over a single format combination.
We built on existing techniques to target matrix multiplications and low-precision number formats. The result is a multiprecision, fused dot product unit with a vectorised datapath. It works by repurposing the multiplication and summation hardware present in an INT8 dot product to implement an efficient FP8 dot product. This involves creating additional hardware for processing the exponents and aligning the partial products, a problem specific to floating-point. The exponent processing, alignment, multiplication and summation hardware is further reused to implement accurate FP16 and BF16 dot products.
Figure 1. Dot product microarchitecture; processing stages with a dashed outline are for floating-point only and the low-precision hardware is reused to operate over larger widths
Input FP8, BF16 and FP16 numbers are converted to a set of internal, normalised number formats. These new formats simplify the arithmetic implementation and ensure the precision remains constant in the presence of subnormal inputs. Products of FP8 numbers are aligned over 16 bits of precision and treated as INT8 products, while products of FP16 or BF16 are aligned over 32 bits. The products are split into two halves and summed using a shared adder structure. Dot products are then accumulated in FP32, FP16 or INT32 precision.
The resulting implementation is more accurate than standard methods on average and its worst-case rounding error is bounded. The empirical accuracy was measured by simulation and compares favourably against reference models implementing floating-point summation in the working FP32 or FP16 accumulation precision. The theoretical accuracy can be analysed using backward error analysis, which confirms rounding errors in arithmetic are small relative to expected quantisation errors.
Vectorisation enabled us to save almost a third of the circuit area, compared to a traditional design using separate pipelines for each number format to accelerate. The implementation has been formally verified end-to-end. A key element of the verification strategy was the availability of a clear, unambiguous numerical specification. This provided a platform for us to continuously optimise the design, making extensive use of vectorisation techniques to maximize logic reuse.
This work was presented at the International Symposium on Computer Arithmetic (ARITH 2026) where the numerical analysis of AI hardware and the standardisation of number formats like FP8 was a hot topic. This research contributed to the conversation on numerical aspects, and it is our hope that it will help accelerate the convergence of research models and hardware implementations towards new industry standards. For more details, please read the full paper in the proceeding or contact our sales team to discuss our latest innovations in the field of edge AI.
Figure 2. Imagination Technologies presenting the dot product microarchitecture at ARITH 2026