Quantization for performance
The following is intended for informational purposes only, and may not be incorporated into any contract. No purchasing decisions should be made based on the following materials. Unity is not committing to deliver any functionality, features or code. The development, timing and release of all products, functionality and features are at the sole discretion of Unity, and are subject to change.
If you have accepted to use functional cookies and logged in using your Unity ID at the top right of the page, then sharing feedback is as simple as clicking a card below, selecting a topic’s importance, adding your point of view, and submitting. If you prefer not to accept functional cookies or log in, you will be prompted to enter an email address and validate it, so we know how to reach out when the topic evolves. For more information read the Feedback and privacy terms.
What? Quantization converts AI model weights (the numbers that take up most of the model size) from relatively large and high-precision numbers (FP 32) to relatively small and low-precision numbers (FP16 or INT8) for performance (inference speed) improvements.
Why? AI models can be cumbersome to run within limited RT3D/game processor budgets, yet inference speed is still very important... it’s why you are running your model locally in the first place. Quantizing can increase the performance dramatically, and we’ll provide some benchmarks once this can be validated across our model test suite. The downside of quantization is quality loss. Calculating quality loss is tricky and subjective, so it’s best to test your results thoroughly to see if the degradation is acceptable. A good rule of thumb is that larger models degrade less with quantization.