ACCELERATION OF TRANSFORMER ARCHITECTURES ON JETSON XAVIER USING TENSORRT

Abstract

Transformer models have become a key component in many natural language processing and computer vision tasks. However, these models are often computationally intensive and require a lot of resources to run efficiently. To address this challenge, this study studies the use of TensorRT, an optimization library provided by NVIDIA, to accelerate the inference speed of transformer models on Jetson Xavier NX, a low-power and high-performance embedded platform.

This research demonstrates the significant impact of TensorRT optimization on transformer models. Specifically, we present two case studies: one involving a Transformer model for text-to-speech synthesis and another featuring a Vision Transformer model for image classification. In both cases, TensorRT optimization leads to substantial improvements in inference speed, making these models highly efficient for edge device deployment. For the text-to-speech task, TensorRT optimization results in a remarkable 60% reduction in inference time while decreasing memory usage by 17%. Similarly, for image classification, the Vision Transformer model experiences over a 60% increase in inference speed with a negligible 0.1% decrease in accuracy. This study not only showcases the practical benefits of TensorRT but also highlights the potential for further optimization and deployment of transformer models on edge platforms. This demonstrates the potential of TensorRT to optimize transformer models, both in terms of performance and memory usage. This could have far-reaching implications for edge computing, allowing more applications to be deployed on low-power devices.