TFLM vs Edge Impulse

The goal of this study is to compare the performance of TensorFlow Lite for Microcontrollers (TFLM) and Edge Impulse.

According to the Edge Impulse documentation, an enterprise account is needed to run RNN models. As a result, this study focuses on FC, CNN, and TinyMLPerf models.

Model Type: Any FC CNN TinyMLPerf     Board: Any NUCLEO-L4R5ZI RenesasRX65N

Models

FC parameters and MACs

CNN parameters and MACs

TinyMLPerf parameters and MACs

Error

FC - NUCLEO-L4R5ZI

FC - RenesasRX65N

CNN - NUCLEO-L4R5ZI

CNN - RenesasRX65N

TinyMLPerf - NUCLEO-L4R5ZI

TinyMLPerf - RenesasRX65N

Execution Time

FC - NUCLEO-L4R5ZI

FC - RenesasRX65N

CNN - NUCLEO-L4R5ZI

CNN - RenesasRX65N

TinyMLPerf - NUCLEO-L4R5ZI

TinyMLPerf - RenesasRX65N

Flash Size

FC - NUCLEO-L4R5ZI

FC - RenesasRX65N

CNN - NUCLEO-L4R5ZI

CNN - RenesasRX65N

TinyMLPerf - NUCLEO-L4R5ZI

TinyMLPerf - RenesasRX65N

RAM Usage

FC - NUCLEO-L4R5ZI

FC - RenesasRX65N

CNN - NUCLEO-L4R5ZI

CNN - RenesasRX65N

TinyMLPerf - NUCLEO-L4R5ZI

TinyMLPerf - RenesasRX65N

Summary

• Model Correctness:

  • Since TinyMLPerf_MBNet was too large for all tests except its int8 only version with TFLM, we have excluded it from the comparison.

  • Some models failed to run on the RenesasRX65N board. (1)

    1. In all cases, the program halts for an unknown reason.

  • The error rates of the remaining models are acceptable. (1)

    1. basic is perfect, Edge Impulse is better than TFLM in int8 only models.

• RenesasRX65N:

  • On the Renesas board, TFLM outperforms Edge Impulse in terms of execution time, flash size, and RAM usage (1). As a result, TFLM is recommended for this board.

    1. One contributing factor could be that the GCC compiler had link-time optimization (-flto) enabled for TFLM, but this was not possible for Edge Impulse (Edge Impulse encountered errors with -flto). However, this cannot be the only reason, because in a few tested scenarios where we have turned off -flto for TFLM, the results were still better than Edge Impulse.

• NUCLEO-L4R5ZI:

  • Execution Time: TFLM performed better for FC models, particularly for smaller models. For other models, the performance of both is similar, with a slight edge for TFLM.

  • Flash Size: Edge Impulse is better than TFLM.

  • RAM Usage: For small FC models, Edge Impulse is slightly better. For the others, the two are almost the same.

• Conclusion:

  • For the RenesasRX65N board, TFLM is the preferable choice

  • For the NUCLEO-L4R5ZI board, TFLM is recommended if execution time is the priority. However, if flash size and RAM usage are more critical, Edge Impulse may be the better option.