User fits 28.9M LLM on ESP32-S3, a feat anyone can replicate
| Source: Mastodon | Original article
A 28.9M large language model has been successfully run on an ESP32-S3.
A significant breakthrough has been achieved in the field of edge AI, as someone has successfully managed to run a 28.9M Large Language Model (LLM) on an ESP32-S3 microcontroller. This feat is impressive given the limited resources of the chip. The LLM can generate text, albeit at a relatively slow pace of approximately 9 tokens per second, using only about 560KB of RAM.
This development matters because it demonstrates the potential for memory-efficient edge AI implementation, enabling the deployment of AI models on compact devices with limited resources. The ability to run LLMs on microcontrollers like the ESP32-S3 opens up new possibilities for applications such as story generation, text analysis, and more.
As this technology continues to evolve, it will be interesting to watch how developers leverage this capability to create innovative applications. With the codebase and project details available on GitHub, others can now replicate and build upon this achievement, potentially leading to further advancements in edge AI and the development of more sophisticated models that can run on low-power devices.
Sources
Back to AIPULSEN