Testing LLM and MLX Runtimes on My Pro 24 GB M5 Reveals Early Findings
gemma llama
| Source: Mastodon | Original article
Testing underway for LLM runtimes on M5 Pro. Performance comparisons are being made.
A user is currently testing several LLM and MLX runtimes and apps on their M5 Pro 24 GB device. The goal is to determine which one achieves the best performance with a local LLM, specifically Gemma 4 12B. So far, the user has tested Ollama, LMStudio, oMlx, and osaurus, each with its own unique tweaks.
This matters because the performance of LLMs on local devices is a crucial aspect of their usability and effectiveness. As the field of AI continues to evolve, optimizing the performance of these models on various devices is essential for widespread adoption. The results of this testing could provide valuable insights for developers and users alike.
What to watch next is how the testing unfolds and which runtime or app ultimately emerges as the top performer. This could have implications for the development of future LLMs and MLX runtimes, as well as inform user decisions about which tools to use for their AI needs. As we continue to monitor the situation, we will provide updates on any significant findings or developments.
Sources
Back to AIPULSEN