GPT4ALL + RP4 = Dead End (for now)
I've tried different LLM models with GPT4ALL and so far I was not able to find one, which can process inference on RPi 4B fast enough. I've described some of these tests in my previous blog post. I was close to giving up on my project or put it on hold until new advancements in this fast-growing area of AI..
Focus on STEM
cgptonline shared his excitement about my project and its potential for STEM in his comments for my first blog for this project. I've started looking for smaller and specialized models, that can fit in RPi 4B $GB RAM.
TinyStories LLM
One such model is TinyStories. It was trained on " a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand".
I've installed it and run a few tests and it was amazingly fast. It was generating text at a speed that was close to the speed of reading.
It was able to generate a story pretty fast. It can be used by kids in school to get started with LLM in a safe and controlled environment. And it can be achieved at a low cost on RPi 4B 4GB device.
I'll explore it a bit more and will share the code and test results in the next post.