When I first began learning about "string tokenisation" in C many years ago, I enjoyed optimising them, and character arrays, and strings, and how C++ handled it all a little differently but the same.
I find it fascinating that we're still experiencing "issues" with tokenising now and it's relevant to AI.
With "ai service" models moving to "token charging" it's crazy how AI works with this. It's untenable.
