This is a continuation of my previous post Using SensiML with Arduino Nano 33 BLE Sense - Model Development .
I had suggested in my previous post that I thought the poor performance of my model was due to trying to classify multiple items within a single data sample which might be sensitive to the size and movement of the classification window across the data sample.
Here are is the Confusion Matrix from those tests:
It seemed like a simple test would be to produce test data samples that only contained a single class within each sample.
I captured two 5 second data samples of each class and retested the model. The results really surprised me. The overall accuracy improved, but now the LeftRight class has issues.
After a quick look at the data, I realized what is probably causing the problem. When I generated the training data, I did a poor job of isolating the 3 movement axes. My intent was to have each class represent movement along a single axis.
Here's a representative training sample that captures sequentially Z(UpDown), Y(LeftRight), and X(ForwardBack):
You can see that I didn't do a great job of isolating the axes. The LeftRight is the worst. I think that I had the board slightly rotated around the X axis and picked up movement on the Z axis. I was pretty consistent yesterday, so I think the model trained with consistently "bad" data.
I was a little more careful when I took the test data today and that seems to have caused a data mismatch. This is probably just a case of having insufficient training data. I should have data that has "some" skew in the other axes for training. The model has probably overfit the data.
I'm pretty happy in general that things tend to work, I'm sure I'll get a lot better at this over time. And investing more time to generate a more robust training data set will help when there is a real problem to solve 
.
I'm going to move to the next step of deploying the model as it should do a reasonable job of classifying.