Typical Road Test Plan for an SBC like Raspberry Pi4

Good idea to break it down like that, dougw ! Once you get into the details it becomes so much more clear why it’s so easy to spend so many hours!

In this case, I'd probably take it from a different angle.

If it was a new style of Linuxy SBC, I'd compare it to the better know ones, then try to find differences and poke into those. To prevent that I'd end up with a story on how Linux works on another board. Try to get the unique things out of the hardware and supporting libraries.

If it'd a Pi 4, I'd only focus on where it's different with the Pi3. And if that changes anything for a user (good or bad). Really focus on the design decisions and differentiators.

The longer I do road tests, the less time I spend on verifying manufacturer claims. I want to see how useful it is for a user (me).

100% Agree. The manufacturer's engineering team already knows the measured performance if it's a charlie release. I myself rarely operate equipment at it's corner points, old schoolers add fudge factors.

In fact, I'd be more interested in the user's perceived performance difference of a RPi4  1 Gb, 2GB, 4 Gb, and 8 Gb models.  

Dougw , your checklist list is a pretty thorough and accurate, nice.  The disclaimer on variables pretty much covers everything. 

One bullet that could be added just as a holding spot is learning curve investigations (going down the rabbit hole). Especially in your description Chapter 1, 9, & 10  I'll always have time spent on investigating things I didnt know, that I didnt know....fixing stuff I broke......and things I thought should take 10 minutes that take 10 hours.    

holding spot is learning curve investigations (going down the rabbit hole)

"Learning curve investigations" is going to be my new name for rabbit holes.

I formally adopted the term "multitasking" when it was used in online meetings as a polite synonym for "sorry, please repeat, I was getting real work done and wasnt paying attention". 

robogary said:

I'll always have time spent on investigating things I didnt know, that I didnt know

Love it... those unknown unknowns...

Yes, I would agree with you that this is pretty thorough.

To cater for our unknown unknowns, I personally like to add in some contingency, even though it invariably gets eaten up. The amount of contingency very much depends on how familiar I am with similar products and whether this is a trailblazing new product with little to rely on or whether it's an improvement on an earlier version.

Contingency is definitely missing and you are right - it always gets eaten up. 

Project manager tip: Do not plan for contingency. Or use a fixed number, like 15%, and consider it taken .

https://www.indeed.com/hire/c/info/what-is-a-contingency-plan   A contingency plan helps you prepare for unanticipated occurrences that might otherwise cost you time, money or even your entire business. Sounds pretty severe.

For a Road Test, contingency really isnt the right vocabulary word, but anticipating and showing some time spent for exploring unknowns, tester learning, and flat out playing is part of the road test. Doug's task list is a good organized "proof"  for the estimated hours spent on a road test, tho adding in a bit for exploring time depending on the road test product's complexity, is food for thought.