Related
Recommended

How To Deal with RoadTest Goals That Were Unachievable

pettitda

Shortly after the Raspberry Pi 3 was released, Element14 announced a new road test entitled RoadTest Review a Raspberry Pi 3 Model B !  I was instantly excited.  I had followed the Raspberry Pi project for several years and from all the specifications, this looked to be a fantastic new member of the RPI family.  So, I went to work crafting a project idea to use the capabilities of the RPi 3 to the fullest.  I had recently lost my home and half of it's contents to a flood and most of what I didn't lose was in storage while my family and I lived with my mother in law.  So, doing a full-on electronics hardware project was out of the question.  I eventually came up with what I thought was the perfect project considering the circumstances; one which was almost totally software work.  This would be an excellent way to keep my mind occupied while I waited for my house to be rebuilt.

 

I had been following the Numenta Hierachical Temporal Memory (HTM) project for several years.  I originally read about it in an issue of the IEEE Spectrum.  I knew the project was based upon a open source code base which aimed to simulate how the brains works.  I also knew that the code was successfully learning patterns of data, predicting the next data point in a series, and flagging anomalies in a data set.  What I didn't know at the time was how exactly the code went about learning patterns or what it's limitations were.  Nevertheless, I went forward to describe a project with what I would eventually come to realize were very aggressive goals.  I proposed to build a raspberry pi based computer which would recognize speech and trigger actions based upon that speech pattern. 

 

In a short amount of time, I learned that I had been selected to road test the RPi 3 and a few days after that I received the Pi.  I jumped in with both feet, ordered a USB sound adapter, microphone, case, power supply, and SD card, then started working on the software. Porting the Numenta project to run on the Pi was slow work, but I found the user group on Github to be friendly and willing to advise me on getting it to build and run.  After two or three weeks the project was ported, built, and running on the Pi.  From that point I spent a couple more weeks getting the project updates for building on RPi up to the project standards for inclusion in the main line code.  While not strictly necessary, I figured this was the least I could do considering how helpful the group had been to me. 

 

From this point, I jumped into the examples online to look for a starting point from which to build my project.  I found examples of predicting the energy usage in a gym and similar sets of data which were long arrays of fairly simple data (i.e. a watt-hour reading for energy usage taken every hour for several weeks).  However, what I didn't find was examples of processing dense, complex data such as a person's voice.  Eventually, I came to the realization that the project goals that I had given myself were more appropriate for a PhD student doing original research for a doctoral thesis than for an engineer tinkering with the software on his off hours.  I was seriously disheartened.

 

If you find yourself in a similar circumstance, here's what you should do.  First notify the road test manager of your problem.  Seriously, these people are friendly and understanding and they want to see you succeed.  Second consider how you could refocus the project in order to make it successful.  Talk it over with the road test manager and get their input.  Post a message on the road test area and get feedback from the other road testers.  Perhaps there's an alternative way of looking at the problem that could get you going.  Third, if you are totally up against a brick wall, you may have to abandon the original project and come up with an alternative.  Again the road test manager can be extremely helpful.  Just because your original project proposal is not achievable doesn't mean that you can't complete a perfectly acceptable review of the product.  In the end, the goal is a review of the product which is insightful and helpful to others considering using the product in one of their projects.  Finally, blog about your failure.  The experience you learned in failure may be extremely valuable to others considering similar projects. 

 

I hope my humbling story will encourage you to not simply give up but make the most of your situation.  Happy road testing!

  • in reply to hlipka

    Thank you Hendriik for sharing your experience.  I think you have clearly illustrated what should be done in a Road Test of any product and how to respond to emergent issues that could derail a well intentioned review plan.

    I'd like to hear from on this approach to Road Testing.

  • From my perspective, I like to suggest what I will do with the product to review it and also want my plans would be post review.

    As many of the comments have correctly stated, dont over promise on the road test itself and then fail. and Don't over promise just to get the goods either, this is unfair to the other potential reviewers and the sponsors. Lastly dont try to get it all done in one post with everything you promised. Break it down and post as you go. for example

    1. Unboxing and what's provided, what is the quality of probes, construction etc. perhaps power it up to see the initial interfaces. Now post this.

     

    2. Tear it apart (If you're brave enough and you dont have to break something to get it apart), this is not for everyone, experienced folks like the TM;s are quite fearless for this kind of thing but dont feel you have to do it is you're not comfortable with it.

     

    3. Do something practical with it, you must have thought of a one or more applications / review topics when you applied for the road test, if you can't actually do them you should not have applied, so do one of them, on a bread board, strip board, or using your other test equipment, whatever it takes to complete your review scenario, comment on how easy it was (Or Not), how usable was the product in the scenario, did you have to add a bunch of other things in order to even get the product up and running, what could the manufacturer do to fix that etc. This is the main part of the review, the manufacturer wants your opinion, not a variation of their own marketing material, they want the good and the bad, dont be afraid to say what you think, just dont make it personal or use profanity, its supposed to be constructive feedback based on your proposed and now executed review scenario...

     

    Now you have effectively satisfied the Road Test requirements, by breaking it up, you have three easy posts and won't be thinking you have way too much to do and leave it to the last minute to post or do nothing. You can post each bit separately and go back and edit to add the subsequent parts.

     

    4 Do something else practical with it.....

    5. repeat 4 until you run out of ideas or steam....

     

    Now you have some more street cred (Community Cred really), your building points, people are seeing you keen and are interested in this nice shiny new toy (Yes for us there toys from our favorite toy store, E14 ) and are demonstrating you want to share your experience beyond just trying to get free gear and your improving your chance to get more in the future, and even better, keep it up long enough with this and other products or projects you may get invited to the Top Members invite only group or be selected for Member of the Month.. now there's a reward

     

    For bigger (AKA More Expensive) products I will often perform an initial review to satisfy the road test requirements (NOT A RE-WRITE of the market material though), more of a how easy is it to use, it the UI (If appropriate) intuitive and easy to use, how much can you do before reading the manual as it were , I will also perform a couple of real life scenarios with the product, either by writing code for a typical application, as would be the case for an ADC, DAC, or other EVM type product, or some practical Measurement scenarios if it is a lab or hand held instrument.

    Now I have satisfied the Official review.

    so it is time to have more fun, for instance getting the 33622A ARB to make a BAT Signal or use it to measure how long a wire is using reflectometry (Still with a BAT theme in this case). the point is that if the product warrants it, people love to see more interesting things being one with it. and if it is a scope, DMM, ARB etc then your typically getting something worth a lot of money so its worth the effort to say thank you in the form of going the extra mile to show people how to use it in more fun or practical ways.

     

    Atleast, thats how I see it.

  • in reply to gpolder

    Perhaps not a design challenge, but element14 were obviously originally hoping for a bit more than just a cursory evaluation if the FAQ is anything to go by:

     

    https://www.element14.com/community/docs/DOC-83454/l/roadtest-frequently-asked-questions-faqs

     

    RoadTests go beyond traditional reviews by providing an objective opinion about a product's quality, ease of use, and performance – usually in a real prototype or bench top project.  Our manufacturing partners benefit as well by finding out what additional documentation might be required or how to address how their products could better perform in various "real life" applications.

  • in reply to hlipka

    Hendrik Lipka wrote:

     

    I think a road test is not a design challenge.

    I completely agree, this is definitely the point!

  • in reply to hlipka

    I think it would be extremely difficult for the road test manager and also the vendor's representative to determine whether every proposal was or was not achievable in the given time frame.  No one is an expert in every field.  If the proposal broaches a field that the application reviewer is not familiar with then there will be a lot of research required before they could have a good handle on the feasibility of the project.  Without doubt, there is some common sense and even gut feeling involved, but even that will not weed out every unreasonable proposal.

     

    In my case, I did not dig deep enough into the details of the project before writing the proposal and submitting my application to realize my goals were not achievable.  In the case of the person reviewing my application, they would have had to get familiar with the project and research it in depth to find out what I had not found.  In most cases, I suspect that reviewing these applications is not the primary focus of their job.  So, they would be hard pressed to put such time into the process.

     

    Perhaps what might help is to have some people in the community who can review applications on a part time volunteer basis and make recommendations based upon what they find.