Continuing The Conversation About RoadTests #4, 2019 Results To Date, Policy Changes, Multiple Accounts, RoadTests at Work

Great summary.

While reading, my thoughts drifted to a mentor-ship program. There are some really brilliant minds visiting and sharing on this site. Getting a chance to partner with them to complete a RoadTest, would benefit them and our community.

Sean

That all sounds good, Randall,

Looks like a good way to handle what is generally a tricky thing to manage.

> ...it is clear to me that a good number of people who do not write reviews do not participate on the community. ... I will be re-emphasizing community participation in selecting roadtesters.

That sounds like a really good idea.

ps, what does a 99% compliance mean? or partial compliance in general.

Cheers,

-Nico

rscasny, just a small thought on the student roadtests. All students do projects in their college final year. Particularly for those with electronics and computer science background, roadtests are a great chance to have the motivation to research more so as to effectively learn and do by their own. Of course, the guidance from the specialists at element14 will be an added value.

Adding to this, it will be beneficial to have an initial student roadtest completely on the basic necessities required for the roadtests that may follow. Most people refrain from roadtest not because they dont want to, but because they simply cant do due to lack of support tools.

Am with your Point jissdeodates !

All good points. I can see the road test program is going from strength to strength recently. It looks like it will continue improving.

I think I probably talk too much around the RoadTest area already as it is - so I'll try to keep it as brief and relevant as I can.

rscasny  wrote:

Here are the results:

• 10 roadtests - 100% compliance
• 10 roadtests - 75-99% compliance
• 5 roadtests - 50-74% compliance
• 2 roadtests - less than 50% compliance

Compliance rates look much better than before, but I think there should be an emphasis on not only compliance but quality and/or engagement. I don't say this only because I feel like I put in a decent effort into my reviews, but ultimately the sponsor and community doesn't benefit from completed three-paragraph reviews that might otherwise tick the "compliance" box. Previously, we may have mentioned supplier feedback or involvement post-review - I understand some suppliers may be a little hesitant to be too involved, but I think it's probably something to think about.

rscasny  wrote:

So, what is a RoadTest? Perhaps it is easier to say what it isn't. I don't think a RoadTest is always a project build. Given the limited time of a RoadTest, there may not be enough time to build a project or design a prototype. Of course, this would depend on the product and the RoadTester. I don't think a RoadTest is a means to obtain a free product to test out in a live system or mission critical application (more on this later).

 

What is a RoadTest? A roadtest is simply an experiment with a product where the tester gets to know it well enough to see how it responds or spot operational issues or problems with the documentation. Of course, a roadtest depends on what a tester wants it to be given the limited time I give for a RoadTest. A RoadTest review should give the reader an experience of the product such that he would be interested in exploring or working with the product himself or herself. Of course, in the end, what a RoadTest is depends on what a RoadTester wants it to be.

Strongly agree with the open ended nature of the RoadTest - I always try to leave some room to pursue features and quirks which may not be apparent from the documentation. That being said, documentation review can easily be done prior to product being received, but somehow, I'm not sure all reviewers take the time to familiarise themselves with it at an early (i.e. application) stage sufficiently to plan their applications, hence the number of vague "I'll figure it out when I get it" responses. Perhaps less emphasis on "free" and more on "this is your chance to get your hands on X and answer those burning questions you have that aren't answered on paper/get hands on experience". A different kind of sell, and perhaps one we've vaguely bought up before.

rscasny  wrote:

I think the length that I give to conduct a RoadTest (60 days from the receipt of the product) has been debated for a long time. Some people think it should be longer. Others think it's fine. And still others look at the 60 days in terms of their own busy professional and personal lives--sometimes it's too short when they are busy and sometimes too long when they have nothing to do.

I encourage you to offer your thoughts in the comments section.

Personally, I think the 60-day period strikes a happy medium and should be retained. I've done reviews outside of e14 on much shorter timeframes - often this is not sufficient to get a good grasp of the product and understand potential durability or deep functionality issues. Beyond 60-days is a tough-sell to some sponsors, especially when their product launches are perhaps strategically timed and they would like content by a certain date. I think some people would like longer to do further exploration, but having nothing delivered in the interim is a big risk and whether they can maintain the motivation in the longer term is a question. Nothing better than a hard deadline to motivate a RoadTester, but also, it's important for the RoadTesters to also have some closure. Once they see the deadline and have delivered, they can "decompress" and feel relieved that they have met their brief as best as they can - otherwise they could be "on the hook" for way too long and never get anything else done on the weekends. I think community expectations, especially if the period is long, could be disastrous as it's not a community RoadTest (per-se) but a RoadTest done by an individual, and I think perhaps some of us (including myself) may have been guilty of expecting too much from others without considering their expectations. It may open up the potential for too much back-seat driving.

rscasny  wrote:

I frequently receive RoadTest applications that propose to use the RoadTest product in a prototype, a mission-critical system, a research project or a machine/control at a manufacturing facility or a laboratory.

Agreed - that's not a great idea. Having to deal with the manufacturers already can be a bit of a back-and-forth, but adding more external dependencies will likely cause issues. Some niche products are hard to RoadTest without access to a particular set-up, but I think such RoadTest propositions probably need a Plan B that runs simultaneously side-by-side, or have the "deployment" as a stretch goal rather than the main RoadTest.

rscasny  wrote:

If a RoadTester has not completed or decides not to complete a RoadTest review, I felt that the product should be returned so I can send it to another person who applied. This policy has caused some inconvenience and some complaints.

I can understand the complaints - often postage from AU back to US is going to cost more than the product, especially if it's heavy. Ultimately, I thought it would be an added incentive to complete the review and not be out of pocket ... but it seems others may have been inconvenienced because they took this in the wrong spirit. Perhaps this may lead to a new "incentive" scheme to review low-value items rather than a "penalty" for not delivering.

rscasny  wrote:

We also had a long discussion about how to get more students (first or second year undergraduate) involved in roadtests. We realize that students are still learning about electronics and may not have the background to do a RoadTest that an experienced engineer may write. And that's fine. What I will be rolling out in the next few months is the Student's RoadTest with clear expectations what this special group needs to do in a RoadTest.

That's an interesting initiative and one I would definitely support as I did try to tell my colleagues and fellow peers when I was still a student to join the program and sharpen their skills while earning some gear. Very few of them succeeded in winning any applications at the time - would be good but perhaps there needs to be clear eligibility criteria to stop us "more experienced" non-students from competing? Perhaps we need to show our student cards and use an edu address .

rscasny  wrote:

If I haven't explicitly clarified it before, let me do it now. If someone is applying for a RoadTest under two or three different accounts, both applications will be disqualified.

Seems fair to me. I never thought anyone would have enough time to submit an RT application under several accounts ... but that's definitely one way not to win.

rscasny  wrote:

New Emphasis on Community Participation in Selecting RoadTesters

While we are getting reviews done at a better rate than in the past, there are still a considerable number of reviews that never are written. I audit the roadtests regularly and it is clear to me that a good number of people who do not write reviews do not participate on the community. If you are engaged (e.g., write a blog, make a comment, take a quiz, watch a webinar) you are more likely to write your review. So, I will be re-emphasizing community participation in selecting roadtesters. Watch for changes in the roadtest application.

When I read the title, I almost thought you meant that there will be an emphasis on the community in selecting RoadTesters which would make things rather interesting. But instead, I realise what you meant is a greater weighting on community participation. I will wait and see what this means with regards to how a RoadTest application looks like.

rscasny  wrote:

One of the things that came up is that some people feel they are not as knowledgeable as they need to be to perform a RoadTest. But isn't that the point of a RoadTest? You take the challenge of doing a RoadTest to learn. Yes, you need to understand the basic idea of the product and you need to be able to comprehend the docs. But then go at it and ask questions.

I think this is a double-edged sword and really depends somewhat on the product. Of course, if the member is willing to learn, the RoadTest process can be highly beneficial for them and they can develop insights from the perspective of a novice. However, other products may demand reviews from a higher technical level to appeal to those who might be in the market for the product. I think the wisest applicants are those who know when they are biting off more than they can chew (with the two months of time given, their existing skill-base and time available) and perhaps elect to let a RoadTest pass by so that they won't be the one posting a review that says it was "too difficult to get working". Ultimately, the applicant who does not apply has no chance of winning, but they in turn may avoid banging their head against a wall or turning in an application which costs time to be judged but would not be competitive anyway.

Of course, this is just my 2c from my perspective, as sometimes you need to be very careful of what you wish for (again, myself included).

- Gough

ntewinkel  wrote:

ps, what does a 99% compliance mean? or partial compliance in general.

I believe rscasny just means the amount of reviews posted within the 60-day window.

So in the case of 75-99% compliance, this means anywhere from 75-99% of the number of expected reviews for that particular product were posted within the 60-day window - e.g. give away 100 Raspberry Pis and 76 reviews are posted in 60-days, or give away 4 power supplies and three reviews are posted within 60-days would be 70-99% compliance.

Which means ultimately "not everyone selected for that particular RoadTest posted their review on time".

- Gough

It's good to see that the "compliance rates" have improved so much under Randall's guiding hand.

I know it's subjective, but I agree that the "quality" of the RoadTest report is also important. That doesn't mean it has to be several thousand words (which can be a bit off-putting), but I think a RoadTester should demonstrate that they have made an effort commensurate with the device they are testing.

Whilst I agree that improving the quality of roadtest is important, you have to walk before you can run, so the improvement in the return rate of the roadtest is great to see.

Improving quality is also harder than improving the response rate and ultimately requires feedback. I wonder if this is something like providing mentors to those completing a roadtest, if they wanted one, could help in improving quality and also not be too onerous for element14, if some of the more experienced roadtesters would agree to assist with the mentoring?

Kind regards.