It's been about a year since I last had a conversation with the Roadtest Group. This will be be my fourth conversation, where I share with you RoadTest results, policy changes, my thoughts on improvements, and anything else that I feel is important or what RoadTest group should know about. So, let me get to it!
2019 Results to Date
I went back and looked at the completed Roadtest list for 2019 and recorded the completion percentage for each one. This list excludes those roadtests in enrollment or recent ones that are now in the reviewing period. Here are the results:
- 10 roadtests - 100% compliance
- 10 roadtests - 75-99% compliance
- 5 roadtests - 50-74% compliance
- 2 roadtests - less than 50% compliance
Pretty good overall! I'd like to thank my roadtester superstars. These are great results. We are in a completely different place than were we were two years ago when I took over managing the RoadTest program. Back then, my first roadtest audit revealed only a 13% completion compliance.
I realize many of the roadtesters have full time jobs with families, so your time is limited. So, these results reveal a high level of commitment and professionalism. Your participation is truly appreciated. I can't thank you enough. It is your work that makes the RoadTest program interesting and thriving on the element14 community.
If you haven't completed your roadtest, I encourage you to post it on element14 as soon as you can. Thanks.
My Philosophy: What Is a Roadtest?
Before I get into some of the RoadTest program news and changes, I wanted to share with you my philosophy on Roadtests. I have mentioned it a number of times to members, sometimes within my comments right on RoadTest pages. But I want to share it with the group again. So, what is a RoadTest? Perhaps it is easier to say what it isn't. I don't think a RoadTest is always a project build. Given the limited time of a RoadTest, there may not be enough time to build a project or design a prototype. Of course, this would depend on the product and the RoadTester. I don't think a RoadTest is a means to obtain a free product to test out in a live system or mission critical application (more on this later).
What is a RoadTest? A roadtest is simply an experiment with a product where the tester gets to know it well enough to see how it responds or spot operational issues or problems with the documentation. Of course, a roadtest depends on what a tester wants it to be given the limited time I give for a RoadTest. A RoadTest review should give the reader an experience of the product such that he would be interested in exploring or working with the product himself or herself. Of course, in the end, what a RoadTest is depends on what a RoadTester wants it to be.
Re-Considering the 60-Day Roadtest
I think the length that I give to conduct a RoadTest (60 days from the receipt of the product) has been debated for a long time. Some people think it should be longer. Others think it's fine. And still others look at the 60 days in terms of their own busy professional and personal lives--sometimes it's too short when they are busy and sometimes too long when they have nothing to do.
Why do I give 60 days? Why not longer? I recall a couple of roadtests where there were some problems and the RoadTesters were going back and forth with the supplier's tech support. This is a special situation and of course the 60-day time limit should be suspended in these situations and the RoadTesters given more time. If something does come up during the course of a RoadTest (a problem with a dev board or a personal health issue), communicate it to me and I will adjust the due date accordingly. Again, these are special situations.
Why am I reluctant to extend the standard 60-day time frame to 70 or 80 days or more for most RoadTests. I can answer that question with certainty: my data shows that if someone has not completed a RoadTest in 60 days, and has not made an effort to communicate to me about a problem (or even respond to my assistant when he sends out follow up emails), the RoadTester will not complete his review at all. What I think happens is that beyond 60 days, peoples' lives change in unpredictable ways. Things come up that have greater priorities and the roadtest is viewed in a rearview mirror. I can appreciate that . I don't know what will come up in my life in three months down the road. But as a policy goes, 60 days was chosen because it's enough time to experiment with the product yet not so long that if some has some life change down the road it will not prevent them from completing the RoadTest. Having said all this, I think there are some situations that would require a long RoadTest; if that's the case, communicate it to me and we can work it out. I encourage you to offer your thoughts in the comments section.
What Happens When a RoadTest Is Used To Deploy a Solution at Work or a Customer's Site
I frequently receive RoadTest applications that propose to use the RoadTest product in a prototype, a mission-critical system, a research project or a machine/control at a manufacturing facility or a laboratory. And this is great; I want our engineers to consider our products in their designs. But what often happens is that the RoadTester has to wait on the client to approve something, or a third party needs to do something before the RoadTest can begin, or even there often is a change in plans and the RoadTest is replaced by other things with greater priority. The question arises from where I sit: is this something that is within the scope of a RoadTest? I would rather help you connect to people in our company who deal with this sort of implementation and/or testing. A RoadTest is 60 days long. You really have only about 45 days to actually do the testing. Not much time in a real production situation. If you are a professional who wants to use a RoadTest product, contact me directly, I will reach out the the people here to assist you.
New Policy on RoadTest Product Returns
If a RoadTester has not completed or decides not to complete a RoadTest review, I felt that the product should be returned so I can send it to another person who applied. This policy has caused some inconvenience and some complaints. I took this problem to my supervisor. After have a long discussion about it with my supervisor, I will not be asking that all products will be returned. We will probably establish a minimum threshold for return in the $100-200 range and up. I think we came to the conclusion that a returns policy is not the issue that needs to be addressed; rather, we need to modify our application screening process such that the people I choose do indeed complete their reviews. Expect to see a small modification to the application going forward.
The Student's RoadTest
We also had a long discussion about how to get more students (first or second year undergraduate) involved in roadtests. We realize that students are still learning about electronics and may not have the background to do a RoadTest that an experienced engineer may write. And that's fine. What I will be rolling out in the next few months is the Student's RoadTest with clear expectations what this special group needs to do in a RoadTest.
Policy Reminder: Multiple Accounts Don't Count
If I haven't explicitly clarified it before, let me do it now. If someone is applying for a RoadTest under two or three different accounts, both applications will be disqualified.
New Emphasis on Community Participation in Selecting RoadTesters
While we are getting reviews done at a better rate than in the past, there are still a considerable number of reviews that never are written. I audit the roadtests regularly and it is clear to me that a good number of people who do not write reviews do not participate on the community. If you are engaged (e.g., write a blog, make a comment, take a quiz, watch a webinar) you are more likely to write your review. So, I will be re-emphasizing community participation in selecting roadtesters. Watch for changes in the roadtest application.
Don't Know Enough to Be a RoadTester--Not!
I recently ran a poll/discussion on why a member is not more active in the RoadTest Program: Poll: How Can I Get You More Active In The RoadTest Program? One of the things that came up is that some people feel they are not as knowledgeable as they need to be to perform a RoadTest. But isn't that the point of a RoadTest? You take the challenge of doing a RoadTest to learn. Yes, you need to understand the basic idea of the product and you need to be able to comprehend the docs. But then go at it and ask questions. That is why there is a comments section on the RoadTest page. Ask the community questions, and I'm sure they will help you. If you feel you need a mentor, I will ask experienced members if they would have a bit of time to provide special help. So, reach out to me. Your questions can help all the people who are reading the comments also seeking help. Last thing: suggest some products you feel you know enough to roadtest. I will see if I can put them on the schedule.
Sincerely,
Randall Scasny
RoadTest Program Manager
