Actions
Forum Thread Details
  • Replies 124 replies
  • Subscribers 2446 subscribers
  • Views 8426 views
  • Users 0 members are here
Related

A Conversation about RoadTests, Missing Reviews, and the Future

rscasny

Several years ago when I interviewed with Newark element14, I visited element14 website as part of the preparation for my interview. Most recently, I had been a technical writer for an industrial connector company that sold a special kind of "switch-rated" connector used in industrial facilities, oil fields and ocean platforms, mines, generators, wastewater treatment plants, among others for high horsepower, 3-phase induction motors that could interrupt currents at full load and eliminate the risk associated with arc flash accidents. So, element14.com was quite a different kind of company for me! I told the interviewer that I was "wowed" by the depth of content, its variety, and the endless amount of participation of the element14 community members.

 

I thought back then, as I do now, that the RoadTest section of the community was remarkable. The reviews, including vivid images, scope shots, and videos were just so special. Even now, I enjoy reading through them; the authors of RoadTest reviews exhibit a passion for their subject and a depth of knowledge that is unmatched.

 

I didn't initially work in the Community when I joined the company. I was assigned to the Video team where I produced product video scripts for our transactional sites, Newark and Farnell. With recent staff changes, I ended up taking over the management of the RoadTest program. It is a lot of work, but it is satisfying when am complimented on the RoadTest area. Of course, these compliments are your compliments-- the community of RoadTesters.

 

There's much more work involved in managing the program than may appear to the casual observer. I interact with suppliers, select products to review, order & ship parts, create the RoadTest pages (with the help of my colleagues and ).Since I am the main contact for the suppliers who want to participate in the RoadTest program, I hear what's on their minds: their concerns and their problems.

 

On the topic of problems and concerns, last December a supplier contacted me regarding a RoadTest Review he wanted to read; he couldn't find it. I just thought he had problems navigating our huge website so I looked for it myself; I couldn't find it, either.  I contacted the RoadTester regarding the review I could not find several times; I received no response. The supplier contacted the RoadTester as well. (The RoadTester had logged onto the site after being contacted by us.) Ultimately, in frustration, the supplier's National Sales Manager contacted the RoadTester asking for the product to be returned. As a result of this experience, I spent a day auditing the RoadTest Reviews. I stopped when I saw 85 were missing over the last year. I realized then I had a problem that I needed to address.

 

Understanding the Problem of Unwritten Reviews Has Been a Problem for Me

 

Prior to writing this piece, I met with other members of our team to discuss the problem. Was it a problem, really?  I think that's a valid question. After all, our RoadTesters are human beings. You get sick. You have family problems. Your day jobs can overload your time. I guess if I were an official RoadTester and a family problem was preventing me from submitting a review by the deadline, I would send a courtesy email telling me (or Kelly or Diane) of the situation. I rarely receive status updates by the RoadTesters, which I found a bit odd. (When I do, it usually concerns shipping issues.) Maybe is an age thing: I am much older than many of our members. I was in the military, trained to "go the extra mile." Who knows!

 

I think why I am having a difficult time understanding this problem is when I began following up with the RoadTesters. I contacted an official RoadTester about his review, which was due 4 months prior to my message, and he messaged me back with the following: "I don't have the resources to just pick it up now. Sorry." I guess I just got blown off.

 

I contacted another official RoadTester asking for the status of his review. He responded with the following: "Well, first I would like to reaffirm that I WILL BLOG HERE and honor what was asked of me." It's been about 75 days since he received the product and I haven't received the review. I guess he is not someone who stands by his word. I messaged him today that without the review I would have to suspend his participation in the program. He shot me a message and saying that's okay by him. He still has the product and I don't have the review. And I have a supplier to answer to.

 

Another person even blamed me for asking for reviews because I was dissing all RoadTesters (presumably for even asking.) That's not worthy of a response.

 

I guess I am perplexed. After all, a RoadTester is given a free product and in return all I am asking for is a posted review. You don't have to write a book. Just provide your objective opinion.

 

So, how will this problem we solved. Well, let me start with a self-critique.

 

Solving the Problem of Missing RoadTest Reviews

 

I could make excuses and complain about my heavy workload, but that won't solve anything. For starters, I think I could do a better job at communicating with the official RoadTesters. I have communicated, but it has been later in the process; I have to do a better job at communicating earlier. I have enlisted my colleague , our Community Manager, to help me. We will try harder.  But open communication is a two-way street; I feel I should not have to always initiate communication on the status of your RoadTests.

 

I also feel I have to do a better job about ordering parts earlier in the process. I have been managing the program for a few months and have realized that many of the products being roadtested are new product introductions, which often don't have a lot of inventory; therefore, I have to order them earlier. Moving forward, I will not create any new RoadTests until I have possession of the products.

 

I think I have to do a better job at screening RoadTest applicants. I review all applications and make recommendations before sending the whole package off to the supplier who makes the final decision on official RoadTesters. I think our current RoadTest application form is too general. Therefore, I am in the process of improving the form for better screening of applicants, regarding their qualifications, knowledge and (most importantly) their participation in the element14 Community. People who participate a lot by commenting, writing blogs, and reporting on project builds deserve to RoadTest that $3,000 piece of test equipment.

 

Finally, I have noticed a lot of the applications that do not communicate clearly what they are going to do for a RoadTest. Some applications are so long that I wonder if the project can be accomplished in 60 days. I've pondered this most of all, asking myself what is a RoadTest? This reflection made me list 5 different RoadTest types:

 

  • Unboxing (14 days to complete): I think an unboxing can be a useful way to RoadTest a product despite its apparent simplicity. It surely tests if the product works out of the box and all the parts and documentation are provided. But this type of RoadTest does not require 60 days to complete. I think it can be done in a few hours. For this reason, if a RoadTester chooses to perform this type of RoadTest, he/she will need to post the official review in 14 days from delivery of the product.

 

  • Test a Product Claim (21 days to complete): I think testing a product claim, albeit more limited than an project-type, is another way to RoadTest a product. For example, some manufacturers claim you can get the product up and running in 30 minutes. Well, go ahead and test it! And report what happens. I give this 21 days to complete.

 

  • Testing to a Procedure or Standard (30 days to complete): For some products, companies will provide a detailed procedure for pre-compliance testing. (There are also testing standards by regular orgs; these may be more difficult to replicate the procedure.) In any event, testing a D.C. power Supply to an established Procedure would be an example of this type of RoadTest. Since it requires a bit of research on the RoadTester's part, I give it 30 days to complete.

 

  • Simple Project (45 days to complete): Projects always take a lot more time and creativity to produce. Plus, the review itself is usually longer. A simple project would be something like taking a Raspberry Pi, adding a sensor on the input and a motor/actuator on the output to perform some type of action. I give something like this 45 days to complete.

 

  • Complex Prototype Build (60 days to complete): Complex projects are the pinnacle of RoadTests. They take the product and use it to build a product prototype. This type of RoadTest would be similar to a Design Challenge project. As such, it would require more time and a more involved review. Hence, it merits more time. I give it 60 days to complete; however, I could be swayed to 75 days if it were really something unique.

 

The Solution: Not Only Me But the Community

 

While it is my job to manage the RoadTest program, and offer my ideas, I think the solution will be forged by the community of RoadTesters as a whole. Perhaps some ideas I have offered in this piece, resonate with you-- tell the community why. Perhaps you think I am off base or wrong; that's fine, tell the community why (but anticipate that I will engage you). Perhaps you have another idea; by all means enter it in the comments section below. I will be rolling out some improvements to the RoadTest Program, so stay tuned. Thank you for participating in our Community.

  • I like that you are now attaching all the needed dates to the road test. Not only gives this the applicants a time-line when to expect that the testers are selected, it gives us testers as fixed deadline when to post our reviews (disclaimer: I have one review I need to finish right now). This makes it clear when we need to deliver ("60 days after receiving the item" is something that E14 can control only partially).

    Given the 5 types of road test you identified, I think you should mention which kind of a road test you expect. This would communicate to the potential testers what you expect from them, so one might think twice before applying. When you want to make previous community participation a requirement (or consideration) for selection, it would not be useful for me to e.g. apply for the keysight scope road test (even though I participated art several challenges and did several road tests).

    Just my 2 cents,

    hli

  •  

    I think you've identified the same issues we observed some time back.

    Without 'inner knowledge' it was simply an observation, but we did question the value of providing high value equipment to people that had signed up to the community about the time the Road test was released.

     

     

    There really is no excuse for not communicating when there has been some change to the the applicants situation.

    In some cases the timeframes slip, which then clashes, but surely there is time to email and advise that.

     

     

    I love that the supplier requested the product back. He has every right to and good on him for following it up.

     

    In theory UPS/Fedex should be reporting back to element14/Newark when the item has been delivered, so the review deadline time should start then.

    A quick email to confirm that they have received it should reinforce that, and allow both parties to understand when the review is due.

     

     

    I do like your timeline, but I never really thought about the testing details.

    Maybe that should be part of the application for high value products.

     

     

    Currently the Roadtest review is a one shot.

    Unlike the Design Challenge where there is a blog area, so maybe there is room to improve and then someone may provide 4 seperate appropriately timed pieces.

    This would also allow for any delays where the applicant has gone back to the supplier for updates/issues etc.

     

     

     

    I've been fortunate to take part is several Road Tests and I would never consider 'just keeping the gear'.

    Maybe that because I'm old shool as well.  image

     

     

    Mark

  • It is disappointing not seeing reviews. That is an exciting part of element14, seeing people's experiences with a product.

    It is rude to not submit anything in return, and the responses sound truly awful. The interaction we receive from element14 contains nothing but friendliness and helpfulness, it would be disheartening to receive the types of responses you mention you have received.

    The changes made recently are a great start, the dates and reminders (even though in an ideal world you shouldn't have to provide them) are very welcome, and it allows us to remind each other too in the comments when we are expecting to see a review, so we can help too.

  • in reply to shabaz

    Shabaz, I agree with your point of view.

    Not always it is possible for lot of reasons to be 100% respectful for the dates but a road tested that is really doing the job he assumed the responsibility is not someone that disappears from the community before the timeline dates. So placing these rules maybe - hopefully - a good lineguide. Then when there are problems I see that iti s always sufficient to notice it, explain the reasons, modify the the rules on the go. That is totally different than thake ths stuff and disappear image

     

    Enrico

  • The Road Test program is definitely an extremely attractive program - I couldn't believe it when I first came across it. Free electronics - and all you need to do is review it. I also find it amazing that there aren't thousands of applications for expensive instrumentation.

    I have noticed your influence on the program and your encouragement of road testers to complete their commitments, and would like to congratulate you on your efforts and tact.

    Your explanations above provide excellent insight into the program, how it works and the various issues - I got a lot out of it.

    I like the various initiatives you are planning, but here are some perspectives from the member point of view (or at least my perspective).

    I don't think categorization of the road tests (your 5 types) is worth spending your effort on. Road test objectives are already spelled out for each road test. When applying for a road test, we are competing against all other applications, so the only way to "win" is to propose something better in some way than other applications. You can be sure all successful applicants will address the road test objectives that are spelled out, so to have a better chance of "winning" you really need to propose something beyond what the road test requires.

    This type of situation can lead applicants to propose projects that stretch their capabilities to the point where it is unlikely they can deliver. And if you do make a mess of things it is a pretty public exposure. Each person handles these types of stress differently, so it is not too surprising that many opt out.

    I think it is important to take some chances on new members and on projects that are a stretch because when it pans out, the rewards can be great.

    Each manufacturer will have some level of tolerance for risk, they just need to understand what the risks are. For example (I know this is a lot of work, but ...) if road testers are given a rating after each road test (by the manufacturer) each tester will have a rating average that can be used to provide a comfortable mix for the manufacturer. They might want 80% of their testers to have a high rating and be willing to risk 20% on unrated testers. On low priced components, perhaps the ratios can be more relaxed, allowing road testers to build up their ratings.

    Or, as has been suggested by others previously, a certain number of blogs may be required before the road test kit is sent to each road tester.

  • in reply to dougw

    Doug,  your valutative analysis sounds very good to me. The only thing that I am not sure to agree completely is about the number of blogs before. By one side this is almost granting to stay in the risk percentage (no matter if it is what you propose of different) but by the other side this maybe a constrain; I mean, there are many users that for many reasons never tried to publish something by themselves but then their capabilities and skills have been unlocked just starting from a road test. The problem I think it is also related to the self-judging of the road testers themselves. In many cases I have been attracted by measurement devices, almost expensive but I have proposed a road test (and not accepted, but this is part of the game) only when I was sure that I was at least able to grant a serious analysis of the device; this was possible until some months ago as I was counting on the support of the NXP hardware lab, not by myself it is rare that I can afford the risk to test a expensive measurement device proposing something that then I ma not in the conditions to do. In these cases the choice is very hard.

  • Food for thought:

    What I don't see in the text is self-reflection.

    Is there a filter to detect the over-promising entries? Do the simple ones that propose a realistic review make it?

  • I think the basic ideas are fine - I'm sure that you'll have some flexibility for different items and circumstances. Having some clear guidelines about what is expected seems a good plan.

     

    It's never going to be possible to screen out all the non reviewers but it may be worth only giving big ticket items to member who have established a track record in some way.

     

    I do think that it's very easy to under estimate the amount of time it takes to do a good review - it might be helpful for people who've done reviews that attracted good comments to say how long they took . Not quite sure how this could be shared but there may be  away.

     

    MK

  • in reply to dougw

    You make some great comments. Thanks. I don't think we can expect 100% of reviews all the time. That would be unrealistic. But right now, from what I found, it's less than 50%. I can even handle someone who has not completed the review at the due date messaging me and letting me know what is going on. That's fine too. But it isn't always happening.

     

    Thanks for your comments about the 5 different types of RoadTests I proposed. I did that as a way for an applicant to balance his/her time vs what is being proposed in the application.

     

    As far as competition goes, it's not always the best application that wins. Sometimes a weaker application with a simpler project that can be completed quickly is beneficial. I am most interested in someone who knows what's going on and can pull it off.

  • in reply to Jan Cumps

    There hasn't been a filter to detect over-promising. That's why the application form is being changed. If someone I don't know proposes a realistic review, I try to go to their profile and see what content they have created. I try to gauge abilities, participation and the application in my recommendations. Unfortunately, it's a judgement call. Most of the missing reviews I saw were for RoadTests prior to my involvement in the program. But I have been told this problem is not new. In any event, I have to forge a solution. Thanks for your input.