RoadTest Reflection – My Scoreboard, My Application & Uncensored Thoughts

It’s the holiday season – traditionally a time for reflection and being thankful for our blessings. Around element14, there are plenty of reasons to be thankful – a community of mostly like-minded professional engineers, mentors, hobbyists, tinkerers and enthusiasts backed by a multi-national electronics component distributor and services firm. In return, we are blessed with a lot of opportunities including design challenges, Project 14 and especially the coveted RoadTest program.

I’ve been quite a significant beneficiary of the RoadTest program and I “count my lucky stars” to this day that I’ve had the fortune of being selected for numerous highly-coveted items. While the upsides generally outweigh the downsides, this has not always been a good thing for me as I have gained a reputation for being a sort of “treasure hoarder”. I guess it was bound to happen sometime sooner or later.

I’ve been mulling over the topic of RoadTests for many years and I’ve seen many replies on rscasny's announcements. These sometimes express a dissatisfaction that they “never win”, suggesting some sort of unfairness, or that there may be some perceived “trick” to getting a RoadTest awarded. There have been calls for people to “open up their applications” to the public and I suppose that has been something that has been weighing on me for a while as it seemed the comments may have been directed to me as a member who they perceive somehow possesses a competitive advantage (or perhaps, I’m just flattering myself here).

Whatever the case may be, I suppose my gift to you as readers (or prospective RoadTest applicants), is my scoreboard to date and a full application of mine which I succeeded with. I’ll also provide my uncensored reflections and thoughts on the program as a whole and my experiences. I hope that after you have read through this post, you may have a better understanding of what I do when I approach a RoadTest application, the level of work that I put into an application, the fact that you don’t have to have “all the experience” to win and the fact that there is no secret to winning at all but to apply, apply often and apply fully. But perhaps more than anything else, it is to deliver what you have promised in the agreed timeframe to maximise your chances of repeat success.

My RoadTest Scoreboard

My involvement with the RoadTest program started back in 2011 when I saw an advertisement somewhere that you could “test and keep a Recom R-78 Switching Regulator for free”. That sounded like something I’d want to do … so I joined Element14 Community Forums back then and never heard back from them.

I totally forgot about the program until the Raspberry Pi came out and my eyes just so happened to scan across another RoadTest advertisement in a newsletter. I applied for that too in 2012 and heard nothing back.

It was not until 2013 that I saw my first success with winning a smaller item – the PiFace Digital I/O expander board for a Raspberry Pi. From then on, it’s been mostly onward and upward, with a hiatus from the RoadTest program in 2017 when I took a “once-in-a-lifetime” trip overseas. I’m just glad that the element14 community welcomed me back on my return in 2018, rather than putting me into “RoadTest quarantine”.

Back in the early days of RoadTests, the program was a lot simpler. The form itself didn’t really have a set of guide questions – we were expected to pitch without knowing what they wanted to hear. Often applications would be just a paragraph or two long and there was no great effort to set a fixed review timeframe or chase down people who didn’t deliver. It was a lot less competitive as the community was smaller but because of these problems, it didn’t offer as much allure to the sponsors and eventually caused headaches for element14.

Under the recent guidance of rscasny, things have changed dramatically. Applications are more involved than before to vet the applicants more stringently, new rules have been put into place regarding review delivery timeframes and review concurrency and new requirements have been put into place regarding community engagement. This may seem a bit onerous, but I think it is quite appropriate given the “maturity” of the program in terms of number of RoadTests and number of applicants.

So, what does my scoreboard look like in the nine years of RoadTest program participation?

I’ve listed all of the RoadTests I’ve applied to in the table above. The left column indicates whether I was successful in my application or not, followed by the number of kits on offer and the number of applicants. The percentage column indicates the “random-chance” percentage of being selected, noting that selection is not a game of chance but a game of skill, and the year in which the RoadTest was offered. The right-most column describes the RoadTest itself and where I was successful, a link to the resulting review is provided. Unfortunately, there seems to be a few RoadTests where the application area was deleted, so no data about the kits and applicant numbers are available.

What does this tell us?

• I applied for a total of 48 Roadtests, of which 47 have had an outcome.
• Of those applications, I’ve won 24 and lost 23 which is pretty close to a 50/50 outcome.
• The random-chance selection percentage shows the average random-chance percentage for my RoadTest wins is 14.29% while the average random-chance percentage for my RoadTest losses is 10.27%. Applying for the less-attractive RoadTests does provide a very slight boost in selection probability.

Upon looking at the list, I think it’s clear that I apply only for RoadTests I am genuinely interested in. I definitely have a passion for test and measurement, sensors, IoT, power and embedded, which is where I have spent my application efforts. The “myth” that I always win the RoadTests I apply for is well and truly busted. I win about half … and I think that probability is going down as the pool of applicants is widening. The outcomes show that you can get a string of wins and a string of losses even as a Top Member with recognition as the recipient of the RoadTest of the Year 2018 award. But also the myth that I only apply for high value RoadTests is also busted – I’ve spent quite a bit of time and effort on things which are perhaps more trivial in comparison.

Perhaps the fact that a few of you reading this may have applied and actually won a RoadTest for which I was unsuccessful for would understand that you too have a fair chance of winning big-ticket items and the fact I’ve signalled my intention to apply for a given RoadTest does not immediately disqualify you from being potentially successful. All I can say is that for all the successful applicants – I am truly happy for you, for you are now able to experience the joy of doing a RoadTest review. I just hope you enjoy the experience, do it justice, make the community proud and concede well and truly that your application was better than mine.

But I think the key takeaway from this is that in order to do a lot of RoadTests, you would have had to apply for even more RoadTests. So, if you’re serious about benefiting from the RoadTest program, the answer is to apply to every RoadTest you have any interest in doing and do a serious job of writing your application. This is not a game of chance – think of it like a job interview, it is in your interests to fully articulate why you should be the chosen one. This takes time and energy which are perhaps not “free” to some of you, but you should want it hard enough to want to invest this time and energy to put your best foot forward. Otherwise, there’s really no reason to complain about not being selected.

A RoadTest Application

I think rscasny has done a great job in making the RoadTest application process a lot smoother than it once was. Now, it’s a matter of filling in a few drop-downs and then answering a series of guide questions. To be a valid application, all questions must be answered.

Some people think that my application may be something special because I haven’t made any of my previous RoadTest applications public and I seem to be winning quite often. I initially resisted releasing my RoadTest applications because I felt embarrassed to share them – I didn’t think they were anything special. But as it seems to be the trend for some reviewers and Top Members to reveal their applications, I think I should add to the corpus of information out there and do the same for one of my own.

I have decided to unseal my application for testing the Keithley 2450 SourceMeter with I-V Tracer. In the interest of full disclosure, I’ve decided to also include all the “braggy” bits that I use to provide background as to who I am and my interests. For the most part, you can write this once, perhaps tailor it a tiny bit for each application or refine some parts. Then you can just copy and paste it for future RoadTest applications.

(a) Where did you first learn about this RoadTest (e.g., Facebook, Twitter, Linkedin, Instagram, element14 website, element14 member, newsletter, etc.)

I first learnt about the RoadTest from the RoadTest group on element14.

(b) Describe your technical background.

I consider myself a hands-on practical engineer that loves technical detail, with an affinity for science and running experiments. I am versatile, a problem-solver by nature and interdisciplinary in every sense.

I commenced my university studies in a Bachelor of Electrical Engineering, later graduating my undergraduate studies in Photovoltaics and Solar Energy Engineering with first-class honours from University of New South Wales, Sydney, Australia in 2012. I was a laboratory instructor for Digital Circuit Design and Embedded Systems Design for four years, as well as designing the lab program for the former course. My first published paper was in Surveying and Spatial Information Systems. The recipient of a Dean's Honour List award, Best Poster Prize and Wal Read Prize for my undergraduate efforts, I undertook an interdisciplinary PhD in water research in Civil and Environmental Engineering, awarded from University of New South Wales in 2016. At present, I am a full-time postdoctoral researcher with the Western Sydney University as a Biomedical Engineer specialising in embedded systems design.

Outside of academia, I have undertaken work with government departments, written for Australian Personal Computing (APC) magazine, become an independent "tech blogger" reviewing/repairing/examining products, completed RoadTest reviews with element14, represented element14 at a trade show and webinar. I am a licensed foundation-level radio amateur (VK2FGYL).

(c) Why did you apply for this particular roadtest?

I applied for this RoadTest as I am passionate about the field of test and measurement. I have previously tested various electronic components and subsystems. This includes testing batteries and DC-to-DC converters using one-quadrant power supply with a DC electronic load and more recently, a precision two-quadrant power supply. Through this testing, I have come to understand the costs, limitations and complexities involved with such test-setups which include limited speed, voltage range, complex interconnections/co-ordination, significant bench-top footprint and the inability to generate negative voltages.

I applied for the Keithley 2450 SMU RoadTest to evaluate how a four-quadrant precision power supply with inbuilt test-script capabilities can simplify the testing workflow and enable testing of a wider variety of devices, especially with the newly-released I-V Curve Tracer feature. I intend to fully review the SMU’s features along with the I-V Curve Tracer option by characterising various components including resistors, MOVs, gas-discharge tubes, diodes, Zener diodes, photovoltaic cells and MOSFETs (with gate bias provided by another power supply).

I had previously expressed interest in reviewing the Keithley 2450 SMU when it was offered for RoadTest review back in 2013, but was unsuccessful at the time. I hope that my more extensive test procedure and increased experience with test equipment would make me a better candidate for reviewing this product this time around.

(d) What level of experience do you have with an SMU Sourcemeter Instrument? (Select one: Expert, Experienced, School , No Experience) Describe your experience?

While I do have extensive practical experience with test equipment (including one- and two-quadrant power supplies and DC-electronic loads) through use in tertiary academic research, hobby and through RoadTest reviews, I presently do not have any hands-on experience with an SMU. This is one of the main reasons I would like the opportunity to review the Keithley 2450 SMU, as the ability to generate negative voltages would likely open up new test capabilities and reduce setup complexity.

(e) What is your testing procedure? (Please describe as detailed as possible.)

My intended RoadTest review of the Keithley 2450 SMU will comprehensively compare the features of the unit (both on paper and in practical terms) with other competitors on the market, explore and test all of its features (hardware and software) as best as I can (given the equipment I have access to) and detail the user experience from unboxing through to standalone use on the bench and by remote control.

The review will be broken into several comprehensive sections, many of which will be published in a self-contained fully-detailed blog post in addition to the summary review. The sections include:

• • Introduction to SMUs & Market Survey – this section explains the differences between one-, two- and four-quadrant power supplies and their applications. It will also compare the on-paper specifications, features and prices of four-quadrant power supplies on the market to understand the Keithley 2450’s position in the market.
  • Unboxing and Basic Set-Up – this chapter details the unboxing process including high-resolution images of the unit and included accessories. A basic introduction to the various interfaces will be given. Basic first-time set-up of the unit will be detailed including any firmware updates if necessary.
  • Benchtop Experience – this section focuses on the standalone bench-top user experience. Specific attention will be paid to the build-quality, user interface features, menu design, LCD quality, positioning of buttons and interfaces. Basic operation will be introduced, along with testing of USB front-panel capabilities. Special display modes, such as graphing, will be demonstrated testing various electronics in sleep mode power consumption. Capabilities of the device which overlap other bench-top equipment will also be evaluated.
  • Onboard Test-Script Control – this section looks at the Keithley TSP and TriggerFlow capabilities of the SMU. This will be done by exploring the documentation available and programming some simple test scenarios to be performed on the unit without PC-connectivity – e.g. battery discharge test, generation of AC waveforms and I-V sweep.
  • PC-Connected Remote-Control – this section will examine the 2450’s PC-connected remote-control capabilities, including the connectivity interfaces provided, the Keithley KickStart software (trial-mode), LXI-LAN Web Interface and SCPI programmability. Mention will be made regarding vendor-supplied drivers, however, testing will not be performed with LabVIEW as I do not have access to a license at this time. Basic experiments will be run using PC-remote control, similarly to the previous, including a basic I-V sweep of diodes and MOSFETs and battery charge/discharge cycling tests.
  • I-V Tracer Testing – this section is dedicated to testing the new I-V tracer feature. A tour of the features and options of this add-on will be given. The onboard feature will be used to characterise a variety of components including resistors, MOVs, gas-discharge tubes, diodes, Zener diodes, photovoltaic cells and MOSFETs (with gate bias provided by another power supply) in order to understand how it performs under various scenarios in both forward and reverse bias. A comparison will be given between on-board TSP, PC-connected remote control and on-board I-V Tracer functionality and performance benefits.
  • Instrument Performance – as the Keithley 2450 SMU is likely to be more accurate than the other instruments I currently possess, it would not be possible to properly evaluate the accuracy of the output (although it may still be attempted). Instead, parameters including the warm-up drift, output programming speed, generation of arbitrary/AC waveforms, trigger I/O & interlock response times, read-back speed, transient step-load response, CC/CV transition response, instrument stand-by power consumption and efficiency will be tested. Thermal/acoustic performance under full source/sinking load will also be subjectively evaluated.
  • Conclusion – in this chapter, a summary of the key findings made throughout the prior chapters will be made to come to an overall conclusion.

It is important to note that while this is my currently proposed RoadTest, slight variations may be necessary to accommodate new discoveries made during testing or to overcome unforeseen issues out of my control. It is my intention to work closely with the manufacturer in case any functional anomalies are discovered during the review process to aid their rectification.

(f) Have you written a blog on the element14 community? If so, please provide a link. If not, say "No."

Yes – I have a personal blog on element14, with a recent posting being It’sa me, Oscilloscope! - Making Music with a 'scope?.

(g) Have you created a discussion on the element14 community? If so, please provide a link. If not, say "No."

Yes – my most recent posting was to bring attention to a typo in the sidebar menu of the RoadTests Area.

(h) Have you attended a webinar on the element14 community? If so, please provide a link. If not, say "No."

Yes – although not usually in real-time due to the time-zone difference. The last webinar I attended was the Nordic Semiconductor Thingy:91.

(i ) Have you participated in Project14 on the element14 community? If so, please provide a link. If not, say "No."

I was a recipient of the grand prize for Giving Gifts on Valentine’s Day and runner’s up prize for RF and Acoustics. I have submitted a series of postings for the recent Project14 Acoustics challenge, my most recent posting being It'sa me, Power Supply! (Part 3) - A PSU that Plays MIDI?.

(j) What other ways have you participated in the element14 community? If none, write in "None." If you are a new member, answer "New Member."

I am a Level 9 member of the element14 community, ranked 19th at the time of application and a member of the Top Members group. As the recipient of the RoadTester of the Year 2018 Community Award, I have been heavily involved in the RoadTest program, with an extensive track record of delivering comprehensive RoadTests on-time even under demanding situations. The full list of completed RoadTests include:

• • B&K Precision Portable Multi-Channel Recorder
  • Omron Environmental Sensors
  • Cypress EZ-BT Module Mesh Evaluation Kit
  • Infineon MOSFET Gate Driver IC Evaluation Board
  • Harting MICA CISS Complete IIoT Starter Kit
  • IDT SDAWIR03 Wireless Flow Rate, Humidity & Temperature Sensing Evaluation Kit
  • Eaton SmartWire-DT (EASY802-DC-SWD) RoadTest (New Year's Grab Bag)
  • Rohde & Schwarz HMP4040.04 Programmable Power Supply
  • Molex 2.4Ghz / 5Ghz Antenna Kit
  • B&K Precision Battery Analyzer BA6010
  • Rohde & Schwarz RTM3004 Oscilloscope Kit
  • Texas Instruments TPS56C215EVM-762 Synchronous Step-Down Converter Evaluation Kit
  • B&K Precision Model 8600 DC Electronic Load
  • Keysight E36103A Power Supply
  • Keysight U1461A Insulation Resistance Tester
  • Tektronix RSA306 USB Spectrum Analyzer
  • Picoscope 2205A Oscilloscope
  • Keithley 2110-120 Digital Multimeter
  • Tektronix PA1000 Power Analyzer
  • PiFace Digital
  • Raspberry Pi New Year's Party Pack

My first RoadTest was selected for publication in the element14 Connect Magazine (Australia) and my RoadTest of the Keysight E36103AE36103A Power Supply was nominated as a candidate for RoadTest Review of the Year 2016. My recent RoadTest of the Rohde & Schwarz RTM3004 led me to receive the RoadTester of the Year 2018 award. I had also participated in delivering a webinar about Getting Started with Raspberry Pi & Accessories.

(k) Can you commit to completing testing and review writing on element14 within 60 days from the receipt of the product? (Yes, No, Maybe)

Yes.

(l) I understand that if I do not complete the roadtest review on element14, I will no longer be eligible to participate in the RoadTest program in the future. If you understand, write "yes."  If you do not understand say "no" and explain what you do not understand about the statement.

Yes.

While I personally don’t think my application to be anything special, I suspect that compared to some other members, my application is perhaps quite long and wordy. In fact, my application is longer than some delivered RoadTest reviews. But this is how I write my applications and I know it’s not the only way to write an application that succeeds. You’ll see there’s nothing particularly special about it – I answer the questions in as much detail as I can.

You will see that the most important section, namely the part that asks about how you would test the product, follows an almost formulaic approach. For me, over the years of RoadTesting, I’ve found myself settling on a style of review that breaks down the process into a market survey – unbox – set-up – bench use – remote-connection use – experiment – conclusion sort of structure with some variances depending on the product. Some products demand more detail due to their complexity and may require detailed blog posts to be delivered while others can settle for a “monolithic” RoadTest report. I’ve made sure to include the details as to what will be delivered, what will be tested but also what will not be tested. Full disclosure is key.

It also pays to research the product ahead of time, learn the various things that might be needed and accessories to be used in conjunction with the product and name them in the application. This should provide the assessor a confidence that you are familiar with the product (at least superficially) and its strengths. It would be a good time to open up a folder in your workspace to collect documentation (manuals, datasheets, application notes, comparison guides) and software in preparation in case you do win.

But more than that, it’s about being honest. You should answer the questions truthfully so they get the person they want for the review, not who you think they might want for the review. Note how I’ve been upfront in saying that while I do have experience with test equipment in general, I don’t have any experience with SMUs. Notice how this did not disqualify me from winning the RoadTest for this rather expensive and exotic item. Don’t think just by seeing such a question that you have no chance just because you don’t have the experience – maybe that’s what they want, or maybe you can demonstrate that you have enough other experience and skills that they’d want to choose you anyway.

Now that you’ve seen my application, I hope you understand that there is no secret sauce to winning a Roadtest. I just followed the procedure, answered the questions and hoped for the best. It’s always best to approach these things with the mindset of “it’s great if I win, but realistically, in all likelihood, I probably won’t.” This way, if you do win, it’s a bonus! Remember that nobody is “entitled” to a win … that’s not how the world works. I’ve caught myself wondering why I didn’t succeed on RoadTests where I thought my application was good enough – but I know now that it’s done, there’s nothing I could have done to change the outcome as I put my best foot forward on the first time, so I can be at ease. There’s really no point second-guessing yourself as every product is looking for the “right” RoadTester and that may not always be who you think it is … and it may not be you.

I hope you don’t just go and copy/paste this into your next RoadTest application. It’s nice to promise things in the application, but the key is that you have to deliver on it within the agreed timeframe or you’ll likely lose your RoadTesting privileges. The same may happen if you try to game the system, so don’t even bother.

Reflecting on RoadTest Experiences

Given my long-time participation in the RoadTest program, what do I think about it? Well, as a RoadTest is an opportunity to give an open, honest and unbiased opinion on a product, I might as well give you my uncensored review of the program.

In a nutshell, I think the RoadTest program is a great thing. It provides an opportunity for virtually anyone to have a shot at trying something new, exciting and sharing that experience with everyone. It is sometimes a chance to get a hold of a product which you wouldn’t otherwise be able to afford, while other times, it’s a good motivator and reason to invest some time into something you wouldn’t go out and try on your own accord. In that regard, it’s absolutely wonderful and the barrier of entry is very low (for most, but not all RoadTests). I have no qualms about the selection process (and you could probably say that for someone that has been selected frequently, they probably wouldn’t), but I do genuinely think that at every turn of the road, rscasny has been working really hard to improve things but satisfying everyone is probably an impossibility.

But it’s also worth bearing in mind that the RoadTest program is a marketing exercise too. While you provide your time and expertise, element14 is receiving your high-quality content and the sponsor is receiving your honest opinion, feedback and hopefully a nice review they can use in future marketing. I make no bones about it – I am fully aware of this and it is something to bear in mind. Because of the multiple parties involved and their various obligations, the choice of suitable RoadTester may be influenced by the application in ways which may be quite subtle. For example, perhaps a less-scrupulous company would like a RoadTester which focuses more on feature X rather than comparison with competitors. While element14 does shortlist and review applicants, as the sponsor has the option to choose a RoadTester directly, I can’t deny the possibility that this may be happening. So, while you might have a great application, your non-selection may not reflect a lack of skill or comprehensive test planning, just merely that you don’t fit into the narrative they may be trying to build, but this is an assumption on my part.

Of course, if you can turn this around and make the best of the situation, you and the manufacturer can stand to benefit greatly from the experience. I’ve found the RoadTest program an opportunity to learn more about things through reading through manuals, trying out practical experiments and pushing towards getting results with a two-month deadline looming over your back. This process has at times been an arduous one, as things virtually never go to plan for reasons that could not be foreseen at the outset. The reward of overcoming all of this is the knowledge that you gain and the opportunity to relay your findings to interested members of the public. Best of all, if you test a product thoroughly and you do find issues or points for improvement, it makes a good reason to contact the manufacturer and get in touch with the more technical side of their operations to hopefully improve the product for current and prospective owners of the product (e.g. through a hardware revision, firmware update or new version of documentation). In this case, you’ve literally become “part” of the product improvement process.

Some people have complained about the number of hours it takes to do a RoadTest review. In that discussion, I considered the question and came up with this answer:

I would have to say that it does depend on the product somewhat, but usually it is going to be more than 50 hours at least gauging by my most recent RoadTests. Even when I started and RoadTests were not quite as well planned, I don't think I would have spent less than 40 hours. Depending on how you count the hours, it can sometimes exceed it quite substantially especially for big ticket items and where things don't go to plan and the manufacturer has to get involved.

The breakdown is probably closer to the following for me:

• • Research Prior to Application and Writing Application - 1-3 hours
  • Performing Market Survey, Reading Documentation, Download/Install Software and Learning Tools - 5-25 hours (more for development kits, less for instruments, depends on the documentation available)
  • Documenting Unboxing and Initial Set-Up - 1-3 hours including processing of photos. This is usually a quick but exciting part of the RoadTest.
  • Running Experiments - Usually 48+ hours, although some products (e.g. sensors, instruments, power converters) I have automated the testing process so the equipment needs minimal intervention while it runs 24/7 over a period of several weeks to a bit over a month (e.g. my Omron Environmental Sensors RoadTest and my Vishay SiC461 are those that may seem quick but testing can get quite involved). If I count only "contact" hours where I am either writing a script or analysing data, I'd say anywhere from 15-30 hours. This may seem long but part of the reason for this is that many times, the initial experiment doesn't go according to plan and is a "learning" experiment, with a follow-up experiment conducted that produces the actual data that is published. Further to this, it also includes the time necessary to grab photos and edit them, scribble some notes for the final report and plan the structure of the review. It also includes time for moving around equipment and building the test setup for testing (e.g. adapters).
  • Writing up a Report and Uploading - 5-10 hours total.

The total of the above excluding the unattended test time would be anywhere from 27 hours (at a minimum) to 71 hours. The unattended test hours could easily add 500 hours+ to some RoadTests (which for me, usually means an occasional middle-of-the-night wake-up to attend to a piece of equipment that is fallen over and beeping at me, or having to tolerate a lot of fan noise ... and pay for the energy bills).

I realise that I am perhaps in the minority in terms of my willingness to take testing to the extreme - but I find it is a duty that comes with the privilege of having a suite of test equipment and also find it to be an interesting way to sharpen my skills and learn something new. It's not always going to be enjoyable, and I agree, perhaps buying the equipment is cheaper in the end, but it provides its own motivation for me to actually explore something with a firm deadline and an audience to report to. A RoadTest is a work that requires commitment, both during the RoadTest and afterwards (e.g. to address follow-up comments and questions).

It is a fair number of hours, but I’d have to say that this is the “cost” of a RoadTest. If you only think about things from the “billable hours” perspective, then in many cases, RoadTests may not appeal so feel free to not apply for them. But I would argue that life in general is more than about making money.

When I first started out doing RoadTests, I really didn’t know what a RoadTest should look like. I knew it was a review and it should reflect some testing and practical use of a given item, but aside from that, it was really up to me to decide what to do, how to structure and present it in my application and in the final report. But just like everything else in life, as you do these things more often, you get better at them. Looking back at some of my earliest RoadTests (and especially applications), I sometimes cringe at the quality (or lack thereof) in some of my earliest attempts. But after doing a few reviews, I’ve got things down to a process which makes things easier for time management as well as delivery. After all, the fun is in playing with the equipment – writing up usually is much less interesting.

On the whole, RoadTests are fun and exciting not only because you’re often reviewing an item of interest that cost nothing to you – it can also be fun and exciting because you just don’t know what to expect and you’re trying your best given your limitations. We would often like things to “just work”, but the harsh reality is that everyone will have a different experience depending on how they go about using a product. That is why having several kits out for review from several different reviewers provides a bit of a different perspective and why I wholly support different reviewers performing the same test and coming up with their own independent results. Those who say “well the other reviewer did it” are just being lazy in my eyes. While some reviewers like to start groups for a given RoadTest and I do participate somewhat, I prefer reviewing independently without being coloured by the thoughts of others so as to be fair to the product unless there is a potential major issue.

Not all RoadTests go smoothly though. Sometimes you have to jury rig up an adapter to get something to plug together, design your makeshift test equipment to measure something, buy an accessory, make “secondary” measurements to try and infer what the unit is doing and more. It takes some problem-solving skill and ingenuity sometimes to come up with interesting, reliable findings. Sometimes these things cost money as well – but investing a small amount is usually no big deal for me since I’m already getting the product for free and I think it’s only fair to chip in a little for things that aren’t “included” in the package.

But there have been a few things that have been a little discouraging. For RoadTests where there are few applicants, the application window can extend once or even twice, pushing out the selection date meaning that if you were planning to do a RoadTest at a particular time (as you knew you’d have some spare time), the RoadTest may be delayed just enough such that if you were selected, the item would arrive too late and now you’d be stuck with an extra obligation. Other issues I have encountered mainly stem from items damaged in transit where the claims process didn’t end up anywhere and I was left with a damaged unit, or in a later case due to a flaw with the unit, I had to chip in the shipping cost to return the unit. While in both cases element14 said they would do something about it, after two or so e-mails, the trail goes cold. I can understand it … they don’t need more additional hassles – after all, they have plenty of applications to read through, especially for popular RoadTests where selection dates can be delayed significantly because of this and sponsor decisions. I like the fact that they do cover postage inclusive of import duty (which would otherwise make it impossible for me to afford some of the items) and the only case I had a mix-up, they shipped me the correct item in a second shipment and I ended up reviewing both.

Problem solving, learning something new is definitely an outcome of a RoadTest, but you will also find that communication and professionalism can go a long way as well. I’d say about 20% of my RoadTests have involved some form of manufacturer contact, usually to report deficiencies, errors in documentation, issues with operation or failures. This experience can vary quite widely depending on the company and the representatives you’ve been in contact with. I won’t name any companies or individuals, but I’ve had experiences ranging from no answer at all, to “won’t fix/can’t fix”, to “will be fixed in the next update” which may or may not arrive anytime soon. In some cases, it has involved being on an international long-distance call (at my cost), dealing with staff who were mid-promotion away from the department and having to explain the situation again to someone new, or dealing with representatives that were in the marketing/non-technical side that were constantly out-of-office on road-shows and the likes. I’ve also dealt in some cases with marketing who were obsessive with filtering all communications between myself and their technical team, corrupting the message and removing detail along the way. Frustrating as that can be, this seems to be part-and-parcel of how many companies operate and the experience is just yet another part of the complex rollercoaster that a RoadTest can be. Sometimes, being helpful can also cost more than just putting out a bad review – but I try not to let it get to that point by giving them every chance to make things right and being accommodating. Just recently, I’ve basically “re-reviewed” an item because of just this – it took a long time to resolve, but if they were willing to put in the effort to resolve the issue, I’d be willing to chip in the cost of shipping and a whole block of hours to re-test the unit and write up my findings.

The satisfaction of completing a RoadTest is something that I really enjoy. When my two months are up and I’ve uploaded almost 100 images, several blog posts and the better part of tens-of-thousands of words, I am happy that my experience has been thoroughly documented for the entertainment, enjoyment, education and amusement of others. Just getting it out there on time means that I have succeeded, that I get to live another day and that I can apply for another RoadTest and start the whole process all over again. But honestly, it’s not over just because the review is delivered. The first day or two after it is published is a critical time, as questions and feedback may come through the comments and providing an expert answer is part of the “service” that comes with the review. I pride myself in being a product “expert” after the completion of a review, so I tend to discharge my duty to answer them even if it means running a special experiment just to answer their curiosity. More than that, sometimes a product may change materially after the RoadTest is complete due to software updates. While it’s not required, sometimes I will go above and beyond to post a short update posting to detail some of these changes to keep things current.

I hope that you can agree that I take this seriously – just as I do for most things that I publish. As a result, sometimes it can be a little disappointing to see that I’ve been “pipped to the post” by someone else who published a half-written review “to be updated” later and have snagged the views. Or perhaps someone else who has published a dissenting review that might not be entirely representative of the product, but since viewers only have enough “mindshare” to read just one or two reviews, they never get around to mine. Or perhaps someone writes a very glowing review that glosses over some deficiencies and that is shared widely on social media whereas my comprehensive review sits with a lot less views. It is disappointing to see low engagement on a piece of work that I’ve put a lot of effort into – but I guess I’ve come to expect this. Not everyone has the mental fortitude to digest a wall of text, not everyone has the inclination to provide feedback or a “like”. But that’s fine. It’s the same amount of effort from my side … just a little less feel-good-factor in return I suppose.

This is perhaps where I get a little annoyed at some of the reviews I do find on the RoadTest platform. Perhaps I am prejudiced by my excessive level of detail and pedantry, but some reviews I have come across barely scrape through what I would consider to be acceptable. Without knowing their application, I’d still wager to say that a three-paragraph summary mostly stolen from the blurb for the product with no images doesn’t really count. Or not delivering a review at all. Or delivering half a review late with promises to bring more postings later which never arrive. This annoys me quite a bit since I may have applied, missed out and I know I would have put in more effort to make things right. So sometimes, I see those reviews and I don’t even bother to leave feedback and I definitely withhold giving them a like because I don’t think that should be encouraged. I suspect extreme cases are unlikely to be selected again in the future, so that may be their only taste of RoadTest joy.

But for the silver lining that is the RoadTest program, I’d have to admit that sometimes, the accusations that are thrown towards me with regards to my RoadTest preferences, motives, application and the implication that the program is somehow “biased” towards my success have been quite hurtful and have sown doubt in my mind whether I am deserving or not. It has, at times, made me feel quite upset and (dare I say) even unwelcome in the RoadTest program. I’ve probably passed on a handful of RoadTests thinking that “well, maybe if I win, it would only cause more problems for everyone.” It shouldn’t have to be like this. The RoadTest program is not intended to be a means to “get your lab setup for free”, but over the years, that seems to be what has happened (at least to me). I can understand the envy and potentially even jealousy that can arise as a result, but as a responsible owner, I’ve also put the equipment to good use in testing other equipment and products both for RoadTests and other reviews which I now see as part of my duty as a previous beneficiary. But the key thing is that this didn’t happen because I wanted it to happen – it happened because I was interested and passionate about test and measurement, I applied consistently to RoadTests of that category (and others) and I delivered my reviews to a consistent level of quality in a reliable manner. Furthermore, I take time to make my applications competitive and to my understanding, there has been no special advantage that any “Top Member” or previous reviewer gets beyond the recognition of their prior community engagement.

Outside of RoadTests

I’ve done my fair share of reviews for the element14 RoadTest program, but I do also operate my own personal blog at https://goughlui.com where I occasionally review items I bought or items which have been donated for review.

I think the element14 RoadTest program has a lot of benefits in this regard, since you basically apply to review a given item. If you’re running your own (small) site, you’re not likely to be approached by any big company with anything of interest. Likewise, if you try and “beg” for a review from some of these companies, it is often unappreciated and your lack of clout usually will doom you to failure before you even begin. But perhaps an even bigger risk is agreeing on terms for the review in terms of the disposition of the item, whether they get an opportunity to do an editorial review prior to publication and more. This could get quite messy, so having the element14 RoadTest program kind of shields you from having to deal with these issues.

The element14 RoadTest program has had an unexpected silver lining in the sense that at least in the case of one company, they have made direct contact with me outside of the RoadTest framework with offers of review items to which I have accepted. Some other companies have noticed my RoadTests and reviews on my own site and have approached me with offers, although this doesn’t always reach fruition as my views and interests may not always align with their marketing goals at the time, so sometimes these leads just fall dead. But good things may come to those who do good reviews … directly.

Of course, I’ve also received items from element14 which are not technically RoadTests but which I have taken the time to review like a RoadTest. These “off-road” tests include:

• Early-Access Review: Epishine Light Energy Harvesting Module Evaluation Kit
• Early-Access Review (Pt 2): Epishine Light Energy Harvesting Module Evaluation Kit
• Quick Review: Multicomp Pro MP700136 & MP700151 Wire Cutter/Crimper/Stripper Tools
• Quick Review: Multicomp Pro MP700130 12-in-1 & MP700124 11-in-1 Screwdrivers (Updated)
• Review: Tenma 21-2310 UK (AT980D) 80W Soldering Station
• Review: Multicomp Pro MP730026 3.75-digit BLE Digital Multimeter
• Review: Duratool D03288 Fume Absorber & Fan (220-240V/50Hz/23W)
• Off-RoadTest: Genuino 101 (and a first project)

There are also countless equipment reviews and follow-ups on my own personal site – the more recent major electronics related reviews include:

• Review, Teardown: HoldPeak HP-605C 6/40A AC/DC Clamp Adapter
• Launch-Day Review: Rohde & Schwarz NGM202 Power Supply (Overview)
• Rohde & Schwarz HMC8043 Power Supply Review – Overview
• Review: EEVBlog HVP-70 (70Mhz) & Micsig DP10013 (100Mhz) High Voltage Differential Probes
• Review: Weekend with the Asus Tinker Board
• Review: Uni-Trend UT210E Mini Clamp Meter
• Great AA Alkaline Battery Test – Pt 1: Battery Testing Fundamentals
• Great AA Alkaline Battery Test – Pt 2: The Contenders
• Great AA Alkaline Battery Test – Pt 3: The Results
• Great AA Alkaline Battery Test – Pt 4: Cell Physical Characteristics
• Great AA Alkaline Battery Test – Addendum: 25mA Constant Current Tests
• Review: Raspberry Pi 3 – More Pi, No Cables Attached?
• Review, Teardown: Keweisi KWS-V20 USB Tester
• Shootout: Four 433.92Mhz ASK/OOK Transmitter Modules Compared
• Teardown, Accuracy Testing: LP7663 Kill-a-Watt-clone Power Usage Meter
• Quick Review: Freetronics LeoStick (XC4266) Arduino Compatible Board
• Valentine’s Day: A Date with the Raspberry Pi 2 (Model B)
• Review: Agilent U1241B 4-Digit Handheld DMM & Accessories
• Investigation: How Accurate is that 3.5 digit Multimeter?
• Testing: Charger Doctor and USB Detector Accuracy
• Review, Teardown: Two Popular USB Charger Doctor/Detector/Current Meters
• Quick Review: Tenma SMD Rework Station Bundle
• Quick Review: Tenma Insulation Resistance Tester 72-9400
• Review: Raspberry Pi Model B+ & Multicomp Raspberry Pi B+ Case
• Quick Review: Agilent Technologies U1733C Handheld LCR Meter
• Review: Manson HCS-3102 1-36v/0-5A Switch-Mode Benchtop Power Supply

I think that you’ll see that to become a good RoadTest reviewer takes practice. I’ve been reviewing things over many years, in the process, refining my process.

RoadTest Tips

It’s been a long blog so far – if you’re still reading, thanks for sticking around. I think you might still be wondering if there’s a point to this blog. I suppose the point is that I want you to know that RoadTests are awesome, can be a lot of fun, are a lot of commitment and I’d like you to have a share in the bounty. You might think I just want to keep all of the goodies to myself – but frankly, I don’t have enough space in my room nor time on my hands to review every piece of good kit that comes by on the RoadTest program.

That’s why I’d like to share with you some of my tips (at least my perspective) to make the RoadTesting experience as positive as possible:

• Be somewhat selective, but not too selective – after all, you will have a better chance of winning if you apply to more RoadTests. Just don’t apply to RoadTests you don’t have an interest in completing.
• Do some research – this means reading product information and probably some documentation prior to making your application so you can speak to the strengths and capabilities of the product while also demonstrating your knowledge and understanding of the field.
• Be organised – keep things sorted and start collecting documentation early, especially if things change during the review (i.e. things are taken down) and you can refer to it quickly during testing. It would also be good to examine the documentation and software even prior to the arrival of the item if you are chosen.
• Be authentic – don’t guess what they might be looking for and just apply truthfully.
• Promise only what you can deliver – otherwise you will not deliver a good review which will probably reflect poorly on you and the community which could potentially make future RoadTests from the sponsor less likely.
• Budget your time – time management is necessary to make sure you deliver on time. Remember, it takes more time than you expect when things go wrong and writing up is usually going to take a while since it’s not as “fun”.
• Do not delay – once the package arrives, examine it for damage and start testing right away. Two months can run away from you quite quickly.
• Think before you do – you only have the one item and you really don’t want to break it before you’ve made your findings. As a result, having a proper test plan and sequence is important, as is making good reference to the documentation available to avoid accidents.
• Stay safe – some RoadTests may involve building things and use of potentially dangerous voltages, so it pays to be careful and be mindful of safety. It’s not worth paying the “ultimate price” for a RoadTest.
• Have a “Plan B” – things don’t always go to plan so perhaps you should think of what you would do if things don’t work out. Even if something has to be omitted because things don’t work out, it doesn’t mean you can’t at least try to proceed with the remainder of your proposal.
• Remain professional and impartial – we like reading RoadTest reviews because they’re honest, truthful and objective as much as possible. The point is not to deliver a positive review – it is to deliver an impartial review that we can trust. It is important to not get emotive and to illustrate your findings with data where possible. Maintaining a professional demeanour is important when dealing with representatives and comments in general.
• Don’t forget to document along the way – have that camera ready, take screenshots and do it frequently. You can always “cut” things out in the end, but if you miss that infrequent glitch, you will have a hard time making a convincing bug report.
• Author in a word processor – the element14 web editor seems to slow down for large documents, so it helps to author in a word processor first and copy/paste the text in. Then upload the images in the right places (as pasting the images directly can cause upload issues as well).
• Break up your writing – try and organise your writing into logical sections which can be read standalone and which flow from part to part to make navigating your review easier. This makes it possible to have an automated Table of Contents at the beginning of the review, assuming you’ve used the right heading style.
• Be comprehensive – I like to see the details as it makes experiments replicable and inspires confidence that you’ve actually tried something and aren’t just taking the manufacturers’ word for it.
• If there’s a problem, let someone know – this means contacting the RoadTest program organisers (e.g. rscasny and danzima or adding a comment asking for help from the community. There’s no point sitting there being stuck with something as time is ticking away.

I’m sure there will be some tips I’ve missed … if you have any comment on my experiences or tips to offer, feel free to leave a comment below.

Conclusion

The holiday season is a time to be grateful and as a major beneficiary of the element14 RoadTest program for the past nine years, I couldn’t be more grateful for the support of element14 and their staff in charge of the program including rscasny, danzima and former staff including Spanner Spencer and Christian DeFeo. It has been an awesome experience applying for 48 RoadTests, having been awarded around half of which I had applied for, learning a lot and amassing a lot of capabilities along the way. It is truly the gift that keeps on giving.

It’s such a good opportunity for some people (such as myself) who are willing to dedicate the time and effort it takes to deliver such a review. It is something that can be quite enlightening, rewarding and joyous. It is something I want to share with you all, despite what you might think of my motives.

In part, this very longwinded and uncensored post has been something I’ve been debating about sharing for a long time because I’ve been a RoadTester so long that I’ve seen almost everything – the good, the bad and the ugly. I’ve been on the receiving end of some comments which had been at times unpleasant, hurtful and made me feel undeserving of my wins which were all the product of hard work and dedication. Combined with the fact that sometimes reviews just don’t get up in terms of views or feedback, it can seem that all that effort is for naught.

As a result, I think I’ve laid it bare in the spirit of being open and transparent with the community – here’s my RoadTest score sheet and an example of my application, as well as my uncensored review of the RoadTest program as a whole, warts and all. It’s not always a walk in the park, but I think it’s still great nonetheless, otherwise I wouldn’t be applying for them over and over again. I hope that after this brutally honest, long reflection that I am still welcome to the RoadTest party … or have I just jinxed myself for the future. I suppose there’s only one way to find out … apply for more RoadTests!

Finally, to all the people who read my reviews, blogs and leave me comments, likes or ratings – I sincerely appreciate it. You probably don’t know how much it means to me to see a reaction on something I’ve put in a lot of effort into that is probably only going to be read by a handful of people in small “niches”. It reminds me that there’s a reason why I put in so much effort … but also, I completely understand if there just isn’t any response. I don’t expect people to read a wall of text about an item they’d never expect to use or own. It’s just the nature of the beast. But I hope this has somehow been helpful for you, even as a yard-stick by which to measure your own efforts and outcomes.