Should you have to pay to opt-in to a RoadTest ?

As a RoadTest veteran with 34 completed (and on-time) RoadTests since my first in 2013, I’ll offer my 2c, as the topic of people running away with equipment and/or producing sub-standard reviews seems to be a recurring one, albeit somewhat less often since rscasny  took the helm.

The RoadTest program is perhaps one of the biggest draws to element14 and I think the publicity does not shy away from the fact the program is “free” to participants. This draws in people of all levels and backgrounds, which is perhaps important given the wide variety of things that may be RoadTested. The choice of RoadTesters based on a comprehensive application process is intended to ensure the best applicants receive the opportunity and the strategy that @rscasny has of picking a mix of both known, experienced RoadTesters and new RoadTesters has been carefully refined over the years to ensure the pool of good RoadTesters can grow over time. To some extent, the Reliable RoadTester badge had been introduced as part of this as well and the insistence that RoadTesters self-identify if they have previously failed to deliver on-time and the clause that they can be excluded based on failure-to-deliver.

When I joined, the process was much more ad-hoc and applications varied wildly in quality and content. Delivery of reviews was very hit-and-miss, quality was much the same. Admittedly, some of my earlier reviews definitely don’t measure up to the quality of my latest pieces of work. I spent a lot of time writing a blog of my thoughts about the RoadTest program back in 2020 (of which I am extremely grateful for).

I think the first issue (which has previously been identified and discussed ad-nauseum) really boils down to attitude and value. Something given away for free is often valued less than one which is paid for. Advertising RoadTests as “free” may give the perception that it is without obligation, which is untrue. Likewise, it may create the perception that reviewers are merely volunteering their time to produce a review. I’ve always taken the approach of this being a professional engagement – goods in exchange for review in a fixed timeframe. Participants should really be put into the frame of this as a “serious job” delivering real-world insights rather than “volunteering their time”. Sometimes things do legitimately get in the way, other times things go awry, and I feel flexibility here is a good thing too, but something to be considered on a case-by-case basis.

If you would have asked me to pay money to review an item back in 2013 when I started, I would have said no. It’s not because I don’t want to pay, it’s that I would have never had the means to pay! As a student with no credit card … it’s just not a possibility. Similarly, the logistics of handling money back and forth, especially with currency conversions potentially happening (which means losses) is a drama and a half. Imagine getting charged a 3% fee + their conversion margin both ways! Yikes!

But if you ask me now … I can’t say that I’m entirely opposed to it provided the deposit is reasonable and we can avoid “losing” money in the process of the refund. I deliver my reviews on-time and to a high standard, and have nothing to lose in that instance, provided you’re willing to go the distance to do it. But as someone else has rightly pointed out – if it’s cheaper than buying the equipment, there’s nothing to stop them from running off with the unit anyway. I still think it is a legitimate concern that by doing so, you will be locking out people like the 2013-version of me … and making some difficult-to-recruit RoadTests even more difficult perhaps.

I think the release condition for the deposit is an issue as well. Some very poor reviews have been delivered just to fulfil the condition of delivering “a review”, but I would be disheartened to see a good piece of equipment go to a member that delivers a two-paragraph, all-text review with no care that demonstrates nothing more than what is on a datasheet. The arbiter of what is sufficient effort for two months may need to refer to the original proposal and agree it meets a certain amount of promised goals to be able to release the deposit. Perhaps proposals should be framed as key deliverables instead.

Because of the administrative overhead, perhaps it is good to restrict this to higher-value items. I wonder if this would be problematic from a taxation point of view as well. Perhaps those with greater than a given number of completed RoadTests could escape paying entirely – although this may be considered unfair by others, perhaps if they have delivered lower-value reviews a certain number of times, they can join the upper-bracket of “deposit-free” RoadTesters. I think it makes sense if deposits are taken, that they be taken on-shipment or selection, rather than on application, again for administrative reasons.

But in the end, I suppose this is just a difficult thing to enforce. Perhaps there is no 100% guarantee in life – even good people may fail from time to time as well. But I feel that it’s an interesting conversation cstanton as having money in the form of a deposit definitely does mean that the reviewer has something at-stake and an incentive to complete. But in return, I feel this may also impact the (perceived, or actual) impartiality of the reviews. If a user feels their money is at risk if they don’t deliver a satisfying (positive) review, then this is only going to jeopardise the process and the trust the readers have in the reviews that are posted.

Maybe for certain products, no deposit is necessary and instead, the licensing system can be exploited to achieve this function. I’ve had gear lent to me from T&M companies that come with time-locked licenses that will refuse to function once your lease period is over (or revert to basic functionality only). Perhaps for such capable equipment, they be shipped with time-locked licenses set to expire one month post-RoadTest. Providing they do complete a satisfactory review, the permanent license keys will be provided to those members, and if they don’t, then the gear is either rendered impaired/useless and then they have an incentive to return it to the manufacturer (or you) to get it re-licensed for another go. This won’t work for all equipment, but exchanging serial codes is probably easier than exchanging money for many members, and it doesn’t restrict functionality during the review period. Won’t stop people parting out the machine for spares … unfortunately …

But perhaps the trust-the-application model we have at the moment is still the simplest, from an administrative standpoint. If the losses are at an acceptable level, perhaps that’s just the “cost of doing business”. I can’t see any changes making things necessarily fairer for all, but then again, there were heated debates by disgruntled RoadTest applicants that were not selected as they felt they were being discriminated against. If we formalise the establishment of a “high tier” RoadTester, I fear we may be inviting this debate back into the fore and reigniting some very unpleasant discussions (as someone who was the target of a few members/former-members).

As far as I know, the companies may also have some say in this … perhaps different companies will accept different levels of risk in non-review. Or perhaps some companies will have the time to make an informed choice by looking at prior work and statistics in terms of members and their delivery track record …

- Gough

As a RoadTest veteran with 34 completed (and on-time) RoadTests since my first in 2013, I’ll offer my 2c, as the topic of people running away with equipment and/or producing sub-standard reviews seems to be a recurring one, albeit somewhat less often since rscasny  took the helm.

The RoadTest program is perhaps one of the biggest draws to element14 and I think the publicity does not shy away from the fact the program is “free” to participants. This draws in people of all levels and backgrounds, which is perhaps important given the wide variety of things that may be RoadTested. The choice of RoadTesters based on a comprehensive application process is intended to ensure the best applicants receive the opportunity and the strategy that @rscasny has of picking a mix of both known, experienced RoadTesters and new RoadTesters has been carefully refined over the years to ensure the pool of good RoadTesters can grow over time. To some extent, the Reliable RoadTester badge had been introduced as part of this as well and the insistence that RoadTesters self-identify if they have previously failed to deliver on-time and the clause that they can be excluded based on failure-to-deliver.

When I joined, the process was much more ad-hoc and applications varied wildly in quality and content. Delivery of reviews was very hit-and-miss, quality was much the same. Admittedly, some of my earlier reviews definitely don’t measure up to the quality of my latest pieces of work. I spent a lot of time writing a blog of my thoughts about the RoadTest program back in 2020 (of which I am extremely grateful for).

I think the first issue (which has previously been identified and discussed ad-nauseum) really boils down to attitude and value. Something given away for free is often valued less than one which is paid for. Advertising RoadTests as “free” may give the perception that it is without obligation, which is untrue. Likewise, it may create the perception that reviewers are merely volunteering their time to produce a review. I’ve always taken the approach of this being a professional engagement – goods in exchange for review in a fixed timeframe. Participants should really be put into the frame of this as a “serious job” delivering real-world insights rather than “volunteering their time”. Sometimes things do legitimately get in the way, other times things go awry, and I feel flexibility here is a good thing too, but something to be considered on a case-by-case basis.

If you would have asked me to pay money to review an item back in 2013 when I started, I would have said no. It’s not because I don’t want to pay, it’s that I would have never had the means to pay! As a student with no credit card … it’s just not a possibility. Similarly, the logistics of handling money back and forth, especially with currency conversions potentially happening (which means losses) is a drama and a half. Imagine getting charged a 3% fee + their conversion margin both ways! Yikes!

But if you ask me now … I can’t say that I’m entirely opposed to it provided the deposit is reasonable and we can avoid “losing” money in the process of the refund. I deliver my reviews on-time and to a high standard, and have nothing to lose in that instance, provided you’re willing to go the distance to do it. But as someone else has rightly pointed out – if it’s cheaper than buying the equipment, there’s nothing to stop them from running off with the unit anyway. I still think it is a legitimate concern that by doing so, you will be locking out people like the 2013-version of me … and making some difficult-to-recruit RoadTests even more difficult perhaps.

I think the release condition for the deposit is an issue as well. Some very poor reviews have been delivered just to fulfil the condition of delivering “a review”, but I would be disheartened to see a good piece of equipment go to a member that delivers a two-paragraph, all-text review with no care that demonstrates nothing more than what is on a datasheet. The arbiter of what is sufficient effort for two months may need to refer to the original proposal and agree it meets a certain amount of promised goals to be able to release the deposit. Perhaps proposals should be framed as key deliverables instead.

Because of the administrative overhead, perhaps it is good to restrict this to higher-value items. I wonder if this would be problematic from a taxation point of view as well. Perhaps those with greater than a given number of completed RoadTests could escape paying entirely – although this may be considered unfair by others, perhaps if they have delivered lower-value reviews a certain number of times, they can join the upper-bracket of “deposit-free” RoadTesters. I think it makes sense if deposits are taken, that they be taken on-shipment or selection, rather than on application, again for administrative reasons.

But in the end, I suppose this is just a difficult thing to enforce. Perhaps there is no 100% guarantee in life – even good people may fail from time to time as well. But I feel that it’s an interesting conversation cstanton as having money in the form of a deposit definitely does mean that the reviewer has something at-stake and an incentive to complete. But in return, I feel this may also impact the (perceived, or actual) impartiality of the reviews. If a user feels their money is at risk if they don’t deliver a satisfying (positive) review, then this is only going to jeopardise the process and the trust the readers have in the reviews that are posted.

Maybe for certain products, no deposit is necessary and instead, the licensing system can be exploited to achieve this function. I’ve had gear lent to me from T&M companies that come with time-locked licenses that will refuse to function once your lease period is over (or revert to basic functionality only). Perhaps for such capable equipment, they be shipped with time-locked licenses set to expire one month post-RoadTest. Providing they do complete a satisfactory review, the permanent license keys will be provided to those members, and if they don’t, then the gear is either rendered impaired/useless and then they have an incentive to return it to the manufacturer (or you) to get it re-licensed for another go. This won’t work for all equipment, but exchanging serial codes is probably easier than exchanging money for many members, and it doesn’t restrict functionality during the review period. Won’t stop people parting out the machine for spares … unfortunately …

But perhaps the trust-the-application model we have at the moment is still the simplest, from an administrative standpoint. If the losses are at an acceptable level, perhaps that’s just the “cost of doing business”. I can’t see any changes making things necessarily fairer for all, but then again, there were heated debates by disgruntled RoadTest applicants that were not selected as they felt they were being discriminated against. If we formalise the establishment of a “high tier” RoadTester, I fear we may be inviting this debate back into the fore and reigniting some very unpleasant discussions (as someone who was the target of a few members/former-members).

As far as I know, the companies may also have some say in this … perhaps different companies will accept different levels of risk in non-review. Or perhaps some companies will have the time to make an informed choice by looking at prior work and statistics in terms of members and their delivery track record …

- Gough