Poll: How Long Does It Take You to Complete a Roadtest and Write the Review?

This is my line of work and wanted to put some estimates and summarise them in a formula:

• (A) Pre-application stage:
  • (1) Research: 8 hours
    • This involves reading the about the roadtest, the products and the fields involved. This also involves, spare time think in different roadtests scenarios, new projects or how it fit to current needs.
  • (2) Roadtest application: 2 hour
    • Just filling the form and sketching a plan.
• (B) Roadtest approval stage:
  • (1) Plan adjustment: 2 hours
    • Once you are selected, you will refine your plan, requirements, scope and set a schedule/steps to follow
  • (2) Learning curve: 32 hours
    • This includes getting to know the device, the tools, the ecosystem and building up the required knowledge.
• (C) RoadTest Execution:
  • (1) Building and testing iterations: 24 hours
    • This starts with the unboxing, physical inspection, "hello world"-like application and complete the steps defined in the plan.
  • (2) Drafting and Content generation: 8 hours
    • You take notes of what you are doing, capture pictures, record videos while doing it.  Or perhaps you make it work first and then re-do it again for media and writing purposes.
• (D) Post writing and edition:
  • (1) Merge content, media, links, repositories and resources: 8 hours
    • Based on the quality of your draft you need to proofread it. In the case of media, edit pictures and videos, discard or retake some.
  • (2) Putting all together in the RoadTest form including galleries and media upload: 2 hours
    • In my opinion, the editor form of the Roadtest is quite cumbersome for uploading media and doing formatting. What I do is to write the review in an external document and just copy-paste at the end.

This is the formula I got:

total_time = (ExpT * ExpF) * (Tb2 + Tc1) + ExpF * (Ta1 + Tb1) + Ta2 + Tc2 + Td =

                    (ExpT * ExpF) * (60) + ExpF * (10) + 20

ExpT and ExpF = Experience with the tools or in the Field:

0.3 = Had worked recently with the tools/product - Have recent experience in the topic

0.5 = Had worked before with the tools/product - Have previous experience in the topic
8 = Had worked with similar tools - Working currently in the field

1 = it is a new tool or family of products - It's a field I have explored

2 = It is a new technology for me - It's a field I wanted to explore but I haven't started yet

4 = What is that device? - What are you talking about?

Based on that, just filling the application will take at least 4.5 hours if you have relevant experience. In case of being selected 30 hours will be spent, 20 of them for content generation, edition and writing.

Please take your time to compare it with your recent RoadTest and let me know if that fits the data

I completely depends upon what you are testing.

Some items take only a few hours.

Others may involved a hundred hours in reading, setting you the test, taking the data and then putting it all together in a coherent test report.

DAB

Good points Gough Lui. I don't think of road tests as taking too much time, just pointing out that for some of us it does take a lot of time. It is probably a good thing for members to realize they need to be prepared for that kind of commitment if they want to be successful, although I wonder if it will discourage members from applying. The danger is that the bar keeps getting higher until only a few members are willing to expend the effort. I don't know the answer to this dilemma - maybe when selecting testers, 60% are chosen from the pile of proposals that imply 50 hour reviews and 40% are chosen from the pile that imply some lesser number of hours. I think this sort of happens now when trying to  get new testers involved.

If there is only hardware or a tool to evaluate, then it doesn't take too much time.  If there is software to install or new toolchains to learn, then it will take much longer.

I don't think I've spent 50 hours on a road test, but certainly have on a design challenge or on the Path to Programmable training module.

If I make a 3d print of an enclosure for a part like the Vishay Buck Regulator or MiniZed, that takes several hours to design, CAD, and print.

If the components are new and interesting (like the Omron sensors) I can spend hours just "playing" with them, trying different software and such.

I try to document and save pictures while I'm working with the product, then put it all together and write the review when I think I've gone as far as I can.

Sometimes other projects take priority and I have to put a road test "on hold" for a week or two (or three). i.e Eaton breaker testing.

There are a few road test products I'd like to try, but just don't have the time to do, some are way too out-of-my-comfort-zone, like most of the FPGA and AI stuff.

Overall, the road test program is one of the best ways to see and use new tech, and I'm glad to be in the group.

Scott

Good question. Comments above made me understand better on how people work on RoadTests.

Earlier, I was involved in one RoadTest Taiyo Yuden Bluetooth LE EVK - Review and marked 40hrs. It may highly depend on what one is testing and persons RoadTest goal. For me it was:

• Researching a roadtest - 4hrs to check docs, literature, product features, applications etc. (did these when the kit was on air)
• Learning Tools - 6hrs to get the software, IDE's, configure tool, follow few tutorials etc. (this is an important aspect and need to revisit again and again while in further steps)
• Testing/Building - 20hrs to check all functionality, fully explore features, capture videos/photos, building some projects etc.
• Writing the Review - 5hrs to write blog, capture few more images, edit visuals, review etc. (this is where all above hours finally go and its important to spend some more time to present)

All in all, I think typical RoadTesting would take around 50hrs and maybe more if done bit by bit over several days.

Gough Lui that's a good breakdown. I agree, the absolute minimum time is around 30 hours to download software, read documentation, complete some given demos, write code, test own project and then write a brief report... and that is if you are familiar with the system or a similar product. If you are not familiar with the kit, or the kit is more complex, then you will need to familiarise yourself with the kit first. This takes much longer especially if problems or misunderstandings are encountered, which should be expected, and you are also documenting as you go along.

However, integrated within the road testing process is CPD (continual professional development) or personal learning/training. This is personal time to develop and expand ones skill set.

dougw - I agree, spending more time is perhaps a statistic that isn't something to be proud of as such, but somehow it has become the norm that is required to turn out a good review. I've always approached it with the idea of it being a challenge of "How much can I exercise and demonstrate the equipment given a two-month timeframe of "spare" time?" I usually promise quite a list of things, but they are things that I do deliver upon and usually just-in with the timeframe.

However, I suppose spending more time is only natural if we are to turn out a good review but also do a thorough test that might uncover any issues especially in very specific application scenarios, and to learn about those little niggles that only those people who have essentially "owned" the equipment would come to realise. If we didn't spend the time, I would fear our reviews would be less valuable and perhaps more in-line with what you might find on other sites where reviews are merely thinly-veiled PR releases (or blurbs in datasheets) rehashed. In return, we can offer a depth and specificity that is often lacking in other reviews and perhaps even answer one-on-one questions from interested members.

It may well serve as a warning to those who intend to RoadTest that it is a commitment, but is it a warning for us that we're putting in too much effort? Perhaps. But I don't see myself changing this anytime soon as I see it as my duty and interest to get to the bottom of these things, as I have a genuine interest in comparing, testing, trying and exploring. To put out a review that is half-baked would probably cost me more in lost sleep .

That being said, one of the things you don't see me doing much of is video. I find the process of making things presentable on video and editing to take a LOT of time, along with bandwidth, which doesn't come easily for me. So I definitely appreciate that things would take a lot of time for you especially when publishing so many videos and actually designing items to go along with it (not to mention the planning and delays which might happen due to manufacturing process). Very often when we look at the final review, it misses out on the effort and process that goes into making it ... which is a bit of a shame.

I wonder if the responses so far are what rscasny expects?

- Gough

Hi Randall,

The time to complete a roadtest review are depending on many variables.

Some depending on personal skills and knowledge.

Some depending on the product itself and manufacturers.

Larger and more complicated products take more time then smaller and less complicated products.

But when you look at the 60days that would be long enough to complete it. For those that can't meet the time slot I think they over played their hand on the roadtest especially on the simpler products. When looking at 60 days it results in atleast 8weekend that adds up to 16days to roadtest the products as we asume people will do this in their spare time.

It takes significant amount of time for me.

Here is my typical process:

• Researching a roadtest
  • Researching subject, analyzing potential value, viability of the technology/product, competition

• • Evaluation potential applications
  • Applying to the road test
  • Plan the project and tests
  • Define a high level solution

• Learning Tools

• • Read documentation, blogs, forums
  • Revise design
  • Install tools, learn how to use them, troubleshoot issues, get support from community.

• Testing/Building

• • Implement design
  • Troubleshoot issues
  • Perform tests
  • Collect test data
  • Analyze test data

• Writing the Review

• • Writing outline
  • Formatting materials
  • Write a blog

Some of these steps require several iterations.

I don't keep track of hours on projects and don't think it is an impressive statistic to be proud of.

(I think it is impressive if someone whips off an great road test in no time flat)

I looked at my road test proposals to see how many hours they would seem to imply by the promises made.

Then I looked at roughly how many hours I think they actually took.

I tend to have a high success rate with road test proposals, probably because I make lots of promises - which means they imply lots of hours to deliver on the promises.

I was surprised to realize that I actually spend a lot more time than my proposals would suggest.

It would seem the secret for me to win at the proposal phase is to promise a lot and the secret to producing a decent road test review is to spend even more time and effort than expected.

My road tests take a lot of time because they usually involve designing applications that use the gear under test. In the last 11 road tests I have designed 10 PCBs and many of them have 3D printed parts. They also include between 4 and 21 videos in each road test. I don't enjoy admitting it usually takes more than 50 hours.

These stats suggest that for someone like me, it takes a large effort to be successful in the road test program.

The time commitment is definitely a factor in which road tests I apply for, but it I don't consider it to be a governing factor.

The big factor is just how interesting the road test item is.