Infineon OPTIGA™ Trust M Board - Review

Table of contents

RoadTest: Infineon OPTIGA™ Trust M Board

Author: daemoninformatica

Creation date:

Evaluation Type: Development Boards & Tools

Did you receive all parts the manufacturer stated would be included in the package?: True

What other parts do you consider comparable to this product?: Typical newer more complex embedded controllers contain peripherals for encryption and verification. In some ways the NXP A71CH (Smart card chip) comes to mind..

What were the biggest problems encountered?: Most the problems were with documentation. Either outdated, incomplete or just plain missing. (Hence the low score for support materials) There was Demo software, but I have a lot of remarks on this in the review. As a result, any attempts to build PoC's with the product and supporting platform failed.

Detailed Review:


Before I start, I'd like to thank Infineon on offering this roadtest. The below review might feel harsh at some points, but is not meant as an attack. If Infineon is interested, I'm open to answering more questions and receiving feedback on how to avoid many of the pitfalls that I encountered. I acknowledge that, while the software may seem different than I'm used to, possibly smarter people than me have worked on this hard- and software.


I'd like to thank Element 14 to select me as a roadtest participant. I'm well aware of the fact I'm rther late with this review. Back in the time, at the end of the roadtest attempts (see below steps) I felt like I failed the review. I more or less forgot about it in the end until Daniel Emailed me. ^_^ Thankfully I'm one to take rather extensive notes and organize my step planning and result storage well. I've managed to compile up this review in a matter of a day..



Without further delay: My review:


It's often said that the 's' in IoT stands for security. I love to see how the realization dawns on people's faces when, after a couple of seconds it starts to dawn on them. Thinking up a concept and designing the basic principle of an IoT product doesn't have to be a complicated thing, but there are things beyond MVP functionality and design: A product connected to the internet (permanent or otherwise) should be (and is more often than not) secure.


Usually, if anything, 'security' or some form of encryption is added as an afterthought. Tacked onto a release. Communication over the internet to some sort of server / portal is done over HTTP, sometimes with some basic scrambling of data. This scrambling is then supposed to be protection against eaves-droppers. (I've heard anecdotes where a supplier of alarm systems boasted unbreakable encryption of data between alarm and a central monitoring station. It turns out that the data was XOR-ed with a single character.)


Scrambling (note that I refuse to use the word encryption here) is often done with a 'roll-your-own' algorithm that might or might not include a key. There are many warnings about using self-built forms of scrambling, but the shortest one is probably "don't".


Smarter people than us have thought longer and harder on ways to do this correctly and to this day open-source libraries (they wrote) are actively under attack and scrutiny. Weaknesses found and mitigated or fixed (or at least the community is warned). The very idea that one would be able to improvise some basic level of encryption that would stand up to scrutiny in the medium / long run is absurd (and dangerous).


So why not use these open source algorithms? If adding TLS is a matter of adding a library and compiling it in, why not?

Executing encryption as part of communication, as an often small component of an embedded application takes a lot of resources. A PC / server has more than enough resources to spare. But a controller is very limited. A relatively low speed processor, couple kilobytes of RAM, some Flash and that's it. (Obviously, I'm ignoring controller peripherals here, but more on that later.) There is simply not sufficient space to do it.

There are simplified versions of encryption, for which there are embedded libraries. It's a light, but valid form of encryption. Still takes resources though..


So why is this a problem?

These days, with the innovations made in IoT and surrounding methodology, IoT devices probably outnumber workstations and servers on the internet. They're so common, you'd be hard-pressed to find a flashlight without an internet connection. (Indeed, many devices used as flashlights have built-in 4G. But I digress...) With the increased integration of IoT in daily life, but good security sorely lacking, scandals and publications of leaked data are abound. And indeed, it's a simple matter to connect an IoT to a local router, man-in-the-middle the communication to eavesdrop on it and chew on the result to not only learn how to interpret telemetry from the device, but also commands and updates from the portal.


This is becoming such a problem that the EU (European Union) is starting to draft rules and laws to enforce protection and encryption in IoT communication. This is all well and fine, but will not suddenly make controllers more powerful. Encryption will have to come from something else.


What is Optiga Trust?


The Optiga Trust series is a series of security implementations in hardware.

The Optiga Trust M is a Crypto-chip. Essentially a co-processor if you will that the main controller can defer encryption tasks to while it's doing other things. The chip is a dedicated hardware implementaton of encryption and decryption algorithms and therefore, contrary to a typical general purpose controller can be very small and efficient. Its serial interface over i2c makes it simple to connect (if not implement) with very few control lines. The entire package (as supplied for the road-test) is several square millimetres.


Why does it exist?

The Optiga Trust M is a product that fills a niche in the market that is obscure at this point in time (Most developers and producers still don't really bother with solid encryption and protection) but will become important soon. Once laws and regulations come into play that enforce the necessity of encryption in IoT communication, products like this will become more important.


This chip should make adding encryption as easy as adding a lego brick to a castle. (Much like most component-based development is pictured. More on component based development later.) This chip can relatively easy be added to products similar to ESP's or other modules and SoC products.


What is Infineon?


Infineon is a microchip developer. They design and produce a wide range of chips in the industry. Including the Trust M.

The Roadtest:


On to the roadtest itself. The target to roadtest is the Optiga Trust M chip itself. To this end, Infineon supplied each roadtester not only with the chip, but also an ARM platform it apparently developed for evaluation and development of industrial IoT products. (IIoT) Out of the gate, I planned on a structured method of exploring the capabilities of the Trust M using the Connectivity Board, but with the realization that the objective of the roadtest was indeed the Crypto-chip itself. At the time of writing I've been long done with attempts to integrate both into my work and the result of my review will be a combination of both, along with some feedback in my conclusion...



How did I plan to use it?

Different types of Roadtests call for different approaches to the test-methodology. Tooling and equipment will often end up as 'daily drivers', or simple test setups will be designed around them.

Hardware like this roadtest, I feel most comfortable to go step by step and explore each offered feature either in turn, or combined with others. Usually I dig up all documentation about it (datasheets, summaries, other reviews and such), study it and from there continue with my own experiments. In this case, since this is literally a crypto-chip implementation for an IoT application, I planned on generating some 'measurement data' (temperature sensor, or maybe a pot-meter I could vary over time) and send that over in a secure way to a server setup, over a HTTPS REST connection. Even if I didn't succeed, the aim would result in insights on the product. And this it did....)





Before I do a roadtest like this, I aim to setup a set of steps to measure progress. This list of steps is not set in stone and more often grows and adapts to insights. This section of the roadtest, I will list the main steps, and then for each step the observed result and experience. In this case, this list of steps contains all planned steps, even the ones I didn't end up doing.... More on that in the end-conclusion.

Everything marked with '+' is a completed step.

Everything with a '-' is not executed.

Everything marked with a 'x' is a failed step. (Failed as in did not end up where I expected'.)


+ Step 1: Unboxing


+ Step 2: Experimentation platform.


+ Step 3: Getting started guide.


+ Step 4: Practical experience with Dave + Generate data to output / Facilitate feedback.


x Step 5: Basic Communication from XMC4800


x Step 6: Encrypt message from external source.


- Step 7: Decrypt message from external source.


- Step 8: Evaluate 'Shielded interface' of chip.


Each step:

Lets go through each step and see what I did, what happened and if applicable my conclusions.

Step 1: Unboxing

Lets start with the beginning. Upon receiving the product to review, the obligatory unboxing:

Nothing especially impressive here. The mail package is feather-light and contains one box (labelled Optiga Trust M evaluation Kit) and some cardboard packaging material. Everything is undamaged and, given the weight of the product and quality of the cardboard I deem the packaging more than adequate. You'd be able to toss the entire package off the roof of a shipping warehouse and it would survive unscathed.

The inner package box contains several plastic bags and a single piece of paper. Two of the bags contain micro USB cables. A third one contains the Infineon evaluation board. A fourth one, small, contains seperate parts to facilitate a mod one can do to daisy-chain the boards. I find this interesting, but haven't done it for several reasons.

  1. I don't have more than one board.
  2. I planned on testing the EtherCAT ports, and from what I gather, the mod would disable the original ports.


The main Infineon board contains several other add-on modules, pre-built and added.

  1. A Wifi ESP Click connected to the micro-bus.
  2. A Shield2Go adapter for Arduino (based) interfaces, connected to the Arduino form factor interface on the board.
  3. The Optiga Trust M 'Shield2Go' which is compatible with the Shield2Go adapter.


The ShieldToGo adapter and the Trust M module on top of that use a method of pin-mounting that appear to be solderless and depend on pin-friction to stay in place. I haven't encountered it before, but have heard of it. It's surprisingly robust. I suppose it saves on solder (environmentally friendly) and possibly allows (if the pins are in fact still removable) for more permanent mounting as daughterboard in other projects.


Last but not least: As a nice tough, the inner-box lid has a neat QR code sticker with a reference to a getting started page. The link is appreciated, but the QR code would imply that I'd open and read it on my phone or tablet. I ended up mailing the link to myself to open it on my desktop...


Step 2: Experimentation platform.


Using the board takes no further assembling, other than connecting USB cables provided and for easy access I use a simple 5-way powered USB 2.0 hub. The hub has a power-switch which allows me to power up and down the platform without yanking cables.

The evaluation board has 4 mounting holes, which I used to mount the platform on a small wooden box. This provides some stability, a possibility to keep everything related to it together in the box and avoids anything sliding under it from shorting out the board there. (Although in this case there's surprisingly little on the underside of the board to interfere with..)


Step 3: Getting started guide

The getting started guide is a reasonably comprehensive 24 page guide that aims to get you out of the starting blocks with this board. It enumerates and describes the different components and the connectors on the main board.

But lacks in several ways:

  1. From what I can tell, it's out of date compared to the getting started project on github. Chapter 2.2 mentions 'Installed Software Components'. I am not sure where they get the term 'installed' from, but the getting started project isn't installed, it's downloaded.I suppose it's nitpicking, but first not mentioning software sources in the previous paragraphs and then assume the reader has the software is not ideal. To remind you: this is supposed to be  the first PDF the reader sees...
  2. I noticed that the description in paragraph is not up to date with the actual output of the software. There's a menu of available tests, and the option to run them all.


Other points of interest installing the utilities, drivers and IDE are Ok. Really no surprises there, although I felt the download of Dave from the Infineon website was... 'cluncky'. Mostly in the way that confusion arose when I followed the links to download Dave, had to jump through unexpected hoops to actually make an account on the Infineon website and after that was finally done, the website was like 'Ok, thank you. So.... Why were you here?' Following the same link to download while logged in thankfully gave me the software.


The documentation mentioned though that the SEGGER utilities would be part of the installation. It was not... These were easily obtained from SEGGER's domain.


Installing the pre-built hex with the SEGGER J-Flash utility was trivial and worked on first try. So did the demo itself. The next step was to build the demo software from source. More on that in the next step...



Step 4: Practical experience with Dave + Generate data to output / Facilitate feedback.

At its core, Dave IDE is an Eclipse-based variant facilitating component-based development with the Dave CE (Code Engine) environment to allow the adding and use of 'Dave Apps'. There are ways to build projects in Dave with and without support of the CE (select at creation of a new project), but without, there is no support for the Apps. (Which are pretty much components).


Using the Eclipse environment for something like this is typically the way most suppliers go. Any hardware / software development initiative can take the base-Eclipse environment, add modules and modify its layout and workflows to suit their way of work. Especially with the progress Eclipse itself has made concerning stability and its API's to facilitate external modules and modifications this makes sense.


But it's also a double-edged sword: I am a reasonably experienced Eclipse user (daily-driver at work for most if not all tasks embedded.) but when you as a producer start adding modules of your own, you run the risk of changing the workflow and methods so much, the initial experience a developer has not only means less and less, but will frustrate the use of the new workflow.


Dave is the first time I used a component-based workflow to put programs together. (I am familiar with Node-Red, and am aware that Scratch does something similar, but I have low regard for both. Call me old-fashioned...)



Build from source:

This is where i ran into my first challenge: Downloading the project from github, It painlessly imported into Dave, which recognized it as a valid project. The method of importing it threw me off for a moment because traditionally one loads an existing project by referencing its root directory. Dave requires you to point to a Dave-directory (deeper into the project), which in turn contains the project data, it seems.


But then: The code appeared incomplete.

The demo project was there, but the actual 'Optiga' directory which the documentation referenced as the Optiga library implementation is there but empty. Stubbornly building the code, appeared to actually complete, but with warnings about missing header. Digging around on the github page, I noticed an actual Optiga project next to the getting-started. Cloning and studying this project, using the warning in Dave and some common sense, I managed to 'merge' the two projects (copying relevant content from the Optiga project to the demo project) and build the code successfully.


Further steps to write to target and run / debug in Dave went without problems.


I studied some more on how the demo project approached the demo-steps for different parts of the chip and because this seemed pretty straightforward I filed it away for now. As an aside: I like how the different tests are each implemented in their respective module in the project, causing no interference or overlap and give a reasonable understanding of the steps to take for a specific feature.


My own attempts and projects:

The evaluation board has several ways of network communication (EtherCat interfaces, Wifi and a SubD 9 pin CAN bus) but surprisingly few user interface elements. There are a couple of buttons and leds. Several of which have dedicated functions. There are some general purpose ones though and I aimed for those to get an understanding on how to build my own I/O project.


The Dave Quick-start (bundled with the Dave installation) has a useful intro on how to use the Code Engine and accompanying 'Apps'. Apps in Dave are a way to implement often used functions and components in a program such as I/O, conversion and configuration steps for example for clocks and other controller peripherals. Even more high-end components like FreeRTOS are simply select-able in there and can then be configured in a dedicated CE interface.


The idea behind this is not a bad one, but took some getting used to and I had to start over with a first project several times because apparently I 'misunderstood'. (Probably fixable, but starting over was faster...)


Installing a module from the 'App library' downloads a template into the project which in turn contains the definition of settings, dependencies and a template of code it will generate upon triggering an update. A GUI allows you to browse the different components and dependencies of the App and modify or configure them as needed. Different apps can be dependent on same or similar underlying components and peripherals and this is done pretty well.

After configuration, code is generated with a single press of a button. The result is then available in a 'Dave_generated' directory. Part of this result is a Dave main module, which takes care of all initialization of the other modules, based on configuration made. A new Dave CE project contains a call in the main to this initialization and all relevant functions from these included modules are available immediately without further need to include in the main module, since Dave is already included.


Initially I had some troubles figuring out the heads and tails of the Apps and their dependencies and components but this went away very quickly when I found out that each App / module configuration GUI has a button in the bottom right part of the panel to open a '.chm' windows help-file. These files are very educational and helped a lot.


My own Basic I/O project:


  1. Get a feeling of component-based programming in Dave
  2. Build a way to monitor what the heck my program is doing
  3. Generate actual data that I can control to send to an 'other side' (simple REST server, I'd cook up in Python) 
  4. An interface to the internet over the WiFi module. After all, what is it otherwise there for?



I'd build a nice little project that would contain the following:

- Serial I/O: A virtual serial port I hoped to 'borrow' from the demo project or somehow figure out myself how to build.


- Add a Led and Variable resistor to the Arduino formfactor connector for input and output. (Aware that the Shield2Go is in that socket for the chip, but it used hardly any pins, and had a lot of space left for expansion.)


- Implement some support for the ESP32 Click.



And here I quickly started running into other problems: I used the CE setup to be able to use components and this allowed me after a lot of doing to, for example implement an interrupt handler for a user-button. But this was really something to get used to. This did however lessen the learning curve for using the ADC to read the resistor. In the end I had input from both the button and pot, and a blinking led to boot.

The left picture above is a simple hardware setup with the components mentioned. The right picture is a Dave Configuration to use both a led and a button.


The nice thing about configuring most of the application in UI's, is that one can then use the button for example to switch off the PWM device:


 * @brief toggle_pwm_state() - Update PWM generator state (start / stop), based on previous state.
 * */
void toggle_pwm_state()

    pwm_is_running = !pwm_is_running;


PWM and MyLED are simply defined and named parts.


I never actually got a COM port working on my project. I got as far as sending serial data out over the SEGGER-RTT data terminal after I figured out how to convince a specific Dave App to do this.


Next Step: time to start talking to the outside world of internet...



Step 5: Basic Communication from XMC4800


Yea.. And this is where it all went sour... The above steps were educational, but frustrating. Costing a lot of time and energy, for little gain in the end. Already spent a lot of patience, I first tried to get the WiFi module working in a seperate project.


Naturally the demo project for the Trust M didn't have examples or references to this module. That makes sense: It's a demo project for the chip, not the evaluation board. But digging around on the net, I found virtually no usable material on this module at all! After a while I learned that the module would contain some sort of AT-Parser and that's good because as I have a lot of experience with broadband modems (which have AT instruction sets themselves), this meant that surely there should be some library that simply takes the reference to a serial UART and wraps AT commands with callable functions and return values.


The good news is that I actually found 3 projects that implemented something that should be able to talk to the AT parser.


  1. A project from Mikroe itself, that contains 'implementations' for several different architectures, but no resources on how to use them, what their dependencies are or examples in existing projects.
  2. A github project with mostly the same effect: I found a set of pages that felt a lot like it was a workshop on how to connect this board to an Azure cloud service to communicate securely. The project was overkill and involved a lot of details I didn't understand at the time (nor had the patience to figure out), but contained the implementation for the Wifi Click. Code, and some bit of information (There was actually a Readme in the Wifi module directory) but this did not contain any solid pointers on how to actually use this.


And claiming that the project is clean of dependencies (should be include-able in any other project with no issues) and then it turns out that it depends on a header file (cmsis_os.h) which after research is apparently apart of FreeRTOS, is impolite to say the least.

Experimenting with attempts to compile this module in a project, I tossed in FreeRTOS from the Apps library but this apparently didn't come with a cmsis_os.h file. (I probably overfsimplified this) Browsing the web I did come across some information abóut this file (what is it for: Apparently RTOS implementations depend on this file to configure methods and RTOS mechanisms) and some ready-made header definitions for architectures and controllers. I eventually stumbled upon one for the XMC4800 and copied it back to my project.


Then, after a lot of messing around with library paths (relative and otherwise) and attempts to initialize communication to the thing, I realized that either I was barking up the wrong tree, or the tree was dead and I was kicking it with no result.


I did find another interesting thing: A project to connect the evaluation board over Wifi to a Raspberry pi. It was a Dave project and for a moment I was re-energized. I ended this attempt when trying to read the project into Dave, and Dave wouldn't recognize the project as a valid Dave project.



This step, I did learn a lot about how the Apps were installed and used. How they interconnect and a lot about how to leverage them in my applications. In my opinion though: There is frustratingly little and fragmented information about them online. Seemingly even from Infineon. One of the things that surprised me the most is that there are no ready-made components for seemingly the most important / relevant parts of the evaluation board they boast: The ESP Click and the Optiga M. (In passing I did find references to an EtherCat interface and a different type of Optiga product, but didn't pursue it) No documentation I found contained any references to sources, libraries or drivers for the ESP.



Step 6: Encrypt message from external source.


Realizing I lost a lot of time in my attempts to get WiFi working, I figured that at least I could try and build my own project containing the Optiga library and get that part working. I could dump the result to terminal and 'prove' at least that part.


Optimistically I started a new seperate project to include the existing Optiga directory and work from there. I would be able to figure out initialization from the getting started demo and run my own messages through the chip instead of the hard-coded message in the demo.


All too soon, during build-attempts I ran into some dependency error concerning a header file named 'zephyr.h'. This was indeed not a file in my project, but in fairness, neither did it seem to be in the demo project.


It was at this point that in a last-ditch effort I returned to the Readme file of the Optiga project for guidance and behold: There was a reference to a 'Zephyr OS driver', including a link to github for a driver with reference to the Optiga Trust M! Too bad the link 404's on Github. The link is dead, and with that my drive to continue.



Steps planned but not taken in the Roadtest:


Step 7: Decrypt message from external source.

Essentially read back the response of a REST request and display it. (Or control for example the led attached. )



Step 8: Evaluate 'Shielded interface' of chip.

The chip boasts a 'shielded interface'. In my trawls through the documentation I found little information about it besides the fact it's easily enabled or disabled. It's an encrypted connection over i2c and I imagine this is useful to avoid probing the data interface between controller and chip when an attacker would have 'hard-access' to a device. I planned to use a logic analyser (either borrowing one or dusting off my logic shield for the Pi I have kicking around if it's fast enough for i2c) and see what the difference between the two is.

Téchnically I could have done that with the demo project running. But I completely ran out of patience.



Conclusion and feedback on the product and roadtest:

There is no doubt that the Optiga Trust M as such is a useful product. I recognize the reason IoT is vulnerable and how this product fills a niche in the market that is (and soon will be way more) important and still often overlooked.

The roadtest as I initially laid it out and tried to continue on was a frustrating one though and I personally think this is because of a severe shortage of information.


  • The Demo project on github itself was incomplete (and I managed to fix it with the contents of a second project on Infineon's github, but this was not documented and not obvious)


  • The Dave App store was somewhat useful in the end, but did not conform to a standard way of working any developer not already familiar with Dave Apps. The attached PWM example was educational and useful, but very rudimentary.


  • I did expect (and not find) Dave Apps for components on the board.



When setting out to roadtest the Optiga Trust M, I resolved to review this chip, rather than the rest of the platform. After all, the platform was mostly offered to facilitate the test. In the end my review ended up to be mostly about the environment in which I tríed to use the chip, but the platform was lacking support.


One thing I did notice:

The demo project was apparently not set up as a valid Dave CE project. (No support for apps or GUI configuration) However, it did contain a Dave_generated directory, which contains instances of modules based on Dave Apps. I gather that at some point, code was generated and then copy-pasted into this project to get it to work. That's one way to do it but there's a problem: This makes the demo-project effectively unusable as a learning reference. This, together with some very weird 'virtual directory' structure in the Dave project overview tree made it impossible to reference the demo project to learn how to set up my own Trust M project.

It was at that point I gave up.


I do realize that it might have helped to reach out earlier, but part of me reasoned that this would defeat the purpose of the roadtest. The purpose of the test would not be to prove it's possible to encrypt and decrypt data. The demo already does that. It would be to prove an outsider to Infineon is capable of harnessing the product and integrate it.



Epilogue and personal words.

I consider myself a reasonably competent embedded developer. I chose to participate in this roadtest, because at some past point in my career (I was an IoT / Embedded software consultant) I meant to establish myself as the company authority on secure IoT. (The EU just started making noise about the idea of regulations and at the time I was encouraged to plan ahead what I wanted my career to shape like.) Products like these facilitate this very thing and I mean to learn as much as possible about them. (There was an amount of irony when a year later the IoT branch of the company was removed.)


But this has no doubt been my most frustrating roadtest in the Element 14 community to date. In the end doubting my sanity, level and abilities.


I feel that Infineon has got the right idea for the products. Even newer controllers have dedicated peripherals for example to calculate AES and other crypto tools, but this still also takes compute cycles.


But it needs to work on support and documentation.

  • The crux of your RoadTest Review and a particular point of difficulty for many RoadTest Review is poor resources to support the product.


    All to often RoadTesters are forced to create an alternate testing procedure than what was pitched in their application because the supposedly documentation support and resources are "404 not found".


    I have done RoadTests were the quick start guide didn't work. Frustration is a mild word to use when the path suggested doesn't even exist.


    I rather enjoyed your commentary on those individuals that roll paper on a stick to write a message and believe they have a workable security solution.

  • Thanks,


    I must admit: In the end, I pretty much had a rage-quit, a virtual table-flip and done with it.

    Compiling my notes and writing out what happened was actually quite cathartic. ^_^


    Reading the other reviews (I noticed they were there before I posted mine, but didn't read them yet so the content wouldn't influence my own) I notice the overarching theme: The product is impressive and promising, but there is not enough documentation and support for it. Lots of fragmented and incomplete data, but there's a difference between the data being available and being able to find it.


    I would imagine that if you offer a platform with some very specific components, you'd offer clear and simple libraries or at least instructions on how to use them. I went up and down every available PDF and page to find for example support for the ESP Click. In the end, I found code for it, but no instructions on how to incorporate it in my own projects. I'm used to copying in modules or directories and  following some pointers / instructions on what to call to initialize and then control.

    The ESP code I found looked professional, but way overengineered.


    Same for the Trust-M module that's actually on the Infineon GitHub. It works, because the demo proves so. But as a low-level embedded engineer, I'm seeking a module / library I can feed a reference to an i2c port and then call wrapper functions. I don't need anything that actually depends on some embedded OS, it's a i2c slave chip!


    Ironically, I feel that the Dave App environment could really have helped here. Had they wrapped the entire shebang into some App, they could have added all the complexity they wanted to. As long as it wouldn't interfere with the rest of a project. Provide some interface and a few (well defined and documented) callbacks, using the chip and / or WiFi-module would become a breeze.

  • Nice honest review.


    I have had my share of obtuse products and missing documentation from vendors and they do themselves no favor by putting such products out into the community.

    I may not remember all of the great products, but I do remember the bad ones so I can warn others not to waste their time.