The third party product testing space performs an important mission within the industry - they keep the vendors honest. That mission is why I was so fired up when I took a job at Neohapsis in 2007 running their product testing lab. And, while I left Neohapsis a few months ago, I'm still fired up about product testing and the important role that a truly independent third party can bring to this industry.
The flip-side of that emotional intensity is that I get rather upset whenever I hear rumors about someone abusing their stance as an unbiased third party. And I recently heard something that made me ill.
I had a source approach me about a product testing firm who he suggested I blog about. This "independent testing firm" apparently does the most blatantly unethical thing I've ever heard in that industry:
They write the results that their testing will discover IN THE CONTRACT with the vendor who is requesting the testing.
It's one thing to have a "wink-wink, nudge-nudge" sort of relationship with a vendor. It's another to tilt the test criteria slightly, or even to accept the vendor's claims as solid assumptions during your testing. Both of those annoy me, but they're the kinds of things that go on in most organizations (though they did not happen under my watch at Neo... I'm too much of an idealist and a pain in the ass).
But when you have the outcome written in to the contract? In my mind, that's fraud if you're going around pretending to be an unbiased third party.
I couldn't believe that anyone would truly stoop to this level. And, since my source had left the company where he had worked with the testing lab, he couldn't get me a copy of the contract.
So, before I went on-record and wrote this one out, I wanted a bit more proof. I asked my amazing team of Indian VAs to check it out for me and provide me confirmation - not the information, just confirmation. Just get one of the contracts and tell me if the results are in there. Search, investigate, and interview customers of the lab to attempt to prove it one way or another.
The response to that investigation?
A cease and desist letter to my VAs from the test lab. (Which is why I'm not posting the lab's name... if I'm too busy to blog, I'm way too busy to bother with lawyers).
So, I'll say it this way - if you're reading a report about a product from a third-party lab that claims to be unbiased, take it with a grain of salt. Especially if the report came to you from the product vendor's sales/marketing team.
Or you can just ask to see the vendor's contract with the testing lab, and see if you get handed a cease and desist letter.