Go to UPA home page The U P A voice
August 2007 Contents

UPA Job Bank

UPA 2007


The Achilles Heel of Product Design Competitions and the Fair Judging Solution

By Rich Buttiglieri

Rich is a Usability Consultant at the Bentley College Design & Usability Center. With more than 19 years of experience in software and hardware product design and evaluation, Richconducts user-entered design studies for mass markets products as well as highly specialized verticals.He specializesin usability of consumer electronics.

“Top 10” lists are everywhere. One can’t pick up a periodical or visit a website without encountering a summary list on some topic, be it people, products, designs, or travel tips.  In the hands of the media the succinct novelty of a “Top 10” list is an ideal vehicle for focusing consumers’ shopping, identifying trends, and drawing attention to product design standouts. To product developers, having their product honored as “one of the ten” brings media attention, industry fame and sales fortune commensurate with the reputation and integrity of the organization compiling the list.

The competitions that create “Top 10” lists themselves tap into a powerful combination of human desires: to challenge, to overcome, to be judged worthy! The very notion that ten items have been assembled for a common purpose implies they are worthy of notice and, more importantly, that the competition that yielded them was honest and fair. Yet scratch just a little at the surface of many product design “competitions” and what you find is often in stark contrast with common perceptions of what judging a competition entails.

I have judged a fair number of national and international product design competitions (five in the past three years alone) and each has made the same procedural mistake: products are assembled and categorized, judging criterion are devised, reputable judges are assembled, and yet we judges never see or touch the products in person. Instead, we receive a set of written documents describing each product, its intended function, and its design process. Imagine an art contest conducted by email and you get the gist of what’s going on out there.

Not being able to observe products in action or to test their ergonomics in person not only reduces our ability to judge the quality of a product’s design and workmanship, but it introduces a disturbing, biasing factor: the writing ability and thoroughness of the submitter.  Rather than exclusively focusing on the merits of the product, design competition judges are insidiously influenced by the quality (or lack thereof) of the product’s description. They are forced to evaluate the idea of the product, not the product itself. This seems categorically unfair to the public, the product, and counter to the spirit of product design competitions. Moreover, it leaves the entire process open to vicious and (perhaps) justified attack.

Last year there was much public criticism of the CES Innovations Award that began on influential blogs like Engadget and Gizmodo and was picked up by the mainstream press because in a prior year the top award winners included a product which was never actually manufactured: it was simply "vaporware." That it topped the awards list made the distinction meaningless for the other winners and called into question the integrity of the competition and CES itself.

If my fellow judges and I had judged the products in person, this could never have happened. In the wake of this public debacle, in my discussions with product design competition judges there is consensus: we could do a much better job judging a product’s design if we see the product in person. This should be patently obvious, and frankly, many of the vendors I have spoken with since are shocked that we rarely, if ever, judge physical products.

I got to thinking: perhaps grade school science fairs got it right. Maybe we should learn from them and design a better judging process that focuses more on the product and less on the hyperbole of its description. We could gather all the products in one venue and have the judges convene to conduct the judging based on a simple Fair Judging process:

  • The vendor would present their product
  • Judges could then ask questions of the vendor
  • The vendor would step aside and judges would continue to evaluate and debate the merits of the product and its design
  • Judges would then move to the next product

Improving the judging process for product design competitions in this way would heighten the prestige associated with awards, lending products additional importance in the marketplace. Using this process would allow judges to do a more thorough job of judging and eliminate the possibility of judging vaporware. It also narrows the set of competitive products to those that are (or are reasonably close to) fully functional. The Fair Judging Solution ensures that competition is fair to products and not just promising ideas that never come to be.

Of course, to judge products in person takes money and coordination. The Fair Judging Solution is simple, but the logistics involved make this a potentially challenging problem to solve. I didn’t say it was easy. But it could be one of the Top 10 ideas of the year.


Usability Professionals' Association
promoting usability concepts and techniques worldwide
Phone + 1.630.980.4997         office@upassoc.org

Contact the Voice