Return to Search Results

People around the globe love ratings, from Car of the Year Slovakia and Los Grandes De Nuevo Mexico Music Awards to Cooks Illustrated's ratings of catsup.

Corporate trainers are no exception. We pay attention to awards contests for training products and often make choices based on them. The contests can serve a valuable function. They help make sense of the sprawling training industry, which is awash in products but has few well-established authorities to guide buyers' choices.

Contests are also popular with companies that run them. The reason is simple: they make money.

SPURNED AWARD
Take a recent contest to identify the most influential individuals in training. Anyone could vote on the web, and the controls were so lax that companies could stuff the electronic ballot box for their CEO.

It wasn't a surprise that some of the individuals who made the list worked for companies that are paying sponsors of the organization running the contest. An award is an effective way to ensure a sponsor's continued loyalty.

I am not saying the people who won weren't worthy. I am saying the contest wasn't worthy of them. There was no transparency, and the results had the appearance of a conflict of interest. In fact, one of the more prominent individuals the contest recognized refused the award.

ANNUAL PRODUCT AWARDS
The corporate training industry has a variety of product award contests and events, many focused on e-learning. Understandably, people tend to pay attention to the results, not the process that led to the results. When you do take a close look at the evaluation process, you encounter red flags that make you wonder how credible the competitions are.

Vendors usually pay a fee for each product they submit. Because award competitions attract attention, vendors feel pressure to participate. To enter multiple products, a single vendor can easily run up a tab of thousands of dollars. With many vendors participating, the entry fees alone can make the contests lucrative for the sponsoring company. You may notice that contests tend to have quite a few categories. These may correspond to well-defined uses of technology, but they also invite more submissions and thus more money for the contest vendor.

It's an irresistible business model. It does have a catch, though. If vendors pay a significant sum of money to enter, they tend to expect something in return. The contest vendor has to walk a fine line between keeping one set of customers (vendors) happy while not alienating the other set of customers (training professionals). Contests don't disclose the number of submissions (a trade secret?) so you can't determine how selective the awards are. It does seem, though, that contests are generous with the number of products they recognize. That may be the implicit quid pro quo between contest vendors and product vendors.

To have a profitable contest, you have to keep overhead low. The biggest potential cost is judging. If you are going to run a competition with many entrants, you're going to need a large number of judges. The preferred solution is volunteer judges who do the work for free.

There is no perfect solution, but the downside of volunteers is that you have no idea how competent they are or how uniform their application of the contest criteria is. Those uncertainties plus annual turnover in judges raise the question of how comparable the contest results are year to year.

In addition, the evaluation criteria can be very broad because they have to be to cover all the different types of products in a contest. Judges are given wide discretion in interpreting the criteria so inevitably you're going to get differences in what they think the standards mean. That would be less of a problem if the decisions were explained. I haven't seen a training industry contest in which the judges explain in any detail why they chose a product for an award.

And I have yet to see a critical word about winners. Every technology product involves tradeoffs and compromises that suit some users but not others. You don't get that kind of information from contests.

Another judging model is for the contest vendor to use in-house people. This could be an excellent solution, assuming that employees have the requisite expertise. It also seems a better bet to ensure year-to-year consistency. You're going to pay the employees anyway so there's only an opportunity cost to worry about.

One well-known competition uses this model. The red flag is that the company does consulting for companies in the industry, and there's no Chinese wall between the consultants and the contest judges. They are the same people. In a recent year, six out of 10 award-winning companies also appeared on a list of consulting clients of the contest vendor. In another category over 50% of the winners were clients. Note: the publicly available client list was partial; it's possible that some of the winning companies are clients but aren't listed.

A third approach is to dispense with judges altogether and run the contest as a poll. In this approach, the contest sponsor has no role in deciding the winners. The weakness, of course, is how representative voters are of the population relevant to the product being voted on. There all kinds of bias possible in a pool of self-selecting voters. Is it more likely that people who love a product will take the trouble to vote than people who aren't crazy about it? Do you know if voters have much hands-on experience with a product--or none at all?

ANOTHER LAYER: CONFERENCES
The awards business has another layer that shouts "conflict of interest"!

Many of the award givers run conferences in their area of expertise. Conferences are great revenue sources, and they double as valuable marketing programs, providing visibility and buzz.

Conferences have sponsors and exhibitors. Unlike golf, where sponsors set up tents with lavish food and drink (at least they did until Tiger put a dent in the industry), at training conferences, sponsors put on more modest events. For this they pay fees--for being a listed sponsor and for exhibiting. They might also get a free session thrown in that they can use to market their wares.

Conference sponsors and exhibitors often receive product awards from the companies running the conferences. With vendor fees coming in from multiple directions--contest fees, consulting engagements, conference sponsorships and exhibitions, print and web advertising--the objectivity of the award givers is in tatters.

WHAT IS ACTUALLY ASSESSED
Have you ever wondered how contest judges evaluate a large number of complex e-learning products in a short period of time?

In the case of one leading contest, the answer is that judges don't evaluate the actual product. The contest rules state that vendors submitting products can furnish a recorded demo, document, PowerPoint deck, or a combination to judges. In other words, the contest is actually a marketing competition!

To be fair judges can test drive trial versions of an application. Still, trial versions often disable key features. It also doesn't seem quite fair to have some judges using trial software for one product while other judges view a slick Flash demo of another. For this reason, if I were a vendor I wouldn't take a chance on evaluators drawing their own conclusions--I would submit a marketing presentation that furnishes conclusions for them.

You also have to wonder how many unpaid judges with busy schedules take the marketing pitches as the path of least resistance. I'm not criticizing judges who do that because I can see myself taking that option.

WORTHWHILE RESULTS?
I don't begrudge people making a buck, and many product contests make a lot of them, year in and year out. I'm not charging contest operators with running rigged competitions.

I've heard vendors complain in private that they feel compelled to participate in some high-profile awards programs even though they're queasy about the process and are well aware of the conflicts of interest. I sympathize with them, but they do have the option of not entering contests they're uneasy about. If they didn't enter one year, would they begin to lose sales? I'm skeptical that the contests have such power in the market.

Some training product contests are poorly designed and allow potential conflicts of interest and bias. The training profession deserves better. Training contests provide information that might be valuable or might not be. The trouble is that you and I have no way of determining the value. Transparency is a well-traveled word these days, but it is what training competitions need.

I've heard people, including myself, bemoan the fact that ASTD, the industry's main professional organization, doesn't make product recommendations based on a transparent evaluation process. However, ASTD has to make a buck too. It runs its own convention and needs to attract exhibitors. It also solicits advertising and has a new "Professional Partners" program that's clearly fee based.

ASTD has to manage its own financial relationships with vendors. They wouldn't want to offend vendors, like Coastal Training Technologies or Mindleaders.com. So it's actually a good thing that ASTD doesn't give out product awards.

Training Media Review has its own modest awards program. In the interests of transparency, I want to note that we don't charge vendors for reviews or solicit consulting from them. The awards are based on detailed written reviews, and reviewers have no ties to the vendors of products they evaluate.

CINE, a nonprofit film and TV trade group with a professional staff, has been giving out awards for over 50 years. They use volunteer panels that look at specific types of productions and have a two-level process to guard against bias and poor judgment.

The key to change is the training profession. As long as we're uncritical about awards, the status quo will prevail because contest vendors have no incentive to change. If we don't demand transparency, then we probably aren't going to get it.