The Big Flaws in Hotel Rankings

Should you trust online customer hotel reviews? TripAdvisor says a generic Candlewood Suites is the 11th-best hotel in Dallas, behind the swank Ritz. Expedia ranks the hotel as one of the worst. Why? Scott McCartney explains on Lunch Break. Photo: Setai Fifth Avenue.

A Candlewood Suites hotel near Dallas Love Field airport is ranked No. 11 among Dallas hotels on TripAdvisor, a popular travel website of customer reviews. On Expedia, however, reviewers rank that Candlewood at No. 368.

Hotel rankings can vary wildly across websites. Here's a sampling:

Setai

HOW THEY RANK: Setai Fifth Avenue in New York: TripAdvisor: #2; Expedia: #9; Hotels.com: #4; Orbitz: #133

Enlarge Image

Close Baymont Inn & Suites

HOW THEY RANK: Baymont Inn & Suites in Dallas: Trip Advisor: #3; Expedia: #19; Hotels.com: #136;Orbitz: #223

In Los Angeles, TripAdvisor reviewers rank a Rodeway Inn on the Pacific Coast Highway at No. 10 in that city. Yet Orbitz customers put the same motel at a dismal No. 325.

Which reviews should you trust?

Just as with cars, books and restaurants, user reviews have changed how travelers make buying decisions. One website, TripAdvisor, dominates the hotel-review arena, but online travel sellers Expedia, Priceline, Travelocity, Orbitz and Hotels.com, which is an affiliate of Expedia, are gaining fast. Scores of smaller, regional websites have customer reviews as well.

{if djIsFlashPossible}

The version of Adobe Flash Player required to view this interactive has not been found. To enjoy our complete interactive experience, please download a free copy of the latest version of Adobe Flash Player here

{else}

This content can not be displayed because your browser does not support the Adobe Flash player required to view it.

{/if}

A wide range of factors affect rankings: the number of hotels listed on the site, the total number of reviews, whether websites themselves tinker with the rankings and whether questionable reviews are effectively policed by moderators.

Researchers say reviews have become increasingly important to push lookers into bookers. Checking the reviews is often the final step for shoppers before making a reservation. "It's hard to overstate how important customer reviews are'' to hotel sales, said Douglas Quinby, senior research director at PhoCusWright Inc., a travel-research firm.

Thus the temptation for hotels to submit fake reviews to pump up their scores, or even trash competitors to improve their own sales. TripAdvisor says it knows of review "mills" and "reputation management" firms offering to stuff the ballot box for hotels willing to pay. Websites like Fiverr.com are full of people offering to post batches of fake reviews for $5 and up. Some hotels have been offering incentives to guests for writing good reviews.

In a study of hotel-review websites last year, PhoCusWright decided to remove one small national brand of hotels because the data were suspicious. "The volume of reviews was off the charts and the [rating] scores were off the charts," said Mr. Quinby. He declined to identify the hotel brand.

Comfort Suites on North Michigan Avenue in Chicago.

Comfort Suites on North Michigan Avenue in Chicago.

TripAdvisor, with more than 60 million reviews, has been under fire. In February, the United Kingdom Advertising Standards Authority ruled that TripAdvisor's advertised claim of "trusted advice from real travelers" was misleading, because fake comments could be posted without verification.

The Boston-based company, which was spun off from Expedia last year as a publicly traded company, says it has deployed technology to filter reviews and flag problematic content. And with so many reviews—40 contributions are posted every minute—most hotels would have to generate big numbers of fake reviews to influence rankings.

But more important, TripAdvisor says customers and hotels themselves are able to police the site for fake or inflated reviews. When reviews don't match up with reality, consumers return to the site to post reviews of their own experience, said Adam Medros, vice president of global product for TripAdvisor. Hotel owners sound the alarm either when another hotel is suspected of pumping in false reviews, or a company makes an offer to an owner to post fake reviews.

"It just works," said Mr. Medros. "The site wouldn't have grown as it has without users coming back and saying the information was useful."

TripAdvisor believes fraud is rare—it currently has fewer than 100 hotels flagged out of 555,000 on the site, said Severine Philardeau, vice president of partnerships. The U.K. advertising standards issue was "very much a technicality," she said, and TripAdvisor had already changed its advertising to "reviews from our community" for other reasons.

And as for Fiverr.com, Mr. Medros said TripAdvisor "content integrity team" has been known to "apply" for jobs writing fake reviews to identify which hotels are behind the ploy. Hotels get a warning and, in some cases, a "red badge" is attached to the hotel's listing warning readers that the hotel "may have attempted to interfere with traveler reviews."

Enlarge Image

Close

Unlike TripAdvisor, Expedia, Hotels.com and Priceline all limit their guest reviews to customers who booked hotels through their websites. They send an email after a stay with a unique link inviting that traveler to review the hotel. Orbitz said it does the same, although it also allows reviews by people who didn't book through Orbitz. The online travel seller identifies reviews on its site as either verified or not, and says all but a few are verified.

Travelocity doesn't verify the reviews it posts and tallies on its website because "social communities are not meant to be 'members only,' " a spokeswoman said. Travelocity said it has moderators who look for questionable reviews.

Rien van Rijthoven

Orchard Garden Hotel in San Francisco.

Many of the websites themselves tinker with rankings rather than simply listing them by their average score. Expedia, for example, takes into account the hotel's distance from the neighborhood a traveler specified and other factors, said John Kim, senior vice president of global products at Expedia.

Customer reviews are quite different from traditional "star ratings" of hotel quality. They reflect the value customers think they received and whether a hotel exceeded expectations. That's why a Rodeway Inn could be one of the 10 highest-rated hotels in the Los Angeles area, Trip Advisor said. But why such good scores for a motel with very low ratings from reviewers at sites that verify the reviewer actually stayed at the property? "We have more reviews than all of those competitors," Mr. Medros said, so ratings could be significantly different.

While experts say what matters most to shoppers is the hotel's overall score, it pays to read the reviewers' comments, discounting the highs and lows. Consider focusing on those written by people most like yourself. Some sites allow you to highlight reviews from people traveling with children, couples or business travelers, for example.

Of course, trying to influence reviews and ratings is a time-honored tradition in the hotel and restaurant industries. Travel-guidebook legend Arthur Frommer said he began printing reader letters about hotels in the 1960s. After about five years, he realized that hotels were writing him letters about themselves. "I was being gamed," said Mr. Frommer. "Hotels are so dependent on reviews that of course they will generate their own. They would be crazy not to."

As for online reviews, which have replaced guidebooks for many travelers, Mr. Frommer has disdain. "It's a vast buzzing, blooming confusion."

—Email middleseat@wsj.com Printed in The Wall Street Journal, page 29