Test. Inform. Protect. Each and every month upwards of twenty million people religiously turn to a non-profit organization in Yonkers, New York, for product ratings and advice on purchasing everything from breakfast cereal to cars. And, for sixty-five years, Consumer Reports magazine has done the near-impossible, publishing its unbiased test results without accepting a single advertising dollar. To top things off, in an age when conventional wisdom says that readers just won’t pay for content on the Internet, the magazine has challenged that notion with over 500,000 paid subscribers to its Web site, consumerreports.org.
Founded as Consumers Union in 1936, the consumer advocacy group started Consumer Reports as a means to present its research findings on a wide variety of consumer goods. Today, its staff of 160 testing experts produces a broad collection of products and properties including a radio show, Consumer Reports television, health and travel newsletters, special edition publications, a children’s Web magazine, zillions.org, and of course, Consumer Reports.
To take a closer look at the inner workings of Consumer Reports, Failure magazine’s editor, Jason Zasky, visited its Yonkers headquarters and spoke with the man responsible for “the largest and best-equipped consumer testing lab in the United States”—Dr. R. David Pittle, Senior Vice President & Technical Director of Consumers Union.
What is the mission of Consumer Reports?
Our whole goal is to try to take the confusion out of the marketplace, because advertising and the claims made [about consumer products] aren’t always reliable. We evaluate the things our subscribers tell us they want to know about. That involves a very long list of what we call ‘sweaty palm’ issues—washers, dryers, refrigerators, dishwashers, automobiles, televisions, microwave ovens—things that have a fairly significant expense to them and people don’t know how to judge their performance when making a purchase.
We also test products the way we think consumers will use them. There are a lot of standards in the world—many of them are commercial standards—created to help a buyer, retailer or distributor, judge what they're buying. But a consumer wants to know; How does it look? How does it feel? How convenient is it? If I use it in a way that’s predictable of a consumer, how will I experience it? That’s the way we look at it. We’re in the consumers’ corner as their advisor.
At the same time, we have to be fair to the manufacturer, because we’re not anti-business, we are pro-consumer, which is a very important difference. If every manufacturer made terrific products, that’s good news as far as we're concerned, because then the consumer has more choices. But if some manufacturers make terrific products and a lot of the others don’t, that’s a burden on the consumer. So from our point of view we feel obligated to describe what we see out of our right eye as well as our left eye. It’s the complete picture. There are some publications who report on products who have an unwritten or unstated policy that seems to say, “We'll report the good news not the bad news.” We’re not here to promote sales, we’re here to help you make the best choice based on what you want.
How else is Consumers Union different from a typical magazine publisher?
We’re not a publisher per se that’s in it for the money. We’re a consumer organization that does research. We’re not trying to accumulate wealth. Everything we get from selling the magazine and Web site subscriptions, we plow that back into buying new products and expanding our information base.
How does Consumer Reports decide which products to cover?
Here’s what happens. We sample our readers’ reactions to what we have in every issue. We go to small subset and ask them what they read and whether it was useful. We also get about 50,000 letters and phone calls [each year].
How does Consumer Reports decide what models to test?
We do a very thorough analysis of the market. We call or write manufacturers and say, “What are you going to be making this year? What are the important features? What is the new technology? What are going to be your hot models?” Then we’ll look at all that and pick what we think will be the best coverage. We can’t test them all. We try to evaluate the models that people will most likely find in the store. We look at market share, new technology, and the features that consumers seem to want. We try to make the most rational choice, because we don’t have enough resources to do it all.
How do you respond to readers who complain that they can’t find the models you review in the stores?
This is a very large country. If a manufacturer makes 50,000 of one model that doesn’t go very far over 50 states. So if we evaluate a television of which they made 50,000 there may not be many left. On the other hand, what often happens is that the retailer has invested money in whatever it has invested its money in. So you walk in and say you want model XYZ. They say, “Well, we don’t have that model. But you don't want that anyway. It’s old. This is what you want. It’s new.”
That’s why we’ve tried to double and triple the number of models we cover. We also try to give the reader information that has legs. If you’re buying a television, we tell you make sure its got a comb filter. Televisions with comb filters have a better picture than most of the televisions we tested without a comb filter. So we try to provide generic information as well as model numbers. You also see by looking at our ratings what models to avoid. I’m willing to bet that there are a lot of consumers who look at these charts and go, “Well, I’ll try to find that one but I don’t want this one.”
Also, you’d be surprised how many products are exactly the same but if it goes into Macy’s it gets one number, and if it goes into Filene’s it gets a different number. Or the differences are very slight. We spend a lot of time and money trying to crack those codes so we can say, “They’re all alike.”
What do you do when you find a product that you feel is a safety hazard?
When we find something we think is particularly dangerous or surprisingly dangerous, we will communicate that to the relevant government agency. The manufacturer finds out about it when we publish the results. But we will petition the government or make them aware and request that they investigate and consider a recall.
When it comes to automobiles we’ve had cars that we said were unacceptable, most recently and most notably vehicles that have tipped rather severely in our emergency handling test, and they’ve all been SUV’s. The rollover problem in SUV’s is something we began writing about in 1988 with the Suzuki Samarai. We’ve since been sued by them on that story. They sued us in 1996, eight years later. The judge granted summary judgement—he said there’s no case, but Suzuki has appealed the decision.
Another case—Isuzu sued us on the story we wrote about the Trooper in 1996. That went to trial and the jury basically rejected all of their claims. In fact, rather then us give them money they had to give us money to cover some of our court costs. They didn’t appeal so that is over and done.
So there have been controversies. But at the same time we have kept the public very much alert to the concerns about rollover, stability and emergency handling of vehicles, which we consider very serious. I can’t imagine anything more chaotic than rolling over in an SUV. We have an extensive 320-acre auto-test facility in Connecticut that is state-of-the-art and a very fine and experienced staff to evaluate motor vehicles of all kinds.
How do manufacturers respond to your criticisms?
The staff of manufacturers tell our staff when they go to technical meetings and conferences, “Oh, boy. Did you guys catch us with our pants down. My boss was really furious and we really made this thing change—we fixed it.”
When manufacturers don’t like what we publish or they’re surprised or they don’t agree with us, they will call and want to talk about it. We say come in and talk with our staff. They will walk away with what to do with their next model. And after a while they can start to get ahead of the curve.
I’ll tell you a story. We tested door locks six or seven or eight years ago. We bought a bunch of doors because we always test in real-life conditions, and mounted the locks. Then we tried to kick the doors in—because somebody could do that. That particular year these things were all kicking out—one kick and the doors would open up. They opened because the striker plate was being pulled out of the wall. The screws that came with the locks were only about this long [indicates an inch to an inch-and-a-half], and when you gave the doors a good swift kick those two or three screws didn't have enough strength to hold the plate in place. So no matter how elaborate the lock, the combination of lock and striker plate were not effective.
Before we wrote that story we said, “Let’s try putting in longer screws.” So we re-tested all the locks again, not with the manufacturers’ screws but our own screws. I think they were two or two-and-a-half inches. We kicked on the doors and the longer screws held the plates in place.
So three or four years later we decided to do another security story. We bought all these locks and opened them up and guess what? They all had long screws. So we didn’t do that test. It tells me that manufacturers pay attention. What does it take to put a little longer screw in there? But it’s not something they did on their own. They did it because we highlighted that the consumer is much more secure with longer screws.
If you give a product an exceptionally positive report, does that ever effect production? Do manufacturers ever attempt to ramp up production to meet demand?
You would think so, but take electronics, for example. Electronics are generally made in Japan or Taiwan. By the time those products are tested and the results published, it’s four or five months after they’ve made the last one. You would think they’d say, “Let’s make another 20,000 because all those people are going to want these.” But by the time they read the story and get ramped up to build it again, it might be three months later and nobody’s interested anymore. But they do know what was important and what played well.
With appliances, which are made mostly made in this country, they sometimes carry over a model because it was highly rated. With cars, they have a life span of three to four years, so they have plenty of time to keep the supply up. We tested the [Volkswagon] Passat and thought it was one of the best cars we ever tested. Right away sales started to go up. I don't know what they did internally, but we didn’t hear that there was a shortage of Passats.
How does one become a staff shopper for Consumer Reports?
We need shoppers because we have to sample products on shelves around the country. We have 160 shoppers—they're part-time people, they receive their instructions, much like Mission Impossible, and they love it. We make sure that they buy with cash (or a credit card) and they’ll walk into a store and pick up five dozen condoms or 20 boxes of cereal. We sometimes have to do some pretty bizarre things.
Do staff members ever suffer injuries during testing?
We’re very sensitive to safety and at the beginning of every project there is a discussion about what tests are going to be conducted, and if anybody is at risk then we take steps to reduce that risk. I don’t ever remember anyone being hurt as a result of a test protocol although I am positive that somebody hit their finger with a hammer once or stubbed their toe on something.
What’s the most difficult product to test?
You’d be surprised…. Actually, the most difficult product to test is the one we haven’t tested before because it isn’t so obvious what to do from an engineering point of view. Tires were a terrific challenge for us; condoms were a challenge. It’s difficult to give people advice about products where there’s a strong element of personal preference.
What are the most time consuming products to test?
The ones that have taken the most time are personal care products. Deodorant is incredibly expensive. Basically, when you’re evaluating deodorant, you can’t just do a chemical analysis. Its interaction on the body really does matter. So, [you] find somebody who basically who smells armpits. We haven’t done deoderant. I think we bypassed that because of expense.
What’s the most expensive product Consumer Reports has ever covered?
Motor oil. To evaluate motor oil and we went down to New York City, commandeered 75 taxis, changed their engines, and put oil in religiously every 3,000-6,000 miles. Then we took the engines out of the taxis, looked at the engine parts to see how much wear there was on the pistons and how much carbon buildup there was and concluded that they all did fine for 60,000 miles. There was no sign of wear on any of them. So our advice was you can change your oil every 6,000 miles rather than 3,000. And we said you can buy the cheapest oil you can find. You don’t have to worry about the $2-a-quart whiz-bang oil, because that’s what we tested and it wasn’t any different.
Now it’s very simple to say in a paragraph, but when we did the test it was very scary because we entered into something that cost us well over $300,000. Suppose the experiment failed? I was very pleased about that motor oil project because it answered a burning question. All the ads about the best oil—you don’t need to spend that money and you don’t need to change it as often because our experience with these taxis demonstrated fairly conclusively that it’s not necessary.
How often do you do a test and the results don’t yield anything meaningful?
There have been very few dry wells. We do a lot of pilot testing to see if the approach we are taking makes any sense. That’s not to say that we haven’t done tests where we said, “You know what? These things are all alike.” But that’s because they're all alike. You don’t want to force a difference that isn’t significant in any way.
Take windshield wipers, for example. They were all good, except we did see that if you buy the whole blade rather than just the insert you get a better, longer lasting product. But I buy the insert because they’re $4 apiece and a blade costs $16.
We also tested household bleach and found out they all did exactly the same thing. In fact, they are the same thing. It doesn’t matter whose name is on it. So we told everybody buy by price and we’re not going to test it anymore because it’s always the same answer.
How has the Web impacted what you do?
On the Web we provide more information and provide it in a way where subscribers can make choices. What’s really useful about the Web is it’s searchable. The Web site provides an opportunity for the user to search and get information tailored to what they want. The Web is the here and now . . . and the future. We have, if not the first, tied for the first paid-subscriber site on the Web. We have well over 500,000 paid subscribers, and it’s growing.
What's the key to Consumer Reports’ success?
People trust us and they believe in what we’re doing. We don’t have an axe to grind—and that’s very important. I think people have learned that in today’s society there are a lot of institutions they just don’t trust anymore. But they do trust Consumer Reports. And we don’t take that trust for granted. We have to earn it every day. I’ll do everything I can to see that we are never wrong but I’d rather be wrong than not independent because I can get right tomorrow but once you lose your independence you are really something very different.