Friday, September 28, 2007

How To Interpret Technology Product Reviews

I'm still considering options to replace the productivity void left by the death of my hx4705 Pocket PC. I'm trying to avoid what would be the easiest and most inexpensive option: learn to live without a Pocket PC. I spent a good part of time the other day looking over the pool of available devices and comparing them to the features that I need, the features I want, and the features I can compromise on. One problem I'm facing is that Pocket PCs, especially phones, are still an emerging technology and a lot of good features aren't completely worked out yet. There has been a lot of progress but I still find it hard to narrow down my choices to one particular device.

As much as I would love to have kept the 8125 if it had worked properly, I am now scalded and do not plan to buy another one, even if it were brand new with a warranty. The model that I had was a store return, reason unknown. Some of the usability problems that I had were widely reported by other users, although my stuck camera button problem seemed to be unique. After spending a decent chunk of time comparing my options, I am seriously thinking about the HP iPaq 6945, which is the unlocked GSM version of the 69xx line of iPaq Pocket PC Phones. However, if I were to buy this phone, I would have to expect it to last for at least two years. I can't decide in two months that I'd rather have a 4" VGA screen, and I can't decide that the thumb keyboard is not adequate for my short, stubby fingers. In order to help me to make an educated choice, I have to rely on hardware reviews as taking a "test drive" of a phone is never easy. I don't know anybody who has this model, and while chances of AT&T having one in their stores are slim, chances of them letting me try to make and receive calls while browsing the Internet and writing a paper for school while taking pictures of my kids are even slimmer.

When evaluating and interpreting hardware reviews, you have to consider several factors:

1) What kind of review is this?

There are professional reviewers and there are user reviewers. One of the first places I like to go when researching cell phones or other gadgets is CNET.com. CNET has been around for many years and has access to all of the cool gadgets. They have the tools to test every aspect of the device. However; CNET falls short on long term usage. They may play with the gadget for a few hours, but they really can't tell you how well it will hold up over a period of six months of heavy use. For that, you need a user review.

Where do you get a user review? Well, what is this? My blog is an example of a user review. I have owned several gadgets, and I write about them. I may not write extensively and I may not be formally trained but maybe I complained about a feature and your Google search dropped you here to read it. CNET does have user reviews for most products as long as users provided them. Some gadgets get more reviews than others; for instance, the Cingular 8525 had 270 reviews last night while the iPaq 6945 had only four. To get more information, I had to go to Google.com and Live.com to search for "iPaq 6945 user review". I found some blog posts and forum discussions, and Amazon.com had 29 reviews that gave me some good information to make a decision with.

2) What is the perspective of the reviewer?

CNET.com has professional reviewers. Typically, the manufacturer will send them a review unit that they can play with, post a review, and send back. They'll tell you what features the device has, how well it operates under certain conditions, and of course they can compare the device against more "absolute" standards. I think of them like movie reviewers, except slightly more useful to society. How many times have you seen a professional movie reviewer dismiss a movie as a total waste when you saw the movie and thought it was great entertainment? The reviewer is trained differently than you and has a different perspective. The reviewer normally has very elite tastes and is trained to recognize concepts like plot and character development. Maybe you just want to see some zombies getting their heads blown off. The movie may not have any real plot or character development but it keeps you entertained for about 90 minutes. CNET.com reviews can be looked at in this light. They are comparing the iPaq 6945 against a higher standard while I'm only comparing it against devices in my price range. They might say that the 8525 is a better device, but if the 8525 is out of my price range, I am not going to be able to compare the 6945 against it. It's the same with cars. If you can only afford a $12,000 car, you honestly don't care how well it stacks up against a $50,000 car; you want to know how it compares against other $12,000 cars.

3) What information does the review contain (especially important for user reviews)?

If you look at reviews on Amazon, and you see something resembling the following:

THIS DEVICE SUCKS!!!!!!!!!!!!!!!!!!!!!!!!! DON'T BUY IT!!!!!!!!!!!!!!!!!!!!!! Mine broke. Tech support is horrible!

What does that actually tell you? You know nothing about the conditions the user tried to operate the device under. You know nothing about the user's knowledge or skill level with this device. This tells you nothing about the user's expectations for the device at the time of purchase. I read some negative reviews about the 6945 from Palm users who preferred the Palm OS better. They compared the 6945 to a Treo. That gave me some information about their background, skill level, and expectations. Similarly, when I read reviews from previous 63xx and 65xx users, I could tell that they had used this platform before, had experience with similar devices and software, and I could relate to their reviews. Their reviews gave me an idea of how the device would operate under my conditions at my skill level. One review from a Palm user said that "todo list doesn't support categories" and "Outlook isn't customizable". I see that these statements show the user to be unfamiliar WITH THESE PLATFORMS, as I do know that Pocket Outlook tasks do support categories and Outlook is highly customizable, but only an experienced power user such as myself can unleash a lot of the customizability of Microsoft Outlook.

Remember to use critical thinking when reading reviews. As an experienced Windows Mobile user, I can sympathize with a "newbie" having problems adapting to the platform but newbies complaining about "this device sucks" really doesn't tell me what to expect if I buy the device.

Actually, after doing all the research I did the other day, I have come to a conclusion for now. I'm going to wait to hear back from PDASmart.com and if my hx4705 can be fixed for a reasonable amount, I'll just continue to use it for a while. This is a tricky time to buy a Pocket PC, especially the phones. Windows Mobile 6 is out, but not in wide circulation yet. If I bought a Windows Mobile 5 device, chances of an update being provided (you heard me, HP) are almost non-existent.  I might as well wait for a complete line of WM6 devices to hit the market and build up reliability and usability statistics before I jump onboard. In the meantime, I borrowed my mother-in-law's iPaq 1940 to get me by. It's WM2003 1st edition and lacks wi-fi, but it will hold my calendar and tasks and I can keep my UOP textbooks on it in RepliGo format. That helped me survive the staff meeting yesterday... There are few things in corporate life worse than an hour or more long staff meeting with no Pocket PC to help pass the time.

No comments: