Bing might be light years behind Google when it comes to number of people who actually use it as their main search engine, but that hasn’t stopped Microsoft from trying to convince people how much better it’s own site is. In its latest effort to try and steal a few million searchers from its hated rival, the Redmond firm set out to show just how many people actually prefer Bing’s results to Google’s – when they’re given the chance of comparing the two in a Pepsi Challenge-style blind test.

That’s what the Bing It On campaign was all about, referring to Microsoft’s study of around 1,000 participants, two thirds of which apparently ended up preferring Bing over Google. Armed with the results of these ‘tests’, Microsoft screamed it from the rooftops to anyone who’d listen, whilst setting up the BingItOn.com website to allow anyone to take the Bing challenge for themselves.

Microsoft seems to think its test is pretty compelling, but at least one analyst says otherwise. Ian Ayres, a professor of law and economist at Yale University, was quick to dismiss Microsoft’s claims as pure propaganda, and went on to prove his theory as outlined in the Freakonomics blog:

“The 2-1 claim seemed implausible. I would have thought… it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.”

To underline his point, Ayres set out to test 1,000 of his own random subjects on the Bing Challenges to see if he could replicate Microsoft’s results.

Ayres Bings It On

If Microsoft’s claims that two thirds of people do indeed prefer Bing to Google are correct, and if its test was genuinely fair, then we should be able to replicate these results in other tests, right? At least, that’s a pretty reasonable assumption – one would think – so why is that not the case?

In his own tests, Ayres came to the conclusion that not only did his subjects not prefer Bing’s results over Google’s, but in fact most of them – when left to their own devices – showed a preference for the latter.

You see, it all boils down to what you’re actually searching for. In the Bing It On challenge, searchers are pressed to search for Bing-suggested terms, most (if not all) of which have been very carefully chosen by Microsoft. But according to Ayres, when his own test subjects began choosing their own search terms, the results were very different indeed.

Ayres contends that Microsoft carefully picks out recommended search terms for BingItOn.com, knowing full well that in these rare cases, Bing comes out on top. He further criticizes the company for misleading consumers in its advertising campaign, noting that it’s yet to release the results of the five million-plus people who’ve taken the Bing-It-On challenge so far.

“[our tests] indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results,” writes the professor.

“It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge. If, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate.”

Always up for a fight, Microsoft was quick to hit back at Ayre’s claims that Bing-It-On is a load of gibberish. In his Challenging the Challenge to the Bing It On Challenge blog post, Bing’s Behavioral Psychologist Matt Wallaert makes the rather weak claim that Microsoft doesn’t publish the results of the five million people who took the Bing It On Challenge because, well, it doesn’t know what they are.

According to Wallaert, Microsoft doesn’t actually track Bing It On site users because that’s apparently unethical:

“Microsoft takes a pretty strong stance on privacy and unlike in an experiment, where people give informed consent to having their results tracked and used, people who come to BingItOn.com are not agreeing to participate in research; they’re coming for a fun challenge. It isn’t conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it.”

What’s that, Microsoft taking an ethical for once?! Or could it be that, perhaps, this is just one of those rare instances when tracking such data could actually be unprofitable for the company?

Either way, it probably doesn’t even matter. The truth is that its not just the results themselves – there are all kinds of reasons why someone might prefer one search engine over the other. Things like the actual site interface, the number of ads and sponsored results, social network integration and privacy concerns all have to be taken into consideration when choosing which search engine we like.

The fact that most people continue to use Google begs the question – even if Bing’s results genuinely are better, does anyone actually care?