The Best Decisions and The Worst Mistakes - Ranking Search Engine Strategy

How did Google become #1? Was it Larry and Sergei's massive brainpower? Perhaps it was the plentiful free food provided to employees?

Of course, there are many reasons that explain why Google is Google, Yahoo is Yahoo, and AOL is Google, I mean, AOL. Here's my brief list of the seven most important strategic decisions and idiotic blunders that have defined the search industry as we know it.

The Good: Super-Smart Decisions

1. Google's partnership with Yahoo Search: June 26, 2000 - a day that will live in infamy. On this day, Google and Yahoo announced their partnership whereby Yahoo would pay Google to power search queries on Yahoo. This deal for Google was like Microsoft's deal to provide the Operating Systems for IBM PCs. It was like getting paid to get tons of free publicity and market share. I could also put this event in the "worst events" category, since Yahoo basically gave what would turn out to be their biggest (well, bigger) competitor a desperately needed shot in the arm.

2. Yahoo's acquisition of Overture: OK, so Yahoo created a monster by falling asleep at the search wheel for five or six years. To a degree, they redeemed themselves by acquiring Overture in July of 2003. The total acquisition price was $1.6 billion, which probably repaid itself in about a year. Just imagine what sort of trouble Yahoo would have been in had MSN or Google made this acquisition.

3. Google's acquisition of Applied Semantics: In April,2003, Google acquired Applied Semantics, a "producer of software applications for the online advertising, domain name and enterprise information management markets" according to Google's press release. More to the point, this acquisition was key to Google's development of their multi-million dollar Domain Park and AdSense Distribution businesses, businesses that exceed the revenue Google makes from running ads on its own site.

4. Yahoo's acquisition of Delicio.us: This one I admit may be a bit of a darkhouse pick, but hear me out. In December, 2005, Yahoo acquired Delicio.us, the application that allows users to "tag" their favorite sites and thus create their own personalized guide to the Web. My sense is that something like Delicio.us could be the killer app that destroys search as we currently know it. Granted, tagging still lacks the critical mass (or understanding by the public) to be too big a concern for Google, but if the next generation of Internet surfers catch on to Delicio.us, general algorithmic search engines could become obsolete! To read more about my far-out theory on this point, check out my posting on search engine death that I wrote earlier this year.


The Bad and the Ugly: File Under "What Were They Thinking?"

1. MSN's failure to win AOL away from Google: When Google renewed its contract with AOL to provide paid search listings in December 2005, there surely was a collective sigh of relief heard in the Googleplex halls. After all, AOL provides a huge amount of traffic to Google (10-15% I think?) and it also happens to be high-quality, low-click fraud traffic, which is becoming more and more important. To me, it is simply shocking that Microsoft (who was allegedly bidding aggressively to become the search provider for AOL) didn't do whatever it took to win this contract. Had Microsoft pushed Google out of AOL, Microsoft would have instantly become a major player in paid search, while at the same time knocking a lot of wind out of Google's sails. Even if Microsoft had to pay two times what Google offered for this deal, they should have done it.

2. Yahoo lack of paid search technology innovation: Yahoo has been resting on its laurels ever since acquiring Overture in 2003. The Overture (now YSM) user interface is nearly identical today as it was in 1999. The YSM content network (called YPN, just to confuse you further) is still non-existent, while Google AdSense continues to dominate. Even the YSM bidding model (pure cost-per-click) is a proven loser versus Google's yield-management cash cow. In short, had Yahoo invested in technology three years ago, they would have a lot more money and market share today. Instead, Yahoo has focused on building a traditional media empire out of Santa Monica, inviting celebrities to campus, and creating nifty TV commercials.

3. Google's launch of many pointless features: Mars, Page Creator, Web Accelerator, Catalogs, Earth, Talk, Orkut, Base, Spreadsheet - the list goes on and on. At this point, I can't keep up with all the new and amazing betas Google launches. Even more discouraging, most of these have little chance of every making any money, are released with oodles of bugs, and often have little actual value to consumers. I'm all for innovation, but my sense is that these releases are indicative of a lack of product leadership inside Google. Perhaps there are too many "yes men" who have convinced Google product managers that they can do no wrong. At some point, if Google keeps spreading its engineering resources across dozens of random products, the products that count - like AdWords and AdSense - are going to suffer.

Google Could Still Become Pan Am

It's interesting to look back and say "what if." Hey, Google has a bunch of really smart folks, they've developed some tremendously innovative products, and they've got brand value second to none. Innovation and smarts, however, do not make a company into a giant by themselves. Google made some very smart business decisions over the last five years - about which the same cannot always be said for their competitors.

Of course, I always remind my friends that the business world is a fickle world. You can be #1 today and bankrupt tomorrow. For example, if you asked someone in 1970 which airline would be the biggest in 2006, they'd probably tell you "Pan Am" (no longer in business). The same would be true for Big Box retailers (Kmart, coming out of bankruptcy), or automakers (Ford - have they been acquired by anyone yet?).

In fact, if you had tried to predict the biggest search engine in 1996, you might have said "WebCrawler" or "Magellan." Later in the 1990s, AltaVista, Yahoo, and Excite all had their days in the sun. And today of course, Google is the search engine du jour.

So yes, Google has been smart, but they can't rest on their laurels. Otherwise in 30 years they'll end up in the same category as Pan Am, Kmart, Ford and Yahoo!

Do Keywords Matter Anymore?

Once upon a time, you could make thousands of dollars a month in search engine marketing simply by finding the "long tail" keywords that your competitors had overlooked. This usually meant one of three things:

1. Misspellings: "mortage rates"
2. Run-ons: "mortgagerates"
3. Multi-word phrases: "san mateo county bad credit mortgage rate loan offers"

These words were veritable goldmines. While your competitors fought tooth and nail just to show up on the first page of search results for "mortgage rates", you often found yourself alone (and with a bid of $.10) on the tail.

As a result of this phenomenon, SEM experts spent a lot of their time focusing on keyword research, perhaps at the expense of ad copy optimization, landing page optimization, and process automation. But who could blame them - they were taking advantage of market inefficiencies and making a lot of profits!

These days, the value of the tail has diminished, if not disappeared entirely. There are three primary reasons for this:

1. The Search Engines are Smarter: All of the search engines now have "broad matching" or "advanced matching" features. This means that a big competitor that bought the keyword "mortgage rates" will still likely show up when a user types in "best mortgage rates" or "discount online mortgage rates." The rationale behind this - from the search engine perspective - is fairly obvious; by populating obscure long-tail searches with results from very competitive keywords, the bids increase dramatically (no more $.10 clicks on the long-tail).

It used to be that the search engines tried to balance their drive for additional revenue with user experience concerns by looking at "token matching." Think of a token as a word. A phrase like "bad credit mortgage loans" has four tokens. The old rule was as follows: if a generic phrase matches at least 50% of the tokens in a long-tail phrase, the search engine would consider that relevant enough to show the broad match results on the long-tail phrase.

So, in the "bad credit mortgage loans" example above, "mortgage loans" and "bad credit loans" would be broad-matched, but "mortgage" and "loans" would not (because they only have 25% of the tokens).

Based on what I see in the market today, this is no longer the case. The search engines have basically moved away from the "token-matching" system and are getting closer and closer to a "category-based matching" approach. In this model, a broad phrase like "mortgage rates" could be matched not only on "bad credit mortgage rates" but on "home loan rates", "find a mortgage", and "san mateo county loans for new home buyers."

One other related point worth noting. In the olden days, even if a broad match had a 50%+ token match and did show up alongside long-tail searches, it had to compete in the same auction for top ranking.

Here's what I mean: let's say that you paid $5 CPC on Google for the keyword "mortgage rates" and you elected to be broad-matched across all long-tail keywords Google thinks are relevant to you. Let's say that I bought the keyword "alabama low mortgage rates" and I was willing to pay $2 CPC for this keyword.

When a user did a search for Alabama low mortgage rates, Google looked at two factors to determine ranking - maximum CPC and click-through rate (CTR). The key point here, however, is that Google looked at CTR on a keyword-specific basis. So, if your broad matched keyword had an overall CTR of 10%, but only a .5% CTR on the specific keyword "Alabama low mortgage rates", Google would use the .5% CTR to determine ranking. So, if my $2 bid had a CTR of 10% (due to the fact that I would likely have highly-specific ad-text, and I might be an Alabama-specific mortgage lender),
my effective "cost per thousand" (CTRxCPC=eCPM) basis would be $200 and yours would only be $25. So, if we went head-to-head against each other, I would clearly show up more highly than you.

That is not the way things work today. As far as I can tell, Google has moved away from the keyword-specific auction model to more of an "overall CTR" auction model. In this new scenario, if your generic keyword has an overall CTR of 5% (but remember, only .5% on the specific keyword) and my specific keyword has a 10% CTR, the eCPMs would be $250 for you and $200 for me - you would show up first. The long and the short of it is that this new system decreases the relevance of the tail by enabling high-CPC generic keywords to outposition targeted keywords.

By changing "token matching" to "category matching" and by changing the concept of a "single auction" to an "overall auction", the results may be slightly less specific-ads for the user, but virtually eliminate market inefficiencies (read: no more low CPCs) for the search engine.

Of course, the flip-side of that is that as user queries are consolidated into a few categories, the bulk of ad-generated traffic is consolidated into those advertisers who can pay top position for the most popular searches. Or, to put it another way, instead of entering into 10,000 auctions for 10,000 different keywords, advertisers are now entering into just a few auctions for those 10,000 keywords. If you want to get traffic, you have to win in one of these few auctions.

2. Competitors are Smarter: Even assuming that the search engines weren't doing everything in their power to move generic keywords into the long-tail results, the advantage of huge keyword lists has been reduced simply because competitors have caught onto the tactic. There are now dozens of companies (Trellian, WordTracker, GoogSpy, BadNeighborhood) that offer reams of keywords for pennies a day. Some of these folks now even offer APIs so that you can directly send the keywords right from their databases to your search engine accounts.

On top of that, the search engines have realized that being open about keywords helps their bottom line. Both Yahoo and Google offer keyword suggestions (taken directly from either user queries or other advertiser's keyword lists) at multiple points in the account set-up process.

Two years ago, perhaps 30% of any keyword market was relatively devoid of advertisers and presented great arbitrage opportunities. Today, my guess is that less than 10% of the keyword universe isn't heavily saturated. Combine smarter advertisers, more advertisers, and better tools and tail keywords are much less valuable.

3. Consumers are smarter: Finally, let's not forget about the consumer. Remember the growth of AskJeeves - the "natural language" search engine. Consumers loved the fact that you could type in a question like "Where can I find low mortgage rates?" and get relevant results. Of course, it turned out that this was basically a gimmick - the system simply excluded noise words like "Where can I" and found results based on "find low mortgage rates." Consumers caught on to this, the novelty wore off, and now Ask has abandoned the "natural language" concept all-together and is just another search engine.

Apparently, consumers are getting smarter on Google and Yahoo as well. A friend of mine who has already poured through the millions of user queries inadvertently released by AOL, tells me that user "query length" (the number of tokens in a search) is getting shorter. This to me suggests that user recognize that you can basically get what you want on a search engine with a basic search, rather than a 20 word diatribe.

No doubt searchers will also gradually learn to eliminate noise words like "and" or "where", as well as to stop typing in searches like www.ebay.com into the search engines (did you know, by the way, that one of the top searches on Google is . . . "Google"? This will decline over time . . . I hope).

In any event, as searchers become smarter, the volume of long-tail searches seems to become smaller, thus reducing the need for huge effort into keyword research.

Conclusion

Combine search engines maximizing revenue by showing generic ad results, competitors who now understand the long-tail, and consumers who have a much better understanding of how search engines work, and I think it's safe to say that the Holy Grail of search engine marketing is no longer the keyword. No doubt keyword lists are important, but you won't be able to retire in Aruba just because you concatenated three adjectives, two cities names, and two suffixes.

This basically means two things: 1) that SEMers are going to have to get a lot smarter about the other elements of SEM - like ad text, landing pages, bid prices, analytics, and filtering and 2) that folks who survived on the long-tail and market inefficiencies are in trouble (especially when you add in Google's Quality Score changes).

A few months back, I wrote about the similarities between SEM experts and eBay power sellers, the concept being that both eBay and SEM used to be easy for anyone to do but is now rapidly becoming the domain of specialized experts. The end of keywords is yet another example of this phenomenon for SEM.

Random Observations from SES in San Jose

Ah, SES week. A week filled with parties, free lunches, colorful business cards and the always interesting Google Dance. I learned a lot of interesting things this week - some of which I'll share today, others of which may merit a full column a bit later on.

In no particular order though, here are some of my observations about SES:

1. A Standing Ovation at Google: When you entered the Google Dance this year, you wear greeted by dozens of Googlers (mostly temp-to-perm client service reps) who gave you a standing ovation and shouted "thank you." I had two (not surprisingly cynical) observations about this greeting. First, I thought: you graduate from Stanford with honors, and your first job is a temp answering phones by day and applauding by night? Leland must be proud! The second was that the greeting reminded me of the "Arbeit Macht Frei" signs at Auschwitz. The slogan means "work will set you free." Of course, that was meant to put new arrivals at ease and to shield them from the truth. As Google rolls out a overly-broad Quality Score algorithm that will literally kill thousands of their most loyal and long-standing customers, I couldn't help feel that the standing ovation I received rang a bit hollow.

2. . . . But Save the White Gloves for the Press. Once past the Stanford cheering section, I waited in line for my little yellow wristband. Here's the interesting part - at SES in San Jose, I decided to be a good Samaritan and offer a ride to some random folks who were waiting in the 500 person deep line for buses to the Google dance. The guys I pulled out of the crowd ended up being two technology journalists for USA Today (actually, one of them was the guy who wrote the big spread on Danny Sullivan a few weeks ago, the other was once the manager of the first underground radio station in the US - KSAN in Berkeley!). So when a co-worker, the two journalists, and I arrived at the Google Dance, we were suddenly surrounded by several Googlers with clipboards. For a split second, I thought maybe I had been randomly selected for some sort of prize or something. It turns out, however, that they were only interested in our journalist friends."Can I get you your wristband? Do you need a tour of the party? Is there anyone you want to talk to you? What can I do to help." In other words, the advertisers who were getting the standing ovation were clearly not as important as the almighty press, who got the white glove treatment.

3. Big Plates and Small Plates. OK, one final Google Dance observation. When I got in line for the food, I noticed that the plates were "appetizer-sized" at best. I mean, these plates made Las Vegas buffet plates look huge. The next day, however, I got to go to lunch at "Oasis" which I guess is the newest Google cafeteria. The plates there were practically serving platters. And every table had fresh bread and a nice cheese plate. They also had a delicious heirloom tomato salad, tuna steaks, and flank steak. Does the size of a plate mean anything? Hmm, you decide.

4. The Show Seemed Dead to Me: OK, as to the show itself, I actually only went to the exhibit hall. I gotta say, it seemed like a ghost town. Especially compared to Ad-Tech San Francisco,SES was virtually empty. It reminded me of a trade show I went to in 2001 after the bubble burst. You kind of wanted to go talk to exhibitors out of pity. One more thing: why does Google exhibit at search trade shows anymore? Does anyone actually go to SES without having a Google account?

5. Domain Parks are Relevant but Affiliates Aren't. At the DoubleClick party, I had some fun playing bocce with Geoff the Bocce Superstar, but I also had a few minutes of conversation with a very visible Googler (I am going to withhold the name - gotta protect my sources!). I mentioned to him that I thought that if Google was going after MFAs and affiliate sites with their new Quality Score algorithm, it only made sense that they would eventually get rid of the Google Domain Park. After all, if you take the absolute worst affiliate Web site and compare to a virtually blank page with AdSense on it, that only gets traffic by capitalizing on misspellings of a trademarked name, surely the cyber-squatter presents a much worse user experience, right? Well, apparently not. The official Google line I guess is that there are "many domain names" that are common English language terms, and that when someone types one of these terms into the browser bar, they are taken to a page with highly relevant advertisements. I would translate this response a slightly different way: there's too much money in domain parking for Google to do anything about it.

6. MSN made Yahoo an acquisition offer over Christmas? Another little birdy told me that MSN made Yahoo a formal acquisition offer last Christmas, but Yahoo rejected it because it was too low. I still think that Yahoo and eBay are the most likely merger candidates, but MSN does have a lot of money to throw around. One person also told me that he thought that eBay was a likely acquisition candidate for Google. If that happened, that would be downright scary. Talk about Internet dominance.

7. Should SEMs root for the underdogs? SEMs must have a rough time at trade shows. It's virtually impossible to really understand the differences between all of them. Moreover, the value they add is often difficult to measure anyway. How much incremental lift can you achieve in your ROI by having some folks in New Jersey do your SEM versus you and a nice Excel spreadsheet? One thing that occurred to me though, is that if the PPC market becomes more diverse in terms of publishers - if Ask, MSN, Yahoo, and maybe one other provider can actually grow as opposed to lose marketshare to Google - the role of an SEM can actually become quite important, simply because it would be very laborious to actually manage 5-6 different user interfaces.

8. Mr. X and the War at Google: A good friend of mine who started at Google in around 2002 (and has now left) had some interesting observations about the war inside Google between "Larry, Sergei and the engineers" and the "business side." Apparently the engineers think that most marketers are "scum" and should be avoided like the plague. The only legitimate marketers are the big brand advertisers or direct sellers of products. In other words, everyone from affiliates, to lead generation companies, to comparison shopping engineers are nothing but leaches on the system. The business side, of course, disagrees, and thinks that with the right rules in place, AdWords and AdSense should take all comers. Until recently, the business side has been winning. But apparently that is changing, and that's partly due to the good work the business side has done in attracting more and more big advertisers. Now that the brand advertisers are getting into SEM, the engineers at Google can argue that the financial impact of booting out the "scum marketers" will be minimal if at all. And the good news is, even if the engineers are wrong, the "Quality Score" algorithm is so ambiguous that it can always be changed later on to allow the scum back in to help hit those quarterly numbers!

9. SES Milan? It looks like there are something like 20 SES shows a year now - Milan, London, Germany, Japan, China, New York, Chicago, Miami, San Jose, etc, etc. Is this really necessary? I could see San Jose, New York, London and maybe Japan. But Milan? Maybe that explains some of the low attendance at SES San Jose - burn out!

10. YPN coming soon? Finally, over at the Yahoo booth I was told that YPN was going to get out of beta "very soon" and start kicking AdSense Ass. Of course, I was also told that Panama would be released on August 15th. I'm betting that by February, YPN and Panama will still be 'in the works.'