GOOGLE PAGE RANK AND ADVENTURE BUSINESS

Via John Battelle, Rick Skrenta’s remarkable piece on what Google have actually built. They don’t just have the world’s best search engine, they have the world’s largest and most scalable platform for developing huge web-based applications.

Google has taken the last 10 years of systems software research out of university labs, and built their own proprietary, production quality system. What is this platform that Google is building? It’s a distributed computing platform that can manage web-scale datasets on 100,000 node server clusters. It includes a petabyte, distributed, fault tolerant filesystem, distributed RPC code, probably network shared memory and process migration. And a datacenter management system which lets a handful of ops engineers effectively run 100,000 servers. Any of these projects could be the sole focus of a startup.

[ ... ]

While competitors are targeting the individual applications Google has deployed, Google is building a massive, general purpose computing platform for web-scale programming.

Search Quality is the name of the team responsible for the ranking of Google search results. Our job is clear: A few hundreds of millions of times a day people will ask Google questions, and within a fraction of a second Google needs to decide which among the billions of pages on the web to show them -- and in what order. Lately, we have been doing other things as well. But more on that later.

For something that is used so often by so many people, surprisingly little is known about ranking at Google. This is entirely our fault, and it is by design. We are, to be honest, quite secretive about what we do. There are two reasons for it: competition and abuse. Competition is pretty straightforward. No company wants to share its secret recipes with its competitors. As for abuse, if we make our ranking formulas too accessible, we make it easier for people to game the system. Security by obscurity is never the strongest measure, and we do not rely on it exclusively, but it does prevent a lot of abuse.

The details of the ranking algorithms are in many ways Google's crown jewels. We are very proud of them and very protective of them. By some estimate, more than one thousand programmer/scientist years have gone directly into their development, and the rate of innovation has not slowed down.

But being completely secretive isn’t ideal, and this blog post is part of a renewed effort to open up a bit more than we have in the past. We will try to periodically tell you about new things, explain old things, give advice, spread news, and engage in conversations. Let me start with some general pieces of information about our group. More blog posts will follow.

I should take a moment to introduce myself. My name is Udi Manber, and I am a VP of engineering at Google in charge of Search Quality. I have been at Google for over two years, and I have been working on search technologies for almost 20 years.

The heart of the group is the team that works on core ranking. Ranking is hard, much harder than most people realize. One reason for this is that languages are inherently ambiguous, and documents do not follow any set of rules. There are really no standards for how to convey information, so we need to be able to understand all web pages, written by anyone, for any reason. And that's just half of the problem. We also need to understand the queries people pose, which are on average fewer than three words, and map them to our understanding of all documents. Not to mention that different people have different needs. And we have to do all of that in a few milliseconds.

The most famous part of our ranking algorithm is PageRank, an algorithm developed by Larry Page and Sergey Brin, who founded Google. PageRank is still in use today, but it is now a part of a much larger system. Other parts include language models (the ability to handle phrases, synonyms, diacritics, spelling mistakes, and so on), query models (it's not just the language, it's how people use it today), time models (some queries are best answered with a 30-minutes old page, and some are better answered with a page that stood the test of time), and personalized models (not all people want the same thing).

Another team in our group is responsible for evaluating how well we're doing. This is done in many different ways, but the goal is always the same: improve the user experience. This is not the main goal, it is the only goal. There are automated evaluations every minute (to make sure nothing goes wrong), periodic evaluations of our overall quality, and, most importantly, evaluations of specific algorithmic improvements. When an engineer gets a new idea and develops a new algorithm, we test their ideas thoroughly. We have a team of statisticians who look at all the data and determine the value of the new idea. We meet weekly (sometimes twice a week) to go over those new ideas and approve new launches. In 2007, we launched more than 450 new improvements, about 9 per week on the average. Some of these improvements are simple and obvious -- for example, we fixed the way Hebrew acronym queries are handled (in Hebrew an acronym is denoted by a (") next to the last character, so IBM will be IB"M), and some are very complicated -- for example, we made significant changes to the PageRank algorithm in January. Most of the time we look for improvements in relevancy, but we also work on projects where the sole purpose is to simplify the algorithms. Simple is good.

International search has been one of our key focus areas in the past two years. This means all spoken languages, not just the major ones. Last year, for example, we made major improvements in Azerbaijani, a language spoken by about 8 million people. In the past few months, we launched spell checking in Estonian, Catalan, Serbian, Serbo-Croatian, Ukranian, Bosnian, Latvian, Filipino Tagalog, Slovenian and Farsi. We organized a network of people all over the world who provide us with feedback, and we have a large set of volunteers from all parts of Google who speak different languages and help us improve search.

Another team is dedicated to new features and new user interfaces. Having a great engine is necessary for a great car, but it is not sufficient. The car has to be comfortable and easy to drive. The Google search user interface is quite simple. Very few of our users ever read our help pages, and they can do very well without them (but they're good reading nevertheless, and we're working to improve them). When we add new features we try to ensure that they will be intuitive and easy to use for everyone. One of the most visible changes we made in the past year was Universal Search. Others include the Google Notebook, Custom Search Engines, and of course, many improvements to iGoogle. The UI team is helped by a team of usability experts who conduct user studies and evaluate new features. They travel all over the world, and they even go to people's homes to see users in their natural habitat. (Don't worry, they do not come unannounced or uninvited!)

There is a whole team that concentrates on fighting webspam and other types of abuse. That team works on variety of issues from hidden text to off-topic pages stuffed with gibberish keywords, plus many other schemes that people use in an attempt to rank higher in our search results. The team spots new spam trends and works to counter those trends in scalable ways; like all other teams, they do it internationally. The webspam group works closely with the Google Webmaster Central team, so they can share insights with everyone and also listen to site owners.

There are other teams devoted to particular projects. In general, our organizational structure is quite informal. People move around, and new projects start all the time.

One of the key things about search is that users' expectations grow rapidly. Tomorrow's queries will be much harder than today's queries. Just as Moore's law governs the doubling of computing speed every 18 months, there is a hidden unwritten law that doubles the complexity of our most difficult queries in a short time. This is impossible to measure precisely, but we all feel it. We know we cannot rest on our laurels, we have to work hard to meet the challenge. As I mentioned earlier, we will continue providing you with updates on search quality in the coming months, so stay tuned.

In May, Udi Manber introduced our search quality group, the group responsible for the ranking of search results. He introduced various teams within "Quality" (as we like to call the group) including Core Ranking, International Search, User Interfaces, Evaluation, Webspam, and other teams. In this post, I want to tell you more about one of these: the Core Ranking team.

Let me introduce myself. My name is Amit Singhal. I'm a Google Fellow in charge of the ranking team at Google. I've worked in the field of search for the past eighteen years, having been introduced to search in 1990 as a graduate student in computer science. In the academic world, the field of search is known as Information Retrieval (or IR). After spending a decade as an IR researcher, I came to Google in 2000, and have worked on Google ranking ever since.

Google ranking is a collection of algorithms used to find the most relevant documents for a user query. We do this for hundreds of millions of queries a day, from a collection of billions and billions of pages. These algorithms are run for every query entered into most of Google's search services. While our web search is the most used Google search service and the most widely known, the same ranking algorithms are also used - with some modifications - for other Google search services, including Images, News, YouTube, Maps, Product Search, Book Search, and more.

The most common question I get asked about Google's ranking is "how do you do it?" Of course, there is a lot that goes into building a state-of-the-art ranking system like ours, and I will delve deeper into the technology behind it in a later post. Today, I would like to briefly share the philosophies behind Google ranking:

1) Best locally relevant results served globally.
2) Keep it simple.
3) No manual intervention.

The first one is obvious. Given our passion for search, we absolutely want to make sure that every user query gets the most relevant results. We often call this the "no query left behind" principle. Whenever we return less than ideal results for any query in any language in any country - and we do (search is by no means a solved problem) - we use that as an inspiration for future improvements.

The second principle seems obvious. Isn't it the desire of all system architects to keep their systems simple? Well, as search systems go, given the wide variety of user queries we have to respond to in multiple languages, it is easy to go down the path where more and more complexity creeps into the system to serve the next incremental fraction of the queries. We work very hard to keep our system simple without compromising on the quality of results. This is an ongoing effort, and a worthy one. We make about ten ranking changes every week and simplicity is a big consideration in launching every change. Our engineers understand exactly why a page was ranked the way it was for a given query. This simple understandable system has allowed us innovate quickly, and it shows. The "keep it simple" philosophy has served us well.

No discussion of Google's ranking would be complete without asking the common - but misguided! :) - question: "Does Google manually edit its results?" Let me just answer that with our third philosophy: no manual intervention. In our view, the web is built by people. You are the ones creating pages and linking to pages. We are using all this human contribution through our algorithms. The final ordering of the results is decided by our algorithms using the contributions of the greater Internet community, not manually by us. We believe that the subjective judgment of any individual is, well ... subjective, and information distilled by our algorithms from the vast amount of human knowledge encoded in the web pages and their links is better than individual subjectivity.

The second reason we have a principle against manually adjusting our results is that often a broken query is just a symptom of a potential improvement to be made to our ranking algorithm. Improving the underlying algorithm not only improves that one query, it improves an entire class of queries, and often for all languages. I should add, however, that there are clear written policies for websites recommended by Google, and we do take action on sites that are in violation of our policies or for a small number of other reasons (e.g. legal requirements, child porn, viruses/malware, etc).

Stay tuned for my followup post, where I will discuss in detail the technologies behind our ranking and show examples of several state-of-the-art ranking techniques in action. Let me just conclude this post by saying, our passion for search is stronger than ever - and as a search researcher, I have the best job in the world.

In my previous post, I introduced the philosophies behind Google ranking. As part of our effort to discuss search quality, I want to tell you more about the technologies behind our ranking. The core technology in our ranking system comes from the academic field of Information Retrieval (IR). The IR community has studied search for almost 50 years. It uses statistical signals of word salience, like word frequency, to rank pages. (See "Modern Information Retrieval: A Brief Overview" for a quick overview of IR technology.) IR gave us a solid foundation, and we have built a tremendous system on top using links, page structure, and many other such innovations.

Search in the last decade has moved from give me what I said to give me what I want. User expectations from search have rightly increased. We work hard to fulfill the expectations of each and every user, and to do that we need to better understand the pages, the queries, and our users. Over the last decade we have pushed the technologies for understanding these three components (of the search process) to completely new dimensions.

When we talk about queries at Google, we use square brackets [ ] to mark the beginning and end of queries (see "How to write queries" by Matt Cutts), a notation I will use throughout this post. (Pages and search results change frequently, so in time, some examples used here may not behave as explained.)

• Understanding pages: Over years we have invested heavily in our crawl and indexing system. As a result we have a very large and very fresh index. In addition to size and freshness, we have improved our index in other ways. One of the key technologies we have developed to understand pages is associating important concepts to a page even when they are not obvious on the page. We find the official homepage for Sprovieri Gallery in London for the Italian query [galleria sprovieri londra], even though the official page does not have either London or Londra on it. In the U.S., a user searching for [cool tech pc vancouver, wa] finds the homepage www.cooltechpc.com even though the page does not mention anywhere that they are in Vancouver, WA. Other technologies we have developed include distinctions between important and less important words in the page and the freshness of the information on the page.
  

• Understanding queries: It is critical that we understand what our users are looking for (beyond just the few words in their query). We have made several notable advances in this area including a best-in-class spelling suggestion system, an advanced synonyms system, and a very strong concept analysis system.

• Understanding users: Our work on interpreting user intent is aimed at returning results people really want, not just what they said in their query. This work starts with a world class localization system, and adds to it our advanced personalization technology, and several other great strides we have made in interpreting user intent, e.g. Universal Search.
  

Finally let me briefly mention the latest advance we have made in search: Cross Language Information Retrieval (CLIR). CLIR allows users to first discover information that is not in their language, and then using Google's translation technology, we make this information accessible. I call this advance: give me what I want in any language. A user looking for Tony Blair's biography in Russia who types the query in Russian [Тони Блэр биография] is prompted at the bottom of our results to search the English web with:

Similarly a user searching for Disney movie songs in Egypt with the query [أغاني أفلام ديزني] is prompted to search the English web. We are very excited about CLIR as it truly brings us closer to our mission to organize the world's information and make it universally accessible and useful.

I could go on and on showing examples of state-of-the-art technology that we have developed to make our ranking system as good as it is, but the fact is that search is nowhere close to being a solved problem. Many queries still don't get satisfactory results from Google, and each such query is an opportunity to improve our ranking system. I am confident that with numerous techniques under development in our group, we will make large improvements to our ranking algorithms in the near future.

I hope my two posts about Google ranking have made it clear that we live and breathe search, and we are more passionate than ever about it. Our fervor for serving all our users worldwide is unprecedented. We pride ourselves in running a very good ranking system, and are working incredibly hard every day to make it even better.

A few weeks back Udi Manber introduced the search quality group, and the previous posts in this series talked about the ranking of documents. While the ranking of web documents forms the core of what makes search at Google work so well, your search experience consists of much more than that. In this post, I'll describe the principles that guide our development of the overall search experience and how they are applied to the key aspects of search. I will also describe how we make sure we are on the right track through rigorous experimentation. And the next post in this series will describe some of the experiments currently underway.

Let me introduce myself. I'm Ben Gomes, and I've been working on search at Google since 1999, mostly on search quality. I've had the good fortune to contribute to most aspects of the search engine, from crawling the web to ranking. More recently, I've been responsible for the engineering of the interface for search and search features.

A common reaction from friends when I say that I now work on Google's search user interface is "What do you do? It never changes." Then they look at me suspiciously and tell me not to mess with a good thing. Google is fine just the way it is -- a plain, fast, simple web page. That's great, but how hard can that be?"

To help answer that question, let me start with our main goal in web search: to get you to the web pages you want as quickly as possible. Search is not an end in itself; it is merely a conduit. This goal may seem obvious, but it makes a search engine radically different from most other sites on the web, which measure their success by how long their users stay. We measure our web search success partly by how quickly you leave (happily, we hope!). There are several principles we use in getting you to the information you need as quickly as possible:

• A small page. A small page is quick to download and generally faster for your browser to display. This results in a minimalist design aesthetic; extra fanciness in the interface slows down the page without giving you much benefit.
  
• Complex algorithms with a simple presentation. Many search features require a great deal of algorithmic complexity and a vast amount of data analysis to make them work well. The trick is to hide all that complexity behind a clean, intuitive user interface. Spelling correction, snippets, sitelinks and query refinements are examples of features that require sophisticated algorithms and are constantly improving. From the user's point of view search, almost invisibly, just works better.
• Features that work everywhere. Features must be designed such that the algorithms and presentation can be adapted to work in all languages and countries. Consider the problem of spell correction in Chinese, where user queries are often not broken up into words or Hebrew/Arabic, where text is written right to left (interestingly, this is believed to be an example of first-mover disadvantage -- when chiseling on stone, it is easier to hold the hammer in your right hand!).
  
• Data driven decisions - experiment, experiment, experiment. We try to verify that we've done the right thing by running experiments. Designs that may seem promising may end up testing poorly.

There are inherent tensions here. For instance, showing you more text (or images) for every result may enable you to better pick out the best result. But a result page that has too much information takes longer to download and longer to visually process. So every piece of information that we add to the result page has to be carefully considered to ensure that the benefit to the user outweighs the cost of dealing with that additional information. This is true of every part of the search experience, from typing in a query, to scanning results, to further exploration.

The start of your search is typing in a query. A common cause of frustration is if you don't know the correct spelling of a word! Spell correction -- which seems like a simple and obvious feature -- hides many technical challenges. No common English dictionaries would ever include the correct spelling of Britney Spears, for instance (who, probably completely unbeknownst to her, has become the poster child example for this feature). We do a huge amount of analysis of the billions of pages on the web and our query logs to determine what are "real words" on the web, and what are likely to be misspellings. The system that gives you the spell correction has to, in a fraction of a second, consider a huge number of possible words you might have meant (vastly greater than any dictionary ever manually constructed) and determine if there is a more likely query you meant to type. When we are confident that you actually meant to type something else, we take a rare liberty with our search results: we try to distract you from looking at the top result on the page. The spelling correction is in your line of sight and colored a bright must-see red. Furthermore, we now make sure that nothing else on the page is red, unless it is as important to you as spelling! (so far, nothing is). The algorithms involved in spell correction are constantly getting better. They now work in a large number of languages and are even better at detecting when you have made a spelling mistake. Getting the spelling of your query right is so important that we are considering showing you the results of the spell-corrected query in the middle of the page (just in case you missed our bright red text at the top and bottom!).

Having formulated your query correctly, the next task is to pick a page from the result list. For each result, we present the title and url, and a brief two line snippet. Pages that don't have a proper title are often ignored by users. One of the bigger recent changes has been to extract titles for pages that don't specify an HTML title -- yet a title on the page is clearly right there, staring at you. To "see" that title that the author of the page intended, we analyze the HTML of the page to determine the title that the author probably meant. This makes it far more likely that you will not ignore a page for want of a good title. Below the title comes the snippet, and a key early innovation was in what Google showed for the snippet. At the time, search engines showed you the first two lines of the web page; Google, instead, showed you parts of the page where your actual search keywords showed up (information retrieval experts call this "keywords-in-context"). Showing keywords-in-context is visually simple and virtually indistinguishable from the simpler style of snippets, but vastly more useful in helping you decide which page to visit. This simplicity belies underlying complexity: when we create a snippet we have to go through the actual text from each result to find the most relevant part (which contain your keywords) rather than just giving you the first few lines.

We have been making improvements to our snippets over time with algorithms for determining the relevance of portions of the page. The changes range from the subtle -- we highlight synonyms of your query terms in the results -- to more obvious. Here's an example screenshot where the user searched for "arod" and you can see that Alex and Rodriguez are bolded in the search result snippet, based on our analysis that you might plausibly be referring to him:

As a more obvious example, we now extract and show you the byline date from pages that have one. These byline dates are expressed in a myriad formats which we extract and present uniformly, so that you can scan them easily:

For one of the most common types of user needs, navigational queries -- where you type in the name of a web site you know -- we have introduced shortcuts (we refer to them as sitelinks). These sitelinks allow you to get to the key parts of the site and illustrate many of the same principles alluded to above; they are a simple addition to the top search result that adds a small amount of extra text to the page.

For instance, the home page of Hewlett-Packard has almost 60 links, in a two-level menu system. Our algorithms, using a combination of different signals, pick the top ones among these that we think you are most likely to want to visit.

What if you did not find what you were looking for among the top results? In that case, you probably need to try another query. We help you in this process by providing a set of query refinements at the bottom of the results page -- even if they don't give you the query that you need, they provide hints for different (likely more successful) directions in which you could refine your query. By placing the query refinements at the bottom of the page, the refinements don't distract users, but are there to help if the rest of the search results didn't serve a user's information need.

I've described several key aspects of the search experience, including where we have made many changes over time -- some subtle, some more obvious. In making these changes to the search experience, how do we know we've succeeded, that we've not messed it up? We constantly evaluate our changes by sharing them with you! We launch proposed changes to a tiny fraction of our users and evaluate whether it seems to be helping or hurting their search experience. There are many metrics we use to determine if we've succeeded or failed. The process of measuring these improvements is a science in itself, with many potential pitfalls. Our experimental methodology allows us to explore a range of possibilities and launch the ones that work the best. For every feature that we launch, we have frequently run a large number of experiments that did not see the light of day.

So let me answer the question I started with: We're actually constantly changing Google's result page and have been doing so for a long time. And no, we won't mess with a good thing. You won't let us.

In the next post in this series, I'll talk about some of the experiments we are running, and what we hope to learn from them.

In my previous post, I described the components of your web search experience and the principles behind creating a great search experience. There are complex algorithms underlying simple features such as spelling correction and the two line snippets that describe each search result. We figure out what works by running experiments - tiny tests for a small number of users which help us determine whether that feature helps or hurts.

Experimentation is a very powerful tool, and we use it very widely to test potential changes to search. At any given time, we run anywhere from 50 to 200 experiments on Google sites all over the world. I'll start by describing experimental changes so small that you can barely tell the difference after staring at the page, and end with a couple of much more visually obvious experiments that we have run. There are a lot of people dedicated to detecting everything Google changes - and occasionally, things imagined that we did not do! - and they do latch on to a lot of our more prominent experiments. But the experiments with smaller changes are almost never noticed.

For example, can you tell the difference between the two pages below?

Choice 1:

Choice 2:

I'm pretty sure I would not be able to tell the difference if I were to see each of them on their own. But apparently you can! At least in the aggregate, there is a measurable difference with a change like this. In case you can't tell after staring, the white space around the first search result has changed, which makes the first result in Picture 2 slightly more visually prominent. This visual prominence conveys the fact that according to our ranking signals, the first result is a substantially better match than the next result. On the plus side, it helps you focus on the first result. But if you were looking for one of the other results, it can disrupt your scanning of the page. An experiment helps us determine which effect is more prominent, and whether a change would help you search faster.

Another change, almost as minimal visually, is between these two results:


At this point, I can't say what we expect from this feature; we're just curious to see how it will be used.

These are a small sample of the kinds of experiments we run as we test everything from the barely visible to the glaringly obvious. So the next time you use Google and it seems a little different - well, maybe it is. Just for you!

This series of posts has described Google's search quality efforts in areas such as ranking and search UI. Now I'll describe search evaluation. Simply put, search evaluation is the process of measuring the quality of our search results and our users' experience with search.

Let me introduce myself. I'm Scott Huffman, an engineering director responsible for leading search evaluation, working with a talented team of statisticians and software engineers. I've been here since 2005, and have been working on search in one form or another for the past fourteen years or so.

When I'm interviewing folks interested in joining the search evaluation team, I often use this scenario to describe what we do: Imagine a Google ranking engineer bursts into your office. "I have a great idea for improving our search results!" she exclaims. "It's simple: Whenever a page's title starts with the letter T, move it up in the results three slots." This engineer comes armed with several example search queries where, lo and behold, this idea actually improves the results significantly.

Now, you and I may think that this "letter T" hack is really a silly idea, but how can we know for sure? Search evaluation is charged with answering such questions. This hack hasn't really come up, but we are constantly evaluating everything, which can include:

To get at these issues, we employ a variety of evaluation methods and data sources:

• Human evaluators. Google makes use of evaluators in many countries and languages. These evaluators are carefully trained and are asked to evaluate the quality of search results in several different ways. We sometimes show evaluators whole result sets by themselves or "side by side" with alternatives; in other cases, we show evaluators a single result at a time for a query and ask them to rate its quality along various dimensions.
  
  
• Live traffic experiments. We also make use of experiments, in which small fractions of queries are shown results from alternative search approaches. Ben Gomes talked about how we make use of these experiments for testing search UI elements in his previous post. With these experiments, we are able to see real users' reactions (clicks, etc.) to alternative results.
  

Clearly, we can never measure anything close to all the queries Google will get in the future. Every day, in fact, Google gets many millions of queries that we have never seen before, and will never see again. Therefore, we measure statistically, over representative samples of the query-stream. The "letter T" hack probably does improve a few queries, but over a representative sample of queries it affects, I'm confident it would be a big loser.

One of the key skills of our evaluation team is experimental design. For each proposed search improvement, we generate an experiment plan that will allow us to measure the key aspects of the change. Often, we use a combination of human and live traffic evaluation. For instance, consider a proposed improvement to Google's "related searches" feature to increase its coverage across several locales. Our experiment plan might include live traffic evaluation in which we show the updated related search suggestions to users and measure click-through rates in each locale and break these down by position of each related search suggestion. We might also include human evaluation, in which for a representative sample of queries in each locale, we ask evaluators to rate the appropriateness, usefulness, and relevance of each individual related search suggestion. Including both types of evaluation allows us to understand the overall behavioral impact on users (via the live traffic experiment), and measure the detailed quality of the suggestions in each locale along multiple dimensions (via the human evaluation experiment).

Choosing an appropriate sample of queries to evaluate can be subtle. When evaluating a proposed search improvement, we consider not only whether a given query's results are changed by the proposal, but also how much impact the change is likely to have on users. For instance, a query whose first three results are changed is likely much higher impact than one for which results 9 and 10 are swapped. In Amit Singhal's previous post on ranking, he discussed synonyms. Recently, we evaluated a proposed update to make synonyms more aggressive in some cases. On a flat (non-impact-weighted) sample of affected queries, the change appeared to be quite positive. However, using an evaluation of an impact-weighted sample, we found that the change went much too far. For example, in Chinese, it synonymized "small" (小) and "big" (大)... not a good idea!

We're serious about search evaluation because we are serious about giving you the highest quality search experience possible. Rather than guess at what will be useful, we use a careful data-driven approach to make sure our "great ideas" really are great for you. In this environment, the "letter T" hack never had a chance.

In previous posts in this series, you have read about the challenges of building a world-class search engine. Our goal is to make Google’s search be relevant to all people, regardless of their language or country. As my colleague Amit Singhal described, we use statistical data as the basis for making sweeping algorithmic changes. Many of these changes can be rolled out across all languages we support, but in some cases the unique characteristics of each language require some algorithmic considerations and tuning. And to make things really interesting, there are cases where the same language is different across countries. Obvious examples are "color" in the U.S. vs. "colour" in the U.K., or "camião" in Portugal vs. "caminhão" in Brazil.

My name is Daphne Dembo, and my focus is improving Google's international search. This is a tough challenge, since Google search is used in many countries and languages where our engineers have little personal knowledge. Initially, the international search improvements were done by Search Quality engineers who were passionate about their languages and countries: Lina from Sweden improved our parsing of compound words in German and Swedish; Dimitra from Greece introduced diacritical support; Ishai from Israel worked on transliteration corrections for Hebrew and Arabic; Trystan from Australia created methods for identifying local search results and ranking them together with foreign ones from the same language; Alex, a bilingual Ukrainian and Russian, introduced morphological understanding of these languages. As the importance of our international search grew, we solicited help from Googlers in all our offices. Finally, we are leveraging an international network of search specialists who help us understand search within the unique combination of their language and country.

Our first step in providing search support for a language is to train our language model on a large collection of documents in that language. This ensures that our language model is more precise and comprehensive — for example, it incorporates names, idioms, colloquial usage, and newly coined words not often found in static dictionaries. For instance, we recently started identifying Swahili, and used pages such as this one for the Parliament of Tanzania to train our system with the language's nuances. Having a trained language model helps to categorize documents during crawling and indexing of the web and to parse the user's query. Once this stage was complete, we launched Swahili search in countries such as Tanzania and Kenya, enabling local searches for the "Dar es Salaam stock exchange" [Soko la hisa dar es salaam], and "cure for Malaria" [Tiba ya malaria]. (As always, we are using square brackets to denote a search query. For example, you can search for "soccer" in Hamburg, Germany by clicking on [fußball in hamburg]).

We learn some things from our users, so as people start using our search engine, we can improve the way we rank in that language. Here are few examples:

• Spell corrections: We recently launched spell corrections in Estonian. If your Estonian is rusty, and you don't remember how to spell "smoke detector," we can suggest a spell correction for [suitsuantur], leading to better search results.
• Diacritical marks: Many languages have diacritical marks, which alter pronunciation. Our algorithms are built to support them, and even help users who mis-type or completely ignore them. For example, if you're a resident of Quebec, Canada and would like to know the weather forecast in Quebec City, we'll serve good results whether you type with diacritical signs [Météo à Québec] or without [meteo quebec]. Czech users can read the same excellent results for a popular kids' cartoon by searching for [krtecek] and [krteček]. On the other hand, sometimes diacriticals change the meaning of the word and we have to use them correctly. For example, in Thai, [ข้าว] is "rice," with completely different results than [ข่าว], which is "news"; or in Slovakia, results for "child" [dieťa] are different than results for "diet" [diéta].
• Synonyms: A general case of diacritical support is the handling of synonyms in different languages. Korean searches showed that "samsung" can be viewed as a synonym of "삼성", so that when users search for [samsung], they find results which have the company's name in Korean.
• Compounding: Some languages allow compounding, which is the formation of new words by combining together existing words. You can see a nice example in Swedish, where we return documents about a Swedish credit card for both compounded [Visakort] and non-compounded [visa kort] queries.
• Stemming: Google has developed morphological models that can receive compound words as queries, and return pages which contain their stem, possibly as part of a different compound. For example, when searching for cars in Saudi Arabia, you can search for [سيارة] and [سيارات] because both are variants of the same stem, and both return many common results. A Polish user can search for "movie" [film], and get back results that contain other variants of the stem, such as "filmów," "filmu," "filmie," "filmy." A user from Belarus will find results for all word forms of the capital, Minsk [Мінск]: "Мінску," "Мінска," "Мінскага."

In addition to these semantic factors, Google does even more to parse documents and queries. Understanding the details of language usage in a country is important. Notation of acronyms is different across languages: In Hebrew it is double quotes before the last (left-most) character, as in "prime minister" [רה"מ]; in Thai — a dot at the end of the word, as in police station [สน. ]; while in the U.S. — dots after each character, as in [I.B.M.]. Chinese users quote works of art with a "《", as in: [《手机》剧情], and denote dates with a "日", as in: [2006年1月13日].

Beyond the linguistic elements of a language, we consider how people enter a query. For example, some languages that do not have Latin scripts require keyboards with dual alphanumeric keys. The user can switch between language input modes by typing special keystrokes. In case the user forgets to type this sequence, the queries end up being gibberish. You can see correct handling of these mistakes in Arabic ([hgsuv] corrected to [السعر]) and ([حقثسهيثىفهشم ثممثؤفهخىس ] corrected to [presidential elections]), Hebrew ([vdrk, kuyu] corrected to [הגרלת לוטו]), and Cyrillic ([rehc ljkffhf] corrected to [курс доллара]).

Another way of avoiding the inconvenience of switching keyboard modes is by typing the phonetic sounds of the query in Latin characters. Recreating the correct query in the target language isn't trivial, since there might be many possibilities. We can see several such examples in which we suggest the same query in the intended language for Russian ([biskvitnyi rulet] to [бисквитный рулет]), "movies" in Chinese ([dianying] to [电影]), and "Bank of Attica" in Greek [trapeza attikhs] returns good results for "Τράπεζα Αττικής". Users of 8 Indic languages (such as Hindi, Gujarati, Telugu) can type the phonetic sound of the query, and choose the words in Hindi script:


Ease of typing and reading is also influenced by the language used. Since every Chinese word requires several keystrokes on a standard keyboard, we provide category browsing by Images and related searches so that people don't need to type as much. Similarly, we are now launching Google Suggest, or real-time completion of queries, in many languages.

So far I described how we improve the quality of search in a language. However, there is a strong effect of the location of the user, even if it is only approximated to the country, since in many cases local content is more relevant than global information. For example, searching for Spanish Yellow Pages [Páginas Amarillas] will result in several documents of global interest and several local results in Peru, Mexico, and Spain. Similar to that, searching for [Côte d'Or] in France will return results for that region, whereas searches in Belgium will return results about the chocolate maker.

Note that the display of information should conform to the standards in that country, so we display "," as a decimal notation for Croatian users who want to know how many millimeters are in an inch [inč u milimetrima], or for Italian users who are interested in currency exchange rates [50 euro in dollari]. Similarly, temperatures in Norway [Været i Oslo] will be displayed in Celsius, while in the U.S. — in Fahrenheit [weather Boston].

If everything else fails, we provide cross-language translations based upon Google's translation technology described in this blog post. We will translate your query to English, search English documents on the web, and translate the returned results from English back into the original query language. For example, Japanese users who are interested in viewing Halloween illustrations (Halloween is a holiday which originated in Ireland) can search for [ハロウィン イラスト]. You can then request a Japanese translation of the English pages (at the bottom of the page), which will bring up the translation page in the screenshot below. Similarly, Korean users can search for the latest on Harry Potter [해리 포터], and Arabic readers can search for the opening of the Sydney Opera house [افتتاح دار الاوبرا في سيدني]. (Click on the image to see a larger version.)



All in all, Google Search is being actively developed for more than 100 languages, in 150+ countries, with dozens of improvements launched each month. So far I've covered the basics of how international search works, but this is just the surface of all the international work we do. There are many other interesting topics that impact international markets like usability, homepage and results page layout, and connectivity. An understanding of real cultural and human factors is essential to creating a search engine that resonates with the people who use it. (Click on the image to see a larger version.)

magine that you need a refresher on how to tie a tie. So, you decide to type [how to tie a tie] into the Google search box. Which of these results would you choose?


Where did your eyes go first when you saw the results page? Did they go directly to the title of the first result? Did you first check the terms in boldface to see if the results really talk about tying a tie? Or maybe the images captured your attention and drew your eyes to them?

You might find it difficult to answer these questions. You probably did not pay attention to where you were looking on the page and you most likely only used a few seconds to visually scan the results. Our User Experience Research team has found that people evaluate the search results page so quickly that they make most of their decisions unconsciously. To help us get some insight into this split-second decision-making process, we use eye-tracking equipment in our usability labs. This lets us see how our study participants scan the search results page, and is the next best thing to actually being able to read their minds. Of course, eye-tracking does not really tell us what they are thinking, but it gives us a good idea of which parts of the page they are thinking about.

To see what the eye-tracking data we collect looks like, let's go back to the results page we got for the query [how to tie a tie]. The following video clip shows in real time how a participant in our study scanned the page. And yes, seriously — the video is in real time! That's how fast the eyes move when scanning a page. The larger the dot gets, the longer the users' eye pauses looking at that specific location.



Based on eye-tracking studies, we know that people tend to scan the search results in order. They start from the first result and continue down the list until they find a result they consider helpful and click it — or until they decide to refine their query. The heatmap below shows the activity of 34 usability study participants scanning a typical Google results page. The darker the pattern, the more time they spent looking at that part of the page. This pattern suggests that the order in which Google returned the results was successful; most users found what they were looking for among the first two results and they never needed to go further down the page.


When designing the user interface for Universal Search, the team wanted to incorporate thumbnail images to better represent certain kinds of results. For example, in the [how to tie a tie] example above, we have added thumbnails for Image and Video results. However, we were concerned that the thumbnail images might be distracting and disrupt the well-established order of result evaluation.

We ran a series of eye-tracking studies where we compared how users scan the search results pages with and without thumbnail images. Our studies showed that the thumbnails did not strongly affect the order of scanning the results and seemed to make it easier for the participants to find the result they wanted.

The thumbnail image seemed to make results with thumbnails easy to notice when the users wanted them (see screenshots below — page with the thumbnail image on the right)...

For the Universal Search team, this was a successful outcome. It showed that we had managed to design a subtle user interface that gives people helpful information without getting in the way of their primary task: finding relevant information.

In addition to search research, we also use eye-tracking to study the usability of other products, such as Google News and Image Search. For these products, eye-tracking helps us answer questions, such as "Is the 'Top Stories' link discoverable on the left of the Google News page?" or "How do the users typically scan the image results — in rows, in columns or in some other way?"

Eye-tracking gives us valuable information about our users' focus of attention — information that would be very hard to come by any other way and that we can use to improve the design of our products. However, in our ongoing quest to make our products more useful, usable, and enjoyable, we always complement our eye-tracking studies with other methods, such as interviews, field studies and live experiments.

Today we're rolling out two new improvements to Google search. The first offers an expanded list of useful related searches and the second is the addition of longer search result descriptions -- both of which help guide users more effectively to the information they need.

More and better search refinements

Starting today, we're deploying a new technology that can better understand associations and concepts related to your search, and one of its first applications lets us offer you even more useful related searches (the terms found at the bottom, and sometimes at the top, of the search results page).

For example, if you search for [principles of physics], our algorithms understand that "angular momentum," "special relativity," "big bang" and "quantum mechanic" are related terms that could help you find what you need. Here's an example (click on the images in the post to view them larger):

Let's look at a couple of examples in other languages. In Russian, for the query [гадание на картах] (fortune-telling with cards), the algorithms find the related terms "таро" (tarot), "ленорман" (lenormand) and "тибетское гадание мо" (tibetan divination mo). In Italian, if you search for [surf alle canarie] (surf at the canary islands), we now offer suggestions based on the three most famous Canary Islands: "lanzarote," "gran canaria," and "fuerteventura":

We are now able to target more queries, more languages, and make our suggestions more relevant to what you actually need to know. Additionally, we're now offering refinements for longer queries — something that's usually a challenging task. You'll be able to see our new related searches starting today in 37 languages all around the world.

And speaking of long queries, that leads us to our next improvement...

Longer snippets

When you do a search on Google, each result we give you starts with a dark blue title and is followed by a few lines of text (what we call a "snippet"), which together give you an idea of what each page is about. To give more context, the snippet shows how the words of your query appear on the page by highlighting them in bold.

When you enter a longer query, with more than three words, regular-length snippets may not give you enough information and context. In these situations, we now increase the number of lines in the snippet to provide more information and show more of the words you typed in the context of the page. Below are a couple of examples.

Suppose you were looking for information about Earth's rotation around the sun, and specifically wanted to know about its tilt and distance from the sun. So you type all of that into Google: [earth's rotation axis tilt and distance from sun]. A normal-length snippet wouldn't be able to show you the context for all of those words, but with longer snippets you can be sure that the first result covers all those topics. In addition, the extra line of snippets for the third result shows the word "sun" in context, suggesting that the page doesn't talk about Earth's distance from the sun:

Similarly, if you're looking for a restaurant review that covers all the parts of the meal, longer snippets can help:

But don't just take our word for it — try it out yourself with your favorite long, detailed query.

These are just two recent examples of improvements we've made. We are constantly looking for ways to get you to the web page you want as quickly as possible. Even if you don't notice all of our changes, rest assured we're hard at work making sure you have the highest quality search experience possible.