SEARCH ENGINE INDEXING - IMPORTANT
Before you can begin to understand how Search Engine Optimisation works it is important that have a grasp of how the search engines work. Some people think that when you search for something on the Internet the search engine you are using searches the whole World Wide Web at that time. This is not what happens. To do so would be highly impractical because of the amount of time it would take.
What actually happens is that the search engines work continually in the background, collecting text information from the web pages they "index", sorting it and storing it in databases on computers within their own data centres.
Google has many data centres located all over the world that store information from websites at these locations. When you query a search engine it searches its own index on its databases for the information and it can do this extremely quickly because the information is all organised to make it search friendly.
This is why the results of a search normally appear almost immediately.
SEARCH ENGINE INDEXES
HOW DOES YOUR WEBSITE GET INCLUDED IN A SEARCH ENGINE’S INDEX?
The web is just a system of links (or hyperlinks as they are properly known). Hyperlinks are the bits of text or images that you click on to go to another page or another website on the Internet. These may be on a navigation menu or just within the text content. Hypertext links can take you a new page on the website or to another part of the existing page.
Most search engines use "robots" or "crawlers" to crawl the web following these links from site to site. They store and sort the text information from the pages that they find as they do so. Robots and web crawlers are computer programs that have been specifically designed for this purpose. They can generally be tracked through your website visitor statistics and they all have different names, e.g. Google's robot is called the Googlebot. After a web page has been indexed by a search engine its robot will revisit it occasionally to see if the information had changed and if so it will record this change and update its database.
If a web page on another website that is already indexed by the search engines links to your site (known as an inbound link), then your site will be found and indexed automatically. You do not have to pay anyone to get your website included in search engines. This is particularly true with Google, which also uses Google Pagerank (basically an indication of the number of inbound links to a page) as one of the factors that determine where it should appear in the results. This means that inbound links are very important for SEO (Search Engine Optimisation) purposes.
When the robot finds a new web page it indexes it and adds the text content it finds there to its database. Depending on how highly the page is rated the robot will return at intervals to update the information that it finds there. On most sites this takes place every few days or weeks but popular dynamic sites (like the BBC for example) can be visited by the crawlers and updated every couple of minutes.
This is an important factor since the search results are determined by what the search engines have actually stored in their database (or "cache"), not by what is currently on the live page. The information on the website and that in the cache can therefore differ. This means that when a web page is updated the new content may not be found by anyone searching for it until the robot returns to the page and finds it.
If you have a large website you should create a sitemap and link to this from your homepage. Sitemaps (more later) are special pages with a list of links to all of the pages on your website that you want the search engines to find.
Getting Indexed by the Search Engines - Other Ways
There are other ways of getting indexed by the search engines. For example you can manually submit your site and wait for it to happen. This used to take weeks or even months to happen but nowadays it normally happens within a few days. The best method of getting into the search engines is to get inbound links and let the search engines find your site naturally. That is what the Internet is all about. Note that we are talking about getting included in the search engine indices here. This is different from actually gaining some sort of rank, which takes much a lot longer.
Note 1: You may see adverts on websites from companies offering to "Submit your site to 2,000 search engines", etc. Ignore these! They're really just a waste of time, because only Google and to a lesser extent Bing and Yahoo will deliver any significant traffic to your site.
Here’s a link to a great Google Video that explains the process of getting indexed in Google.
Next Page - The Search Engine Algorithm >
