Getting listed in search engines is the biggest problem a beginning Web site owner or blogger can face. The three
major Web search engines are Google, Yahoo! and MSN (now called "Live"). You can submit at least the main page to each
of them, and to Yahoo affiliates around the world. If you are listed in those three search engines, all the others will
eventually find your site, and if they don't it may not matter. There are hundreds of search engines that are set up for
various purposes, some of them exploitative. Google, Yahoo and Live generate most of the traffic.
For Google, go to:
You can also claim your site at Yahoo site explorer, which might induce the search engine spider to crawl it faster.
In order to be listed in Google, a page or Web site should be linked from somewhere else on the Web. Some other
search engines require (or allow you) to submit your site. Yahoo changes its policy from time to time, sometimes
allowing free submissions and sometimes charging for them. Google also accepts submissions or accepted them in the past,
but never guaranteed to list the site. Google and other search engines now accept XML site maps that are prepared in a
special format and show every page in the site. It is not clear if they use this information or how they use it, and it
is certainly impractical to resubmit the map every time you add a new page.
Once your site is well established, you will find that Google indexes new pages fairly quickly - often within a day.
Yahoo and MSN seem to be much slower at present.
The main page of your site must usually have a link somewhere on the Web in an existing page, so that the spider can
follow it. Every page in your Web site must be linked from a page that links to the main page so that spiders - the
robot software that crawl the Web - can find those pages. Usually, for small to medium sized site, a site map or series
of site maps should have full path html links to every major section in the site and possibly to every single page. A
site map page should not have more than 100 or 200 pages linked from it.
Fortunately, there are ways to ensure that your main page is linked in. The Web hosting company may have a
"directory" that lists all of the Web sites it hosts, and that listing will eventually be found by search engines. A Web
log that is created in blogger.com and other free blogging hosts also will be listed somewhere by those hosts. If your
posts create an RSS syndication feed that is submitted to technorati and other RSS aggregators then they will be listed
there and picked up eventually by spiders.
It is important to submit your site to directories, especially
However, Dmoz reviewers are not going to
like brand new Web sites that are mostly "under construction, so submission to directories has to wait until a
substantial part of your site is built.
Getting lower level "deep links" listed for a new Web site is even harder than getting the main page linked
sometimes. One technique that seems to work is to create a free Web log at blogger.com and/or www.bloglines.com/. It
doesn't much matter if no human reads those posts - evidently search engine spiders do. List your pages there in several
blog posts and you will probably find yourself listed in Google.
Do not waste money on "services" that submit your page to "thousands of search engines." The search engines are
usually non-functional or not used by many people. All the major search engines seem to sell databases to others or
share them in one way or another. Free search engine submission tools may be useful for getting deep listing for pages.
Just for bloggers - Most Web logs have automatic software that knows how to ping aggregators like technorati
automatically, but some older Web log software does not. Pingomatic.com is a useful tool for pinging several aggregators
at once, though it does not always work correctly.
Directory listings - Be sure to list your site in directories - see Web
Directory Listings for details.
