Why Can't I Get Indexed by the Search Engines?
Unfortunately, this is an all too common question. If it makes you
feel any better, you're not the only one frustrated about the length
of time it takes to be indexed, or the many pitfalls involved. It
often takes anywhere from two days to as much as six months to be
listed on a search engine. For example, last month Excite finally
updated its index for the first time since last August! Luckily,
Excite is the most extreme case lately, but waiting several weeks to
a month can also be extremely frustrating.
The WebPosition Submitter report will give you current time estimates
for each engine so you'll know what to expect. However, an engine at
any time could choose to delay their indexing beyond the "norm" for
maintenance or other reasons. On the flip side, you could get lucky
and submit just a couple days before an engine does a complete
refresh of their database. Therefore, submission times can never be
an exact science since we're all ultimately at the mercy of the
If you've submitted your site and have waited the estimated time to
be indexed and there's still no listing, what do you do now?
Here are 16 tips that should help you solve this problem:
If any of the above scenarios apply to your submission, you should
make the necessary adjustments and re-submit. If that still does not
work, you should consider e-mailing or calling the search engine and
asking them politely why you have not been indexed yet. Sometimes
they will reply back with "Sorry, there was a problem with our system
and I've now made sure you'll be indexed within the next couple
days." Or, sometimes they'll tell you why you were not indexed. In
other cases, they will ignore your e-mail and you'll have to keep
e-mailing or calling them until they respond. Still, it's definitely
worth the effort to get your site listed with the major engines
assuming you also take the time to optimize your pages so you'll
achieve top rankings.
- First, be sure you're not already indexed but just don't know it.
Unfortunately, none of the major engines are kind enough to e-mail or
notify you as to if and when you've been indexed.
The method to determine if a page or domain has been indexed varies
from one engine to another, and in many cases, it's difficult to tell
for sure. Never assume that you're not indexed just because you
searched for a bunch of keywords and you never came up in the first
few pages of results. You could be in there but buried near the
In addition, it's not very practical to check the status of a number
of pages on each major engine each week. Fortunately, WebPosition has
a URL verification feature in the Reporter that makes this process
much easier. Each time you run a mission, it will report which URLs
exist and do not exist in each engine. If you're using WebPosition
and are not finding your URLs after submitting, be sure to see this
page for common pitfalls to watch out for:
- Make sure you have uploaded the pages to your site before
submitting them. This one seems obvious, but submitting a page that
does not exist or submitting with a subtle typo in the URL is a goof
we might all make at one time or another. If you're using
WebPosition's Submitter, there's a checkbox on tab 2 that forces
WebPosition to verify that all your URLs are valid before submitting
- If you have information inside frames, that can cause problems
with submissions. It's best if you can create non-framed versions
of your pages. You should then submit the non-frames versions of your
pages which can of course point to your framed Web site.
Alternatively, you can enter your relevant text within the NOFRAMES
area of a framed page which most search engine spiders will read.
- Search engine spiders cannot index sites that require any kind
of registration or password. A spider cannot fill out a form of any
kind. The same rule applies regarding indexing of content from a
searchable database, because the spider cannot fill out a form to
query that database. The solution is to create static pages that the
engines will be able to find.
- Dynamic pages often block spiders. In fact, any URL containing
special symbols like a question mark (?) or an ampersand (&) will be
ignored by many engines.
- Most engines cannot index text that is embedded in graphics.
Text that appears in multimedia files (audio and video) cannot be
indexed by most engines. Information that is generated by Java
applets or in XML coding cannot be indexed by most engines.
- If your site has a slow connection or the pages are very
complex and take a long time to load, it might time out before
the spider can index all the text. For the benefit of your visitors
and the search engines, limit your page size to less than 60K. In
fact, most Webmasters recommend that your page size plus the size of
all your graphics should not exceed 50K-70K. If it does, many people
on dial up connections will leave before the page fully loads.
- If you submit just your home page, don't expect a search engine
to travel more than one or two links away from the home page or the
page that you submitted. Over time they may venture deeper into your
site, but don't count on it. You'll often need to submit pages
individually that appear further down into your site or have no link
from the home page.
- If your Web site fails to respond when the search engine spider
pays a visit, you will not be indexed. Even worse, if you are indexed
and they pay a visit when your site is down, you'll often be removed
from their database! Therefore, it pays to have a reliable hosting
service that is up 99.5% of the time. However, at some point a spider
is going to hit that other 0.5% and end up yanking your pages by
mistake. Therefore, it pays to keep a close eye on your listings.
- If you have ever used any questionable techniques that might
be considered an overt attempt at spamming (i.e., excessive
repetition of keywords, same color text as background, or other
things that the WebPosition Page Critic warns you about), an
engine may ignore or reject your submissions. If you're having
trouble getting indexed in the expected amount of time, make sure
your site is spam-free.
- If your site contains redirects or meta refresh tags these things
can sometimes cause the engines to have trouble indexing your site.
Generally they will index the page that it is redirecting TO, but if
it thinks you are trying to "trick" the engine by using "cloaking" or
IP redirection technology, there's a chance that it may not index the
site at all.
- If you're submitting to a directory site like Yahoo, Open
Directory, NBCi, Looksmart, or others, then a human being will review
your site. They must decide the site is of sufficient "quality"
before they will list it. I recommend you read the submission guide
on the directory tab of the WebPosition Submitter. It contains tips
to improve your chances of obtaining a good listing on these
- A number of engines no longer index pages residing on many common
free web hosting services. The common complaint from the engines is
that they get too many "junk" or low-quality submissions from free
web site domains. Therefore, they often choose not to index anyone
from those domains or they limit submissions from them. It's always
best to buy your own domain name (very important) and place it on a
respected, paid hosting service to avoid being discriminated against.
- Some engines have been known to drop pages that cannot be
traveled to from the home page. HotBot has been rumored to do this.
You may want to consider submitting your home page that links either
directly or indirectly to your doorway pages.
- Make sure you're submitting within the recommended limits. Some
engines do not like more than a certain number of submissions per day
for the same domain. If you exceed the limit, you may find that all
your submissions are ignored. Fortunately, WebPosition's submitter
will warn you regarding current limits and recommend you stay within
them. Some submission consultants feel it is dangerous to submit more
than ONE page a day to a engine for a given Web site. For those who
wish to be ultra-conservative in their approach, the WebPosition
Submitter includes a checkbox to limit submissions to one URL per day
- Last but not least, sometimes the engines just lose submissions
at random through technical errors and bugs. Therefore, some people
like to resubmit once or twice a month for good merit in case they
do lose a submission. Certainly if you've followed all the "rules"
and are still not listed, re-submit! Sometimes a little persistence
is all that's needed.
The above article, or portions of it, have been reprinted with
permission from the MarketPosition Newsletter and FirstPlace
Software, Inc. and is copyright 1997-2001. FirstPlace produces
WebPosition Gold, the award-winning software product to track and to
improve your search engine rankings. You may download a FREE trial
copy of WebPosition Gold from: http://www.webposition.com
PRODUCT OF THE WEEK
Free Insider Report|
Receive free monthly marketing updates, tools, tips and strategies to help your online business succeed. Get immediate access to our bonus ebooks AND 150 back issues.
Site Navigation Guide
Copyright © 2000-2009 by Startup Internet Marketing Services* All rights reserved|
104 Main Street. Kirkland Lake, ON, CA P2N 3E8 705.567.5850
This site is powered by Host4Profit.net