Is Your Site Invisible?

Easy ways to determine if search engines can find your site

If search engines can’t find your site, it may as well be the Invisible Man. Search engines like Yahoo!, Google, Bing and others are often a primary source of traffic and determine your ranking. Unfortunately, publishing your files to the Internet does not guarantee that the search engines are going to find them.

Search engines crawl the Web, indexing pages and following links to find more pages. That’s their job. Pages that are newly published can appear in a search engine’s index (and in search results) within minutes, but sometimes it takes hours or even days.

This article will help you to get an idea of whether the search engines are finding your site, and what they see. In the examples below we’ll look at Yahoo!, Google and Bing, the three search engines with the highest market share in the U.S.

How to tell if your pages are being found by search engines
To determine if your site is indeed being indexed, do this simple search in the search box on any of the search engines by putting “site:” before your URL. Specific instructions like this are called “operators.” By typing that before the URL, you’re using what’s known as the “site operator.”

Example: site:yoursite.com

Don’t leave any spaces in the query. It should look like this:

Lippay_1

The results will bring back pages from that site only. If you do not see any results from a site:yoursite.com search, then the search engine is not finding your site.

Google does not show duplicate pages in these results, but it does allow you to see what’s been filtered. In order to see all pages, including ones Google deems as duplicates, look for a link after your very last search result listing that says, repeat the search with the omitted results included. Click on that link to see all pages that Google considers duplicates of the ones listed in the initial query results.

Or simply add &filter=0 to the end of the URL in your browser address bar and hit enter.

If you don’t see the “repeat the search with the omitted results included” link or do not see any changes when you add &filter=0 to your URL string, then you don’t have any previously filtered duplicate pages. This is a good thing because duplicate pages can split your in-link value among many landing pages instead of one, potentially hurting the rankings of your canonical landing page.

Is your content being crawled by search engines?
Search engines may find your page URLs when they crawl the Web following links, but they may not have the content of your pages indexed. To determine what search engines are actually indexing, you can click on the “cache” link on listings in search results.

If you’re looking for any page on your site–You can use the same site: operator referenced above.

If you’re looking for a specific page on your site–You can do a search for the page by entering the exact URL in the search box.

Lippay_5

If you’re looking for specific content on your site—Enter the site: operator followed by an exact phrase in quotes in the search box (no spaces).

Lippay_4.2

To check what each search engine has cached, click on the cache link under the result you’re interested in. Here are screenshots of the cache link on Yahoo! (Google and Bing look pretty much the same).

Lippay_7

When you click on the cached page, you will see the content that the search engine has actually indexed. Compare that to the page you see in your browser when you visit the page itself. Do you see any content missing?

Note the content may have changed since the last time the search engine crawled your page. Search engines also sometimes choose not to index “noise” on a page such as advertisements. What is important to look for here is that the topical content of the page at the time the crawler visited was indeed indexed. If there is important content missing, there could be various reasons why.

Read the Search Engine Guidelines referenced below for more information, and stay tuned for the next SEO article where we’ll discuss steps you can take to make sure you’re doing everything you can to help the search engines index and crawl your content.

Search engine guidelines for webmasters

—Laura Lippay, Director of Technical Marketing

(Image courtesy ‘J’, via Flickr, CC 2.0)

Posted by Administrator

[ Categories: Uncategorized ]

34 Comments Add your own

1. John | February 19th, 2010 at 1:33 pm

Great post, Laura. Although this is quite basic, it’s good to see a search engine giving tips on SEO. Looking forward to the rest of the series.

- John
2. Josh Ziering | February 19th, 2010 at 1:46 pm

Nice Post Laura. Getting the idea of operators and operands in to the minds of non-seo people is not an easy task. This was well written and informative.

Josh
3. SearchCap: The Day In Sea&hellip | February 19th, 2010 at 2:13 pm

[...] Is Your Site Invisible?, http://www.ysmblog.com [...]
4. Google On Page SEO Tips &&hellip | February 19th, 2010 at 9:04 pm

[...] Yahoo! Search Marketing Blog » Is Your Site Invisible? [...]
5. pligg.com&hellip | February 20th, 2010 at 10:35 pm

Yahoo! Search Marketing Blog…

Yahoo! Search Marketing Blog…
6. ross wrigley | February 24th, 2010 at 8:55 am

How do I get found on Yahoo and Bing
7. Yahoo! Search Marketing B&hellip | March 11th, 2010 at 2:24 pm

[...] For more on getting your site noticed, refer to Laura Lippay’s previous post “Is Your Site Invisible?” [...]
8. Jason Shindler | March 22nd, 2010 at 7:44 am

This is a great post. One question I get a lot is about Meta tags. People ask “What meta tags get my site listed in search engines?” Of course, meta tags have little to no relevance to search results anymore, there is a lot of misinformation out there. Perhaps this post and similar posts by yahoo could help.
9. light bulbs | March 22nd, 2010 at 8:34 am

I hear meta tags mean nonthing anymore yet I see they do changes search resaults. If we leave them off a site does it hurt it in any way.
10. Clive | March 22nd, 2010 at 8:43 am

Ignore the meta tags at your own peril!

While it is true that the relevance of meta tags has gone down dramatically they still play an important part – especially the meta description.

Most search engines will use the meta description when your page appears in the search results. If you don’t have one, then they will try and extract some text from the page. Normally not a very good result.

Think of your meta description as the advert for your page – it deserves some careful thought and can make a huge difference to how many people will actually click and therefore visit your site.
11. Accessories Bags | March 22nd, 2010 at 9:51 am

I would still not ignore the meta tag. It never hurts to have your meta tags up to date.
12. Is Your Site Invisible? |&hellip | March 22nd, 2010 at 9:57 am

[...] a nice post that gives you easy ways to determine if search engines can find your site – and how to fix any flaws in your [...]
13. Michael | March 22nd, 2010 at 11:21 am

Great article. Thanks for the insight.
14. Nadine Torosian | March 22nd, 2010 at 12:47 pm

Thank you VERY MUCH! Extremely useful information –
15. Ellie | March 22nd, 2010 at 1:29 pm

Thanks Laura for sharing the SEO insight! You’re very informative and helpful. Love the screenshots included in the article. I’m glad I signed up with Yahoo Adwords. I must add, the service representatives at Yahoo are great too. They’re patient and eager to help answer customer inquiries. It’s always nice to do business with people who value their customers and take pride in their work.
16. Stephen | March 22nd, 2010 at 2:14 pm

very interesting and informative.
I was not aware of the “site” query.
I can now check all my sites.
Thanks!
17. Paul Gilchrist | March 22nd, 2010 at 3:20 pm

One of my clients’ sites (www.USALaw.com) has content from 5 years ago in the Yahoo index, despite numerous content changes and additions since then, and despite manual and site map submission. The odd thing is that while the index hasn’t changed, the site has been crawled relatively recently. This apparent invisibility of the site undermines Yahoo!’s credibility as a search engine.
18. Air Purifiers Direct 2U | March 22nd, 2010 at 4:33 pm

This may be one of the best articles that I’ve seen on here. Thanks for the info.
19. Eugene | March 22nd, 2010 at 5:52 pm

Good stuff, thank you
20. Anupam | March 23rd, 2010 at 3:37 am

Good article, will be waiting for your next article.

Thank you vey much
21. ben | March 23rd, 2010 at 6:45 am

call 678-255-8550 more in
22. Liz | March 23rd, 2010 at 8:21 am

Very helpful info on caching and duplicate pages. Also good info from Clive on meta descriptions. Thanks!
23. bluehost | March 24th, 2010 at 5:49 am

very basic, but nonetheless, good info
24. David Block | March 24th, 2010 at 3:39 pm

Very helpful info. My web site gets more visible by the day. Thank you very much.
25. BuyFastpitchGloves | March 25th, 2010 at 6:57 am

Nice informative article. I was aware of this but it is always good to have a refresher. I look forward to followup articles.
26. Twitter Weekly Updates fo&hellip | March 26th, 2010 at 1:07 pm

[...] Your Site Invisible? http://www.ysmblog.com/blog/2010/02/19/is-your-site-invisible/ [...]
27. stan | March 31st, 2010 at 4:57 pm

Great information to help you find your site.Thanks Laura for sharing the SEO insight!
28. joey | March 31st, 2010 at 5:03 pm

I can make a huge difference to how many people will click and therefore visit your site.Great post, sometimes you find some usefull info, nice.
29. 12 Stellar SEO Tools | He&hellip | March 31st, 2010 at 5:47 pm

[...] you to see what the search engine indexed on its last visit to the page. Our blog post “Is Your Site Invisible?” provides details on how to determine if the search engines are seeing your site, and what [...]
30. Divya | April 6th, 2010 at 7:42 am

Great information. Very helpful for starters like me. Keep it coming!
31. Internet marketing home study course | April 8th, 2010 at 5:42 am

Meta tags should not be ignored rather it should be given at least a minute before posting any content doing this helps me to better target the users searching for my targeted keyword
32. Murrells Inlet Real Estate | April 27th, 2010 at 7:04 am

Wonderful post, you have provided tons of valuable information for me to try and use. I really feel all of it will be of a tremendous benefit to me in increasing traffic to my sites.
33. Rae | June 15th, 2010 at 12:42 pm

This is some good basic information, as others have stated above, however it would be good to know how to actually get indexed quicker….what is the quickest way for the search engines to find the sites. And does submitting the sites to the search engines even help at all?
34. The 9 Steps | July 1st, 2010 at 1:29 pm

Excellent post! This was very helpful.

Some HTML allowed:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Trackback this post | Subscribe to the comments via RSS Feed