Search Engine Optimization Blog - SEM News & Opinion - David Ogletree » Googlebot http://www.ogletreeseo.com If you think it's expensive to hire a professional to do the job, wait until you hire an amateur - Red Adair Sun, 27 Feb 2011 20:47:18 +0000 en hourly 1 http://wordpress.org/?v=3.1 Never let a Search Engine make a decision http://www.ogletreeseo.com/207.html http://www.ogletreeseo.com/207.html#comments Thu, 12 Feb 2009 08:08:13 +0000 ogletree http://www.ogletreeseo.com/?p=207 When you build a website you need to completely understand how a search engine looks at you. Never let a search engine make any decisions. When a search engine comes to your website you need to know exactly how it is going to crawl your site.

Pages should only have one url. If you have a database driven website it is very common that the same content can be pulled up with an unlimited number URL’s. You have to enforce your url’s. Your content needs to be tied to only one url. Put logic into your site so that if it detects that content is being pulled up by the wrong url that you 301 redirect to the correct url. All URL’s should be consistent. The best way to do this is to make every url lower case. If you use a windows server you need to be extra careful because IIS will send you to the same content and produce duplicate content.

]]> http://www.ogletreeseo.com/207.html/feed 5 firefox google bot user agent – How to Impersonate Googlebot using Firefox see what Googlebot sees http://www.ogletreeseo.com/22.html http://www.ogletreeseo.com/22.html#comments Wed, 25 Jan 2006 00:58:00 +0000 ogletree http://www.vsns.com/ogletreeseo/wordpress/2006/01/24/how-to-impersonate-googlebot/ Impersonating Googlebot using Firefox in the past and people have asked me how to do this using Firefox so I thought I would give a little tutorial on how to see what googlebot sees.]]> I have spoken about Impersonating Googlebot user agent using Firefox in the past and people have asked me how to do this using Firefox so I thought I would give a little tutorial on how to see what googlebot sees. I will start off with why somebody might want to impersonate Googlebot and the pros and cons of impersonating it.
Googlebot is the spider that crawls the internet looking for web pages. Whatever the Googlebot sees is what Google uses to decide how a page will rank in the search engine. Knowing this some people have decided to design a web page exactly how they think Google wants it to look like so it will rank better. Often times this can be a very ugly page so they design another page that looks how they want it to look. They then cloak their page so that the Googlebot user agent sees the page written for it and everybody else sees the page written for humans. This is against the Google rules.

When you surf the net your Internet browser sends out messages telling the website things about itself. Websites keep track of visitors in their log files. One of the bits of information that is sent out is something called a “User Agent”. Each browser has its own user agent string. You can find the user agent that you are sending by clicking here.

So if you want to surf around impersonate googlebot all you have to do is change your user agent. The way that I do it is to use Mozilla Firefox with the User Agent Switcher extension. Both sites have a download or install button just follow the directions it is quite straight forward. One the user agent switcher is installed you go to Tools -> User Agent -> switcher -> options -> options. Then click on “User Agents” and then click on “add”. Put “Googlebot” in the Description. Put

“Googlebot/2.1+(+http://www.googlebot.com/bot.html)”
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

in the User Agent.

firefox google bot
firefox googlebot

Technorati Tags: , , ,
]]> http://www.ogletreeseo.com/22.html/feed 17