Apogee Web Consulting LLC
Services Case Studies Why SEM Pricing
your site. elevated.
search engine marketing services



Use trackSpiders.cgi to track spiders from major search engines (Google, Yahoo, MSN). This free web analytics software will examine an Apache web server log and will display visits from Googlebot, Yahoo Slurp, and MSNBot. It takes an optional query string in the format s=ygm where each letter corresponds to the search engine spider for the major search engines. For example, specifying s=g will only display pages crawled by Googlebot.

Each hit from a search engine spider will be displayed in a row with these columns:
  • Time (timestamp of the spider visit)
  • IP Address (remote ip address of the spider)
  • Spider (name of the spider)
  • Status (http error code)
  • Page Crawled (url of the page visited)
If more than one spider visit is found, a bar chart is displayed comparing the number of hits from each search engine spider.

trackSpiders.cgi - Show all three search engine spider visits
trackSpiders.cgi?s=gy - Show Googlebot and Yahoo Slurp spider visits
trackSpiders.cgi?s=m - Show only MSNbot spider visits

Download script [UNIX] [Windows] to your machine.
Edit the line that reads:
so the CGI script will know where your Apache web server log file resides. Upload trackSpiders.cgi to a directory on your web server where you can run CGI scripts. Change permissions to make the file executable (chmod +x or chmod 755). If any of these instructions do not make sense, ask your webmaster or web hosting company to handle this for you. Or, contact us to become a client and we will install this free web analytics software and other custom search marketing reports on your site.

This free CGI script only works on web sites using the Apache web server on a UNIX platform. It will parse a web server log that is in Apache combined log format. The script is open source code.

Maine Munchies Ad

About Us  |  Contact Us  |  Privacy Policy

Copyright © 2006 Apogee Web Consulting LLC
Free Web Analytics Software