Scotland Web Sites and the Search Engines Spiders
Search engines regularly send out their spiders to crawl through your web pages checking to see what content you have and what sort of links you have. Yes, they even crawl through Scotland based web sites. Search engine spider – sounds rather nasty doesn’t it? The word spider is rather strange but has probably come about due to the term ‘web’ to describe the internet.
Spiders are really just a program that follows links and reads the contents and other links that are found. What is needed within a web site is a set of instructions to help these ’spiders’ quickly read your pages before moving on. In fact their may even be pages that you don’t want read or followed by the search engines spider.
Good SEO web sites have two files that are normally found in the sites root directory. These files carry the instructions that search engine spiders follow when reading your site.
The first file is a simple text file names robots.txt. This file carries the information related to what the search engine should or should not index (record) and which links can or cannot be followed.
The second file is a special sitemap file that provides the search engine with a map of your website. Both are fairly simple and straightforward files. The sitemap may look complex but it is really just a list of all the pages to your site.
Often, a website will employ a professional team to design and implement their website, this includes the creation of both the sitemap and the robots file. However, over time, the site owner takes over control of the web site and continue to develop the site, adding pages, changing links and making any other changes they need.
What the web site owners often fail to do is to update their sitemaps and robots files. The search engine comes to read your site, reads the sitemap and robots file but finds all the information out of date. Not a good move.
If you are controlling your own site then make sure you update these files on a regular basis.
Recent Entries
- Why A Search Engine Googopoly Is Bad For The Internet
- Blogging For The Sake Of Blogging Is Not Blogging
- How To Create An Effective Social Marketing Tweet
- Internal Linking – Don’t Be Afraid Of The Long Tail
- Adding Your Scotland Based Local Business To AOL
- Don’t Be Too Quick To Dismiss Google Buzz
- Are Guest Posts A Bad Link Building Strategy?
- Can You Overdo SEO?
- Review Sites And SEO For Scotland Local Businesses
- What Effect Will HTML5 Have On Web Design And SEO