Robots.txt Files Can Make Your Life Easier
According to Keith Hogan from Ask:
i) Less than 35% of websites have a robots.txt file
ii) The majority of robots.txt files are copied from others found online
iii) On many occasions robots.txt files are provided by your web hosting service
If your website or blog uses a lot of images that you don’t want crawled or indexed by the search engines’ photo search features, you can build a page of photos and include a robots.txt file to tell the search engines not to crawl that page. This is extremely important. It can be used for any content that is for your personal use but you don’t want to the world to see.
Add a page to your website but don’t link it to any other page on the website. It’s basically a file on your domain. Then, put a robots.txt file on that page to notify search engines that you don’t want it crawled. In this manner you can keep photos, video files, podcasts, newsletters, links, anything that you want protected and not indexed without keeping it all on your hard drive. This is a very effective strategy for bloggers and others who use material on a regular basis but need a way to access it easily and quickly.
ASM Development can help you with all your SEO & Web Design needs
SEO Blog
Recent Entries
- Ensure Your Web Design Is Browser Compatible
- Domain Names Have Their Own Trust Value
- How Does Fresh Content Affect SEO
- Do You Understand The Search Engines Jargon
- Can Blogging Make A Difference - Ask HP
- Optimizing Documents For Search Results
- Content Development For B2B Blogs
- Branding And Marketing Starts Day One
- What Are The Age Factors For SEO?
- Blogging Is Here To Stay - Make Use Of It