Spider food is hyperlinks that are placed on webpage to attract a spider’s attention. These links are not visible to visitor and purpose of these links is to direct the spider to keyword rich “doorway” or “hallway” pages designed to fool the search engines.
When spider visits your site first it looks for a file called “robot.txt”. This file contains instructions for the spider on which part of the web site to index and which to ignore. “robot.txt” is an only way to control the search engine spider.
All spiders follow some set of rules. According to one rule spider can load only one page a minute. Early spider loads entire website at once. The result was bad search result by the search engines. With modern fast web servers a spider might visit your website several times a day.
When spider visits your site first it looks for a file called “robot.txt”. This file contains instructions for the spider on which part of the web site to index and which to ignore. “robot.txt” is an only way to control the search engine spider.
All spiders follow some set of rules. According to one rule spider can load only one page a minute. Early spider loads entire website at once. The result was bad search result by the search engines. With modern fast web servers a spider might visit your website several times a day.
No comments:
Post a Comment