Why is KnowledgeStorm crawling my site?

The KnowledgeStorm Crawler gathers data from sites that provide information within the technology industry. KnowledgeStorm is the Internet's top-ranked technology solutions and search site. The KnowledgeStorm Crawler visits the contents of the publicly available portion of the World Wide Web.

How can I request that KnowledgeStorm crawl my web site?

You may send your site URL to KnowledgeStorm. Submission of your site will trigger a review process that will ultimately determine if the contents of your site are included in the KnowledgeStorm index.

How can I prevent KnowledgeStorm from crawling my site?

The KnowledgeStorm Crawler will not crawl anything that you wish to remain private. To indicate pages or directories that you do not want crawled, enter them in your robots.txt file - a standard process that is followed across the web. A tutorial on creating your robots.txt file can be found at http://www.robotstxt.org/wc/exclusion.html#robotstxt.

What if the KnowledgeStorm Crawler is crawling my site too quickly?

Please contact our Crawler Administrator. If you feel the KnowledgeStorm Crawler is crawling your site too quickly. You will need to be prepared for this conversation by having your web logs ready, showing a history of KnowledgeStorm visits to your site. With this information, we can work together to solve the problem.

What if I have additional questions or concerns?

You can contact our Crawler Administrator. This person will be able to help you with your questions or concerns.