Robots Text Generator
The robots.txt file is the first thing search engine "spiders" look for when indexing a website. The robots.txt file tells search engine spiders (robots) which files and/or directories they are NOT allowed to index. This helps to prevent incomplete site indexing as well as prevent exposing the files and directories that you don't want "all over the world wide web". You may disallow, for example, "Google Images" from indexing certain directories and pages on your site, but not block "Google" itself from indexing those same files.
Buy this tool, unbranded, with full source code.
Buy this tool, unbranded, with full source code.