SmartViper.com BotSmartViper Bot obeys the Robot Exclusion Standard. Specifically, SmartViper Bot adheres to the 1996 Robots Exclusion Standard (RES).
SmartViper Bot obeys the first entry in the robots.txt file with a User-agent containing "SmartViper"
- If there is no such record, it will obey the first entry with a User-agent of "*".
- If it is not able to retrieve a robots.txt file, it will assume there are no restrictions for SmartViper Bot. It will keep trying to retrieve the file, and will obey it if becomes available.
Disallowed documents, including slash "/" (the home page of the site), are not crawled, nor are links in those documents followed. SmartViper Bot does read the home page at each site and uses it internally, but if it is disallowed, it is neither indexed nor followed. If a page has robots.txt standards disallowing it to be crawled, SmartViper Bot will not read or use the contents of that page.