Mirrors
/
slackbuilds-current
mirror of https://github.com/Ponce/slackbuilds.git
5
0
Fork 0
Code Releases Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
45796 Commits
614 Branches
809 Tags
301 MiB
 
 
 
 
 
 
Tree: a498775197
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'a498775197'
${ noResults }
slackbuilds-current/perl/perl-www-robotrules
Zachary Storer 177af84a22
perl/perl-www-robotrules: Fix source URLs. 		11 years ago
..
README perl/perl-uri-escape: Fixed dep information 13 years ago
perl-www-robotrules.SlackBuild various: Update find command to match template. 12 years ago
perl-www-robotrules.info perl/perl-www-robotrules: Fix source URLs. 11 years ago
slack-desc various: Fix slack-desc formatting and comment nit picks. 12 years ago

README 

This module parses /robots.txt files as specified in "A Standard for
Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html>
Webmasters can use the /robots.txt file to forbid conforming robots
from accessing parts of their web site.
The parsed files are kept in a WWW::RobotRules object, and this
object provides methods to check if access to a given URL is
prohibited. The same WWW::RobotRules object can be used for one
or more parsed /robots.txt files on any number of hosts.