webrobots

This is a library to help write robots.txt compliant web robots.

Usage

require 'webrobots'
require 'uri'
require 'net/http'

robots = WebRobots.new('MyBot/1.0')

uri = URI('http://digg.com/news/24hr')
if robots.disallowed?(uri)
  STDERR.puts "Access disallowed: #{uri}"
  exit 1
end
body = Net::HTTP.get(uri)
# ...

Requirements

Development and testing require Ruby 3.3 or later.

Contributing to webrobots

Releasing

Update the version and push the release commit to master. After CI passes, push the matching vVERSION tag. The CI workflow tests the tag and publishes the gem to RubyGems.org through Trusted Publishing.

The trusted publisher uses repository knu/webrobots, workflow ci.yml, and GitHub environment rubygems.org.

Copyright © 2010-2016 Akinori MUSHA. See LICENSE.txt for further details.