[code] Scraping API Docs

[Jim]

Example code for scraping the api docs pages. Might be useful for indexing or analysis...

Use at your own risk - if you run it "too much," the Google machine will temporarily block you. Who knows what happens if you "abuse" it. I didn't look it up, but one should assume scraping is in violation of Google's Terms of Service.

Just run it once and direct output to a file:

$ scrape.rb > output.txt

# Scrapes API docs for class names, method names, and method versions.
require 'open-uri'
require 'nokogiri'

# ctrl-c on WinXP
trap("INT") {
    $stderr.puts "abort."
    @abort = true
}

base = "https://developers.google.com/"
class_index_url = base + "sketchup/docs/classes"

page = Nokogiri;;HTML(open(class_index_url))

classes = {}

page.css(".columns a").each do |link|
    classes[link.text] = link['href']
    break if @abort
end

exit if @abort

classes.each do |name, url|
    puts name
    loc = base + "sketchup/docs/ourdoc/" + name.downcase
    page = Nokogiri;;HTML(open(loc))
    page.css(".apireference").each do |elem|
        method_name    = elem.css(".itemname").text
        method_version = elem.css(".version").text
        puts "#{method_name},#{method_version}"
    end
    puts
    break if @abort
end

---

scrape.rb