security, pentesting, oss, economics, hacking, music
There's been a fair bit of interest in this presentation from our annual BeaCon meetup (after SOURCE Boston), so i'm linking it here. The main topic of the presentation is performing OSINT on pentests, and it goes a little further than most of the presentations i've seen for this audience. Inside, you'll find info on exactly what OSINT is, data sources which make it really easy, and some information on the tools you can use to perform it.
Oh, hey, awesome, there's a video too!
Did you get an email from github.com or elsewhere asking you to confirm your ssh keys?
The quickest (both in terms of development and scraping time) i've found to scrape an AJAX page is using Capybara. Capybara is a really nice wrapper around htmlunit / selenium / webdriver. It's designed for acceptance testing, so it does a good job of emulating a real browser. Which is nice if you're trying to scrape data that just isn't otherwise available w/o a browser.
I've included a usage sample below.
-jcran
# Install capybara and all deps
############################
# DEPENDENCIES:
#
# Install the capybara gem:
# $ gem install capybara
#
# Then, follow instructions from https://github.com/thoughtbot/capybara-webkit#readme
# and install the capybara-webkit gem and drivers:
# $ sudo apt-get install libqt4-dev libqtwebkit-dev
# $ gem install capybara-webkit
############################
#!/usr/bin/env ruby
require "capybara"
require "capybara/dsl"
require "capybara-webkit"
Capybara.run_server = false
Capybara.current_driver = :webkit
Capybara.app_host = "http://www.google.com/"
module EAR
class Client
class Google
class SearchScraper
include Capybara::DSL
def search(term)
visit('/')
fill_in "q", :with => term
click_button "Google Search"
all(:xpath, "//li[@class='g']/h3/a").each { |a| puts a[:href] }
end
end
end
end
end
scraper = EAR::Client::Google::SearchScraper.new
scraper.search("Pentestify")
Since I always seem to forget, here're the commands to build and publish a new gem:
'gem install bundler' gets the latest version of the bundler gem.
'bundle gem pentestify' creates the initial layout.
'rake -T' to list the tasks created by the 'bundle gem' command.
Mod the gemspec to your liking.
'rake build' to build the gem.
Once you've created the .gem, publish to Rubygems: 'gem push pentestify-0.0.1.gem'
Handy reference.
-jcran
