When you just need a bunch of gems quick...

by Gregory Brown

I've been running local special need gem servers as well as a gem server for beta versions of Ruport and support code (namely plugins). I got tired of manually wget'ing the gems I needed and their dependencies, so with a little help from mechanize and archive_tar_external, my friend Dinko and I came up with this quick (pretty messy) hack.

require 'rubygems'
require 'archive/tar_external'
require 'mechanize'
require 'fileutils'
include FileUtils
include Archive

module RFDownloader

def populate(archive)
mkdir ".rf_downloader"
cd ".rf_downloader"


Tar::External.new( "../#{archive}.tar",
"*.gem", "gzip")
cd ".."

def get(pattern,options={})
link = "http://rubyforge.org/projects/#{options[:project]}"
agent = WWW::Mechanize.new
page = agent.get(link)
page.links.find { |l| l.text[/File/] }.click
link = agent.page.links.find { |l| l.text[pattern] }

def fetch(opts,&block)
populate(opts[:archive]) do
module_eval &block

module_function :fetch, :populate, :get


With this, we were able to grab automatically a tarball filled with all the gems needed to run Ruport 0.9.0 with a tiny little script:

require "lib/rf_downloader"

RFDownloader.fetch(:archive => "ruport_deps") do
get /fastercsv.*gem/, :project => "fastercsv"
get /pdf-writer.*gem/, :project => "ruby-pdf"
get /color-tools.*gem/, :project => "ruby-pdf"
get /transaction-simple.*gem/, :project => "trans-simple"
get /RedCloth.*gem/, :project => "redcloth"
get /hoe.*gem/, :project => "seattlerb"
get /rubyforge.*gem/, :project => "codeforpeople"
get /mailfactory.*gem/, :project => "mailfactory"
get /mime-types.*gem/, :project => "mime-types"
get /scruffy.*gem/, :project => "scruffy"
get /build.*gem/, :project => "builder"
get /gem_plugin.*\gem/, :project => "mongrel"

Elegant? Nope! A way to be lazy and quickly snapshot the newest versions of a number of gems, you bet ya. Maybe someone will see this and build a nice little library that does this 'the right way' :)

UPDATE: Thanks to Aaron Patterson for helping me get rid of the wget call


2007-03-11 06:14:58
What are you using this for exactly? I'm curious b/c this comes very close to my own needs -- I have project made up of many subprojects. setup.rb handles this layout gracefully via the packages/ directory, but gems has no way to handled it. And given how gem works I have no desire to add tens of depependencies to a "stub pacakge".

Among other possibiles, I've been considering creating a tool that can merge multiple gems into a single gem. Would that be of use to you?

Personally I wish -y was the deafult mode of gem. This is mostly a perceptual issue.

2007-03-11 09:41:05

As far as I can tell, if you want automatic dependency resolution, the dependencies must be available on your gem_server.

(Or you must have your users fire up a laundry list against RubyForge, or post a stub package there, which I consider to be pollution)

So i'm using this tool to identify a series of gems that I need for any given gem server, run it, get a tarball back, sftp it to wherever it needs to be, and then reindex.

It's actually a fairly common operation for me, because I build lan appliances using Ruport+Camping for clients, and some want a very smooth install process, which I solve through setting up a gem server for them.

About a gem merger: Yes, that'd be nice. Then I could sling around one or two gems instead of 20.

about -y, it's unrelated to the issue. If you use -y against a non-rubyforge gem server, it will cry that it cannot find the gem in the repository if you don't have it there. (But I agree it could be a nice default, since saying no cancels the install). If it were the default, we'd need a Linuxy style pretend flag, for auditing.

Daniel Berger
2007-03-12 08:07:18
Someone's using archive-tar-external? Yay! :)
Gregory Brown
2007-03-12 08:35:13
Five minutes of zlib didn't find me multiple file love so I searched RAA for tar. archive-tar-external works great. Wish it'd glean format from the filename, though.
Daniel Berger
2007-03-12 10:49:52
Wish it'd glean format from the filename, though.

You mean for the uncompress method? That makes sense. If you're referring to something else you'll have to clarify. Or, put in a feature request. :)

Gregory Brown
2007-03-12 10:59:37

Tar::External.new( "../#{archive}.tar","*.gem", "gzip")


Tar::External.new( "../#{archive}.tar.gz", "*.gem" )