Archive for February, 2010

Ralf: S3 logfile downloader and merger

Today we released a new gem called Ralf, it stands for Retreive Amazon LogFiles

Download the gem at http://gemcutter.org/gems/ralf or browse the source at http://github.com/kjwierenga/ralf

Usage

Prerequirements:

  • An S3 account (duh)
  • Enable logging on S3 (use cyberduck for example)

Execution:

r = Ralf.new(:config => '/my/config.yaml', :date => '2010-02-01')
r.run

Or run it in one go:

Ralf.run(:config => '/my/config.yaml', :date => '2010-02-01')

Parameters:

:config   a YAML config file, if none given it tries to open /etc/ralf.yaml or ~/.ralf.yaml
:date     the date to parse

:aws_access_key_id      (required in config)
:aws_secret_access_key  (required in config)
:out_path               (required in config)
:out_prefix             (optional, defaults to 's3_combined') Prefix the output file

You can ommit a configuration file when you supply the required parameters :aws_access_key_id_, :aws_secret_access_key and :out_path
They take precedence over the config file

Caveats

  • There is no logrotation as we know it like on regular unix machines

More info: S3 Server Access Logging

TODO

  • CLI executable
  • Grouping/combinig on week/month

Credits:

This plugin is created for kerkdienstgemist.nl for processing the logfiles generated by S3.

Authors: Leon Berenschot and K.J. Wierenga