Proposal for version 1.0 (past)

First iteration of the record book will be called the bt-RecordBook.

It will be created by Brian Tanner (and company) at the University of Alberta (please tell us if you want to help).

The interface for RL communication will be based on RL-Glue.

Agents and environments will come from the RL-Library, with usability extensions (for exposing parameters) from RL-VizLib

The initial version of the record book will be based on Java environments and agents in order to get us moving quickly.

The experimental data will be stored (in a format yet to be discussed) in Amazon's Simple Storage System.

Experiments can be verified and validated on any computer, but all official runs will be done remotely on virtualized hardware provided by Amazon's Elastic Compute Cloud.  For now, this will be paid for by Brian Tanner's personal research budget.

Proposal for 1.1 (current)

Still working on the bt-Recordbook, but things have been progressing well so I'm going to be a bit more ambitious.  This proposal starts with 1.0 and builds on it.


Running all official experiments on the Elastic Compute Cloud is a cool idea, but running hundreds of thousands of experiments will not be cheap.  Also, if the software was really designed such that it could only be run in this mode, we'd be excluding many "free" resources in the form of spare computing cycles in our homes and labs.  The problem with running the software on regular machines was that the distributed framework for communicating results and experiments between different parts of the recordbook use other Amazon services, like the Simple Queuing Service (SQS), the Simple Storage Service (S3), and more recently their Simple Database (SDB) service.  Using these services requires access to my account's private keys, which I can't be distributing to everyone.  To resolve this problem, I created a new project that allows a remote system to do the authorization, and patched open source projects Typica (SQS + SimpleDB) and Jets3t (S3) to use this service. 

So now, I can distribute the software to people, and by logging with with username and password I assign, they can access these services without me compromising my secret key.

The plan now is that we will put the software to do bt-recordbook computation on all sorts of computers around the lab and run them at times of low use (overnight and away from paper deadlines).  We will also invite the reinforcement learning community to do the same.  Basically, any computer with Java 1.5 or higher and an Internet connection can be adding results to the recordbook.

I believe we can get hundreds, maybe even thousands  machines working at the same time generating RL results to fill in the recordbook and make it a game-changing sources of RL results.

But why would anyone want to go through the hassle of creating an account and running the software on their computer?  Credits.  Each participant will earn recordbook credit for running these experiments.  The exact economy hasn't been worked out yet, but the basic idea is that for every K hours of computing you do, you early 1 hour of computing credits.  Credits can then be exchanged for time on other people's computers -- having your experimental results computed using the recordbook "grid".  Credits can be spent on public results (common good), or on private results.

We don't all have the same deadlines, so credits can be used at "crunch time" to run a large number of experiments when putting together a paper.  We can also use scalable computing resources like the elastic compute cloud to ensure that we get high throughput even at big paper deadlines that we share.  With a financial support system (I'm working on it), we can afford to spawn hundreds of virtual machines in the cloud to pick  up the slack when everyone tries to spend their credits at a major paper deadline.  This can be partially supported by making it possible to buy credits at a small premium.

For example, say you need about 400 hours of computation done within 12 hours and you don't have machines to do the work for you.  If our price was .25 cents per credit-hour, then you could spend $100 and have the computation done immediately by the grid, within the 12 hours.  If the grid is loosely loaded, then your work is done by others who are earning credit, and the $100 goes into the bank to pay for virtual nodes in the cloud  for times when the grid is heavily loaded.  If the grid is heavily loaded right now, then the money is spent directly on virtual nodes in the cloud to run your experiments.  You  you don't need to buy, configure, or manage the machines or infrastructure to run and coalesce these experimental results!  You pay a reasonable price for your computing time, and you get your results: fast!  Better yet, if you've been saving credits by doing work for other people in the past, it's free!

The only thing stopping this from working is initial capital.  We need some money to get started with such a system, if we want to be able to expand to the cloud at deadline time.    I think the strategy for this will be for me and/or my colleages/supervisor etc to "pay" into the account for having our own experiments and experiments for the community's good at first.  We would want to run these experiments anyway, and don't have enough computers to run them, so we can pay for the initial credits that people are earning.  Once there is a couple thousand dollars set aside, we can see how things go at the first major paper deadline.  Even $1000 should actually go pretty far.

The infrastructure for this system is all in place.  I don't have a user signup yet, but I can manually add users, and then they can run experiments and start earning credit.  I also don't have an automated way for users to submit their own experiments to run yet, or for experiments to be private, but I'm working on it.


Currently, the recordbook software is very dependent on Amazon services for queueing, storage, and database.  This is great for building a big distributed application, but not for people that want to do small runs on their own to test their agents, or for people that want to keep their results private.  The next evolution of the recordbook software (as I see it), will be to write an abstraction layer between the core logic and these services, so that they are not required.  I think we should support 3 modes for the recordbook.

Single Computer

Use for debugging agents and for running very small experiments for checking out ideas.  This approach would store things locally on the hard drive, use local files instead of network queues, and would store results in a local database, probably using SQLite or some other lightweight system.


This is for running class competitions and for doing larger scale experiments on the level of benchmarking an algorithm for a paper submission.  This version would be similar to single-computer, except that a centralized storage service and some other tricks would be used to allow the computation to be distributed between multiple machines. The easiest way to think about this would be if there was a shared directory like NFS, and 5 computers were running experiments and writing to the same set of data files.  Or, these nodes are writing to local datafiles and then scping them to another machine who is aggregating the data into a result database.   Either way, none of the data is going through Amazon services.

Cloud / RL@Home

This is the current model of the recordbook.  Anyone with a recordbook account can submit and run experiments, and the results are all aggregated on virtual machines in the amazon compute cloud.  Currently only public results are available, this will be extended to allow for private results.  Also.  Private results that were created in cloud mode will be publishable directly into the recordbook. Results created in Cluster or Single Computer mode will not.

Todo/Wish List

If you mail me your ideas for what the recordbook should support, I can put them on this wishlist.

Strikethrough means it is "cancelled", green means it's done.

Some of these things are listed as official "issues" already, some are not

  • User Signup
  • User Control Panel (view credits etc)
  • Viewer App downloads results and stores them for future use
  • Viewer App can run without an account
  • Viewer export from compressed binary results to .csv files
  • A way to upload an experiments/agents/environments
  • Abstract and implementations for local experiments without the cloud (clear sky version)
    • Abstraction Layer for SimpleDB 
    • Local Implementation of SimpleDB
    • Abstraction Layer for SQS
    • Local Implementation of SQS
    • Abstraction Layer for S3
    • Local Implementation of S3
  • Support for "private" results that are only accessible to the experimenter
  • Multi-language support
  • Google Web Toolkit based web application to create new experiments and schedule trials
  • Matlab front end that hooks into Java viewer in an interactive way to download and visualize results from within Matlab.  Should be easy, like the RL-Glue Matlab codec.