So I've finally found some time to write about the RCOS mirror I mentioned Sunday. We've managed to create a mirror that has provided 60GB to the RPI campus while only consuming 20GB from the off campus links.
In this article I'll be discussing the configuration of the tracker, the seed box, and the public www manager in hopes that others can recreate this system to spread (legal) content and save bandwidth.
The Tracker
The tracker we use is PyTracker. I posted about PyTracker in February when I first created it, and have since been testing it. It holds up under high load; its performance will depend more on lighttpd and PostgreSQL.
I've got the PyTracker software running as a FastCGI process under lighttpd. The lighttpd user is given permission to operate on the 'peers' and 'whitelist' databases.
It should be fairly straight forward to setup the tracker on any host with FastCGI or WSGI interfaces.
The Seed Box
The 'official' seed boxes are all configured the same. Each has a user dedicated to the torrents (the user 'torrent' in these examples).
The user runs rtorrent in a detached screen session. The rtorrent instance watches a particular directory for new torrents, and downloads all ISO files to a separate directory. There is no bandwidth limit set in rtorrent's config.
To prevent the seed box from running wild, iptables is configured to reject all connections the user torrent initiates that connect to off RPI's campus.
New torrents are pulled to the seedbox using rsync for the torrent files, and bittorrent for data transfer.
Providing Users with Torrents
To provide the user with torrents, I opted to make them available over flat HTML files. All the torrents have been modified to use the tracker set up under the 'tracker' section in addition to the trackers provided in the default torrents. All the torrents provided via rsync for seed boxes are modified to only use our tracker.
To accomplish this, I wrote a small script available here that will take a directory hierarchy and transform all .txt files to .html, copy all images, and perform the modifications that the torrents require.
I'm storing all torrents in a git repository for easy modification. When I push to the server, the pycms code is run to update the webroot (for users), and the rsync directory (for seed boxes).
Summary
I have given most of the details of our setup (without delving into boring configuration details). It performs rather well on our hardware (a dual core Intel with 1GB of RAM) and has already provided a significant service to the RPI community.
If you're looking to setup your own software mirror, I'd be happy to provide help with setup and configuration of the above components. Please don't solicit my help for sharing of files which you have not obtained permission to share.