WARNING: This post was originally published in 2009 and hasn't been updated since.
The tips, techniques and technology explained here may be outdated. If you spot any errors, please let me know in the comments so I can adjust the article. Thanks!
About 9 months ago, I was toying with the idea to automate downloading my favorite TV-shows using BitTorrent. Just like millions of others, I downloaded my shows through ThePirateBay or MiniNova manually each week, and it was turning into a drag. I figured there had to be an easier way to do this.
I realised most BitTorrent clients supported RSS feeds, which could notify them of new releases automatically. I tried the popular TVRSS.NET, but it didn't do what I needed: it download episodes multiple times, because the torrents existed multiple times, it downloaded fakes, bad quality ones, ... No matter how hard I tried to finetune it, it just didn't work the way I wanted to.
And that's where my idea came in: FeedMyTorrents.com. "An RSS feed for torrents that just worked." That was our slogan -- pretty catchy, isn't it? :-)
It had potential too: the service would make sure that no duplicates existed in any torrent feed, and that the quality was the best possibly attainable at that time. It was supposed to be a win-win situation. So I got to work, and quickly mocked up a first version of the website. It was working from a technical Point-of-View, but it lacked design & proper consistent layout. So I poked a friend of mine (Jonas Elslander), who quickly created a clean design for it. And so, the public beta of FeedMyTorrents (FMT) was born.
We were happily trying it with about a dozen users, and it seemed to satisfy our needs. The backend system filtered perfectly, and only allowed unique torrents that had proven quality.That same backend logic had quite some AI built-in, too: it used a system of "VIP uploaders", where known uploaders were given a higher score/preference, it analyzed torrent names to favor XViD quality over DivX, it filtered torrent names to quickly identify the TV-Show episode/season that was uploaded, ... All of this was combined with user approval for those torrents that couldn't automatically be parsed, to make sure only the best got through.
At first, it was easy to manage: we had about 20 TV-Shows so it wasn't hard to monitor it all.
Then we thought: "Hey, since this thing is actually working pretty well, why don't we go public? Let the world enjoy!". It's that very train of thought that indirectly brought the end of FMT. We wrote to several major websites, and all were enthousiastic: we got covered on nearly every popular site.
- TorrentFreak.com: "Top 10 BitTorrent And RSS Feed Tips" (point 4 & 6)
- MakeUseOf: "FeedMyTorrents Keeps TV Shows Downloading Automatically"
- MakeUseOf: "Cool Websites and Tools (#257)"
- LifeHacker.com: "FeedMyTorrents Offers TV Torrent Feeds without Duplicates"
- DownloadSquad.com: "Automatically download TV episodes with Feed My Torrents"
- Zepy.net: "10 New Tools for Torrenting"
If you Google for "FeedMyTorrents" you'll now see about 16.000 hits. You can take my word for it, a couple of months ago, when we were at our prime moment, we reached hundreds of thousands of hits. It had more succes than we'd ever hoped, all through social media and a killer concept. These were our daily pagehits for the site.
Allow me to let that sink in for a moment: we were receiving one million hits per day. On the same server that was once serving our "happy little project for a dozen friends". We obviously had issues: everything was running on a default LAMP stack, which was by no means tuned for this kind of server hammering.
The problem was our users didn't diminish, they kept increasing. As more and more people spread word via Blogs, Twitter, Facebook, ... Our MySQL kept crashing, our HTTP-requests kept queueing up, ... It was an endless race, our current set-up couldn't handle it anymore. To make things worse, we kept increasing the amount of TV-Shows we tracked. We started with 20, and reached 150+ TV-Shows in no-time. Everyone could find his/her favorite show on the site.
So we started a typical "fund raising", a simple message on our homepage & blog to ask people for a donation as little as 1$. It sort of worked, about 0.002% of our visitors actually made a donation. It was enough to get extra server capacity at some shady dealer in the Netherlands (for obvious legal reasons), and keep things going. But it wasn't enough for the long run: we hade more "leechers" than actual contributers, and the odds were against us. We kept experiencing server instability, despite running a highly tuned Apache + PHP + MySQL config.
Soon, we ran through our funds and could no longer afford our hosting, which was becoming a killer money-sink. We did everything possible to save money: we hosted our static site resources (css + images) on a different host, we gzipped all our server-content before sending it to the user (which actually saved us about 95% in bandwidth, due to the textual nature of all our content being XML files). But it couldn't last.
In the end, we had to pull the plug. Not because of legal threads (as some reported), but because it was no longer viable. It would cost too much time, money & devotion to keep everything going. We've had numerous requests to go open source, and get the "community" involved in keeping it running -- and while the idea sounded great, it wasn't going to work for us. Recently, I did publish all source code every written for FMT.
During our "prime time", the guilty verdict of ThePirateBay's admins was also made public, which caused us to think twice about what we were doing. We hadn't received take-down notices yet, but it was bound to happen -- and we didn't want to sit around and wait for them. Combine that with MiniNova 's active filtering of "copyrighted material", it pretty muched spelled the end of our service. We relied mostly on The Pirate Bay & MiniNova's tracker, and the latter just had all the content removed, where the former had its admins found guilty. It was all coming to an end, sooner or later.
I'll be running a few articles on how it all happened, how the parsing & serving went, how we handled a million hits/day on shady hardware, what we finetuned, how to overcame certain issues, ...
I've learned a great deal from this project, and have received an insane amount of feedback too. We had a good thing going, but it wasn't ment to be. It does make me wonder: popular torrent sites have to receive a tremendous amount of donations to keep things going, or have super-duper sponsor deals with providers. We were a small fish in the pond compared to the big players, but it cost us nearly an arm and a leg.
If you were once a FMT-user, don't worry: the user-database has been destroyed, as well as all log files and user data. I firmly believe in privacy, and your data has been permanently removed.
Oh, and if you once donated money: we used nearly every last penny on hosting. The last 20$ that was left, I greedily kept to myself and bought me a fine bottle of champagne, to celebrate a once successfull project. It tasted good, too.