Like anything else, once you get good at programming a large part of your improvement comes in your ability to be more efficient. Can you get the same results with less code, fewer files, less queries to execute, etc. If you don’t, any inefficiencies really become magnified as a project scales and your hand gets forced.
When I whipped up Music-Alerts over a weekend last fall I really just wanted something that worked for ME. I didn’t exactly plan it to scale. I also wasn’t as good of a programmer as I am now. When I made a database caching system for feeds, I figured I’d done everything I needed to for it to scale.
However, once I moved it over to our new server we started having issues. Even after an upgrade to 4GB of RAM the script to generate the feeds was killing our MySQL server in terms of number of connections and number of queries. While the caching system limited the number of times the web service connected to Amazon, it didn’t minimize the number of database connections. For most sites this wouldn’t be a big deal, but RSS feeds get checked by feed readers several times per hour. With 4,000+ feeds – and some subscribed to in multiple feed readers – you can do the math and see that our MySQL server was getting crushed.
So yesterday I spent a few hours building a static caching system that literally creates the feed as a file on the server and only connects to the database once every 3 days just to sync up with Amazon for new releases. I also trimmed the core code from about 1,000 lines down to just over 100, eliminating a few unnecessary queries each time it does connect to the database. In less than 24 hours our bandwidth consumption, server load, and MySQL usage on the server have all plummeted.
Lesson learned: any RSS service I build in the future will be built with static feeds and minimal database connections.