Now Running On Our New Server Cluster!

Two Saturday’s ago, after a lot of preparation, we migrated Detailed Image to a much-improved server configuration. Thankfully, this wasn’t prompted by any sort of major issue. Rather, in the process of disaster planning we identified a few spots where our redundancy wasn’t as strong as we’d like, the most glaring of which was our server.

Since the total chaos of our move in 2008, we’ve been on a dedicated server. We upgraded once in 2011, and that machine lasted us right through this migration. We had all sorts of backups, including real-time offsite backups, but we were still limited by the fact that our entire business relied on a single machine…which was less than ideal.

Now we’re running what’s referred to as an active-passive high availability cluster, meaning that there’s a second server that’s constantly syncing to the primary server, and if that primary server is unavailable for any reason, the secondary server kicks in seamlessly. This also allows us to do things like a planned hardware upgrade without any noticeable downtime for our customers – take one server offline, upgrade it, put it back online, and then take care of the other one.

Of course, these types of migrations are never smooth, especially with our 10+ years of data and custom code. As far as our customers were concerned, we were down for 2-3 hours early on a Saturday morning. That was quite a bit longer than planned, but still not all that bad. But on our end, I was working nonstop for the week leading up to the move, and then that Saturday I worked until I dropped. I learned a ton (and took good notes!) so that next time will be smoother. Our last move in 2011 was so long ago that the ticket no longer existed, and of course in the middle of one of our problems I had a flashback to 2011 and remembered the exact same thing happening.

So all in all, I’m very relieved that this is complete. Almost immediately, a few lingering issues that I couldn’t solve previously were instantly resolved (for instance, our newsletter system was having send speed issues the past year or two and I had debugged it to death with no luck). I never look forward to these types of major moves. I first had this setup quoted to us in 2015…I’ve been stalling that long. I’m not a sys admin, but I try to learn as much as I can for a migration like this so that I can prevent/minimize any issues.

Now that it’s complete, we’ve got a level of redundancy that removes that lingering worry that I had 24x7x365 that the site could go down at any time and I might be sprung into action. There were a ton of great business reasons to do this migration, but that peace of mind is priceless to me.

4 comments on Now Running On Our New Server Cluster!

  1. Rob says:

    Congrats! Another good step towards a solid system.

    Don’t Liquidweb offer some kind of autoscaling/”cloud” /elastic type solution where you don’t even have to worry about this stuff though?

    Have you tried cloudflare BTW? Been migrating a few sites over to it, it’s very neat, and can do interesting things with per URL rules. One little thing I did yesterday which has hugely reduced load on a client server is to cache their podcast RSS feed. Now instead of thousands of phones pinging the feed hundreds of times a day, cloudflare just requests it hourly and rehosts it seamlessly.

    Bit annoying to see your post today though, I have to say! I’m currently in the middle of a migration (magento to shopify) that has gone a bit sideways, I think to do with the dns- some of the CNAME rules seem to be being applied backwards or something 🙁

    • Adam McFarland says:

      Ah, sorry to hear you’re having trouble with your migration. Does Shopify provide help with migrating to them, or are you kind of on your own?

      LiquidWeb does have a cloud server set up. We used it for LockerPulse and we’re using it for all of our non-DI sites like this blog. It’s pretty good, but there’s a little less control, anecdotally it seems to be down more (they regularly schedule maintenance to the platform), resizing a server isn’t quite as fast and easy as they make it out to be (for added hard drive space it would take LockerPulse hours), and last time we looked into it there were PCI compliance concerns…although I’m sure those can be overcome now.

      If we were starting new tomorrow I would definitely look at a cloud solution. I actually love some of Google’s offerings – they have a few products that literally scale automatically without any effort, which is great for someone like me who would rather focus on the development and growth of a product. The downside is that some functionality is limited, so you’d have to build your app with that in mind (and really, separate any Google specific code so that it could be ported somewhere else). I had a phone call with them last year and was impressed with the amount of assistance you get in terms of dedicated reps and technical contacts. Not what you’d expect from Google.

      I haven’t used Cloudflare. It’s one of those things that’s been on my list but I’ve never gotten to. Again, much easier when you’re starting new to configure it precisely how you want it. With an older site like DI everything is more complex unfortunately.

      • Rob says:

        They sort-of help with it, but between shopify, the registrar and a couple of plugin providers that fudge with how things are displayed there’s a lot of finger pointing. I think it’s mostly sorted now, just some geo-ip detection redirection stuff that was causing issues. I just got fed up with it and turned it off for now!

        I think there are always tradeoffs and it’s not until you’re knee-deep in a product that you realise this. With reliability or flexibility often comes the loss of a certain level of control and perhaps fewer options.

        AWS/Azure/GCC seem to have some very interesting offerings, I prefer dealing with a host for whom hosing is their bread and butter though – you know it’s their core business so they’re going to stick around not be so many orders of magnitude bigger than you that they don’t give a crap about you.

        Cloudflare is interesting. Give it a play on a non critical site first perhaps? In its simplest form it works just like a normal DNS (but propogates much faster), but it also gives you a bunch of options, for instance, easily offloading static resources to their CDN, fine-tuneable regex type control over what is cached and how long it’s kept etc. There’s also some DDOS protection stuff built in.

        • Adam McFarland says:

          I think there are always tradeoffs and it’s not until you’re knee-deep in a product that you realise this. With reliability or flexibility often comes the loss of a certain level of control and perhaps fewer options.

          Exactly!

          AWS/Azure/GCC seem to have some very interesting offerings, I prefer dealing with a host for whom hosing is their bread and butter though – you know it’s their core business so they’re going to stick around not be so many orders of magnitude bigger than you that they don’t give a crap about you.

          Very true, and a good reason to stick with a company like LiquidWeb. For all of my minor complaints, they always meet their 30 minute response time on any ticket, the person who responds is always knowledgeable, and I do get the feeling that they care about our business when I talk to our rep.

Leave a Reply

Your email address will not be published. Required fields are marked *

Commenting Rules

I'm honored that you found this post interesting enough to leave a comment. Before posting, I have a few ground rules:

  • Please keep your comments as relevant to the post as possible.
  • No personal attacks or any other nastiness.
  • Your first comment is subject to my approval.

Thanks!