That Time The FedEx API Stopped Working on Cyber Monday

We were cruising along on Cyber Monday afternoon when I noticed that the FedEx Rate API was down, which is kind of important since we use it to pull shipping rates during checkout. No big deal I thought, it will default to the backup system, which was a rate table in our database. FedEx has scheduled downtime regularly and this backup has always worked well. Just to be certain though, I started investigating and realized that since this “downtime” wasn’t scheduled, we weren’t receiving a quick error message back. Instead it just hung on us, and in turn the checkout page just hung on our customers.

I myself couldn’t get the checkout page to load, and a quick scan of orders showed that the flow had significantly diminished. Oh shit, I thought to myself. This is a problem. I quickly programmed in functionality to default to the backup rates, bypassing the live API, and deployed it to the site. Problem solve in less than 20 minutes…sort of.

Immediately I noticed two things:

  1. The checkout page was faster, noticeably faster, like applying a coupon code was instant instead of hanging for 10 seconds.
  2. Our backup rates weren’t very accurate because, you know, they were backup rates only to be used temporarily

#1 led me to the conclusion that we needed to stop using the live API. Speed = conversions = $, especially on mobile. A lot happens on the checkout page, so I always (wrongly) assumed that the slow-ish speed was due to a combination of factors – the massive amount of queries and calculations along with the API call. It had been a while since we tested the backups, but we never noticed this obvious increase in speed during testing…I’m not sure how, but we didn’t. Anyway, not using the live API was an accidental huge win for us. Nothing like a 5-10x speed increase on the most important part of your site by doing nothing!

Except that #2 was now a big problem. The reason we used the live rates is because FedEx rate calculations are a beast. It’s not just a simple rate table like we had developed for our backups, there are all sorts of surcharges that can be applied in an almost endless variety of circumstances. Rather than develop for those, it always made sense to just pull the rates from the API live, as I think most sites and shopping cart software do.

Now that the stakes were raised, we all decided it had to be done. I spent the remainder of the week immersing myself in the minutia of FedEx rates. By the end of the week we had a shiny new rate system. It hits the API overnight to generate the base rate table, but if there’s an issue with the API call it just slows down how fast our rate table is updated, it doesn’t have any impact on customers checking out. It also is now able to:

  • Quote rates based upon whether an address is residential or business (it’s financially beneficial to ship Ground to businesses and Home to residents despite the services being almost identical)
  • Apply the appropriate residential surcharges for FedEx Ground & Home
  • Detect if a zip code is considered “rural” and then apply the appropriate Delivery Area Surcharge
  • Support multiple-box shipments
  • Calculate appropriate rates & surcharges for SmartPost-only areas like P.O. Boxes and U.S. Territories
  • Has support for 2015 dimensional (DIM) pricing
  • Populate rates using test zip codes not subject to any of the above surcharges so that we’re always starting with a base rate for a zip code range and then adding on any surcharges after that
  • Works with our Guaranteed 1 – 3 Day Shipping

I then spent quite a bit of time checking the rates for every single order that came through. After those all looked good, I set up regular automated alerts, reports, and spot-checks to continually ensure the system is still working.

Ah, good times.

6 comments on That Time The FedEx API Stopped Working on Cyber Monday

  1. Tim says:

    Wow, talk about a perfect storm part 2! Smart catching it and having the resources/ability to remedy things so quickly. API’s are great, but definitely have limitations when called on in real time, much like with tracking pixels, you’re only as strong, or in this case as fast, as your slowest partners services.

    Now where’s the annual box pile!?! I’ve been eagerly awaiting the picture.

    • Adam McFarland says:

      Now where’s the annual box pile!?! I’ve been eagerly awaiting the picture.

      We actually didn’t take any pictures this year. Part of it is that we now get weekend pickups so that Monday pile isn’t nearly as big anymore. The other thing is that we’re noticing a leveling off of sales throughout the holiday season, as I think most retailers are (and I think this is somewhat by design). We’re starting our big sales earlier and earlier, and running them longer, like I see a lot of retailers doing. Cyber Monday was still probably our largest sales day of the year, and November was up quite a bit from last year, but the big source of growth in November was spread out throughout the month instead of on one or two days. When you think about it, this trend is better for everyone: retailers don’t have huge spikes causing inventory and staffing concerns, consumers get the best deals for longer, and no one has to deal with website crashes or dangerously crowded Black Friday stores.

  2. Rob says:

    How frustrating! Hope you didn’t lose too much custom because of it & also didn’t lose on mispriced shipping.

    Do many people use the free shipping coupons you put out?

    Also looking forward to the 2014 box pile pic!

    • Adam McFarland says:

      Thanks Rob, it actually wasn’t too bad. It was difficult to checkout for maybe 45 minutes. Not ideal, but not catastrophic either. I was just about to shut down and head home. If I had done that it could have been a few hours before catching it.

      As you sort-of alluded to, we were saved a bit because we have free shipping over $99 without a coupon code running through the holidays, which saved us because more customers than normal were getting free shipping and didn’t require perfectly accurate quotes.

  3. peter says:

    We had the same issue. Good thing was that we also offered USPS shipping, and we quickly turned off FedEx so that our customers are able to checkout.

    • Adam McFarland says:

      Whoa, glad to hear you had a backup option available Peter! And glad to hear you were on top of things like us. I wonder how many were affected? A quick Google search shows it wasn’t just you and us. Kind of hard to believe FedEx of all companies wouldn’t be prepared for this.

Leave a Reply

Your email address will not be published. Required fields are marked *

Commenting Rules

I'm honored that you found this post interesting enough to leave a comment. Before posting, I have a few ground rules:

  • Please keep your comments as relevant to the post as possible.
  • No personal attacks or any other nastiness.
  • Your first comment is subject to my approval.

Thanks!