I don’t much mention SportsLizard or the Price Guide anymore, but it is still alive and kicking. I’ve got it down to a science where I probably only spend less than 5 hours a week to maintain the site. The Price Guide is at about 30,000 users but – because of our inline help and FAQ – requires very little customer service. Premium accounts are only a small fraction of that, but we still make over $1k/month on them for doing essentially nothing.
However, this morning my biggest SL nightmare became reality – the Price Guide software mysteriously stopped working. A bunch of cancellations followed, as did an email from a loyal user who has priced 11,238 items using our Price Guide (yup – he’s run 11,238 searches!).
This was at about 8:30 AM while I was sitting in a Starbucks. I had planned to do some relaxing work and then meet my cousin for lunch…but all of a sudden I was in panic mode. The application relies on a lot of things to be working right, most notably our syncing with Google Base, and I had no clue where to start.
I thought Google might have disabled our API keys…nope.
I thought someone might have hacked the SL database and screwed with the integration of SL and Base results….nope.
I thought that eBay, NAXCOM, and Beckett could have all stopped listing items on Google Base (essentially killing the service)….nope.
Then I tried a pre-launch beta version that I made that spits out results as a text file without any graphs or a fancy UI. Amazingly that worked. So I went line-by-line through my code and compared the beta version to the live version. After about an hour I figured out that a filter I had put in to weed out items returned with a $0 price was no longer supported (correctly at least) by Google Base and was throwing the results off. The same search would go from 30 legit results to 500 with the filter removed. Very rarely are items listed at $0 anyway – I just put that in there as a redundancy – so I removed it and everything began to work fine again.
The last time I successfully used the Price Guide was a week ago, so it looks like it was in bad shape anywhere from 36 hours to 1 week.
Now the hard part – damage control:
- At 8:30 I had immediately placed a notification in red on the Price Guide page saying something to the effect of “We’re experiencing technical difficulties, we’re working to resolve them ASAP, we apologize for the inconvenience”. I took that down once I was certain I had solved the problem.
- I emailed that uber-loyal user back who initially alerted me of the error. I refunded his most recent payment and offered him a free premium account for life.
- I emailed everyone who had canceled in the past week apologizing for the inaccurate results and offering them the same free premium account for life.
- I wrote a post on the SL blog (which is auto-emailed to newsletter subscribers as well) notifying everyone that the issue had been resolved and extending the same free premium account for life offer to anyone who I did not email that also experienced troubles with the Price Guide during the past week.
Whew. That was a stressful morning.
As much as it sucked, it allowed me to re-familiarize myself with the app I built almost a year ago. More importantly, it created an opportunity for me to show my customers how much I care about them and their experience with the Price Guide. It also gave me a chance to show that we address our issues rapidly – within hours of being notified of them…even on a Saturday – and with a candid, honest, and genuinely remorseful attitude.
No one wants to have technical snafu’s like this, but when presented with one you’ve got to make the best of a bad situation.