1ShoppingCart Down One More Time

Hi Mike, Jenn,

I'm not here to try and justify the downtime we have experienced this week or apologize for it again. At this point you are right, apologies simply do not make up for the service interruptions that our users experienced this week.

We have worked hard and long over the past decade to earn a reputation for reliability only to see that reputation shattered in just a few days. But we are still committed to providing the most reliable and stable solution to all of our customers and the businesses that we've had the privelege of working with and supporting for all these years.

The reason I wanted to post is simply to tell you what happened, what we are doing about it and why 1ShoppingCart is still committed to providing the best possible uptime and stability to all of our customers.

On Saturday morning one of our database platforms that underpins the store, cart and email marketing systems began to report database corruption. We have a comprehensive monitoring and alerting system that acted immediately by isolating the corrupt systems and paging our Emergency Response Team.

We called in every resource we have including data center staff, on-call staff, senior engineering and database administrators to determine the root cause and extent of the incident and execute our recovery plan. It took us some time to diagnose the issue and bring our backup solution online

On Sunday morning we had a replay of the incident. This time we responded as we did Saturday, but we incorporated additional testing into our recovery process. We acted to simplify the architecture to remove as many possible points of failure as we could at that time. Additionally we tapped our data center partners to begin bringing up a completely new hardware environment for us to migrate to immediately.

On Monday we began to form up and harden the new standby environment so that should we experience a further corruption issue, we would have a place to move our customers to while we wait for our final replacement hardware to be provided.

Unfortunately, we experienced the issue again today. Due to the fact that we were prepared for it and had the server hardware ready, we were able to completely recover in a much shorter period than we experienced last weekend.

We now have our solution entirely off the suspect hardware and have a further hot spare standing by being brought constantly up to date with our production data.

While we can't predict the future, we don't expect that we will see a recurrance of these issues, but if we do please be assured that we have prepositioned assets and resources ready to be implemented immediately.

Despite the failures in our infrastructure we have been able to safeguard your data throughout these events. No data loss has occurred. We are acting vigorously to mitigate any further risk to our service to you and doing whatever it takes to get our solution rock solid and reliable.

We know how important it is that you are able to rely on our service to power your business and we assure you that stability and the restoration of your confidence in us is our top priority.

Please contact me directly if you feel that I can be of any further assistance.