task MySQL Server and App Load Optimization

One of our latest websites/apps is getting some good traction. The client had us build what is essentially a white-label product: the C2ES Carbon Calculator. The client can launch an unlimited number of versions of this tool, each one customized for one its corporate partners. Last week a new partner invited its entire workforce of 50,000 to use the tool. The demand was tremendous and greater than expected, which led to a short server outage (<10mins).

To help reduce the likelihood of another outage, we spent a few days analyzing the traffic, pinpointing the weakness (mysql server load), and implementing some great optimizations. By smartening up our aynchronous AJAX calls and javascript, and configuring Memcache(d) to utilize cached queries, we were able to improve the load tolerance of the application by upwards of three times its original capacity. We also implemented some advance monitoring tools so that we'll know when the server is getting over-taxed before it actually fails, and allow us to react and hopefully keep everything running.