|
Tuesday 04 January 2005 9:00:22 pm
Hey guys - I'm sorry to lose the business, but understand. As one of the principles here at FireBright, I'd like to respond to your experiences. For starters, in terms of credit cards, we had some sneaking database corruption on our main server. No issue right? Well, we were keeping an archive of our billing data for two weeks, and after that only once a month for archival purposes. The corruption happened over a several week period, and by the time we discovered the problem, it was too late to do anything about it. The backups we had all had the same problem going back for quite some time. Our staff sent out a very apologetic letter to clients explaining all this in our monthly newsletter, but it only went to active clients. Also of note was that we extended a free month of service to anyone who contacted us who was unable to get a credit card in the system -- I'm sorry you were on vacation but there isn't much I can do about that! ;-( We've seen some intermitten outages for some clients resulting from DNS resulting the same problems with the database corruption above -- it wasn't just billing data but all parts of the customer records -- it's just that you can rebuild the customer records -- the credit cards are encrypted. A small number of our clients had DNS issues, but we were able to resolve every single one of them almost as soon as they cropped up (DNS however, has a fantastic way of causing problems LONG after you have fixed the problem). We did not have a 12 hour outage at any point. We also had a planned outage (2-4 hours) when we moved all of our systems from HE.net to Mae West. This was unavoidable, as we couldn't replicate our infrastructure in the new facility. We ran out of power at HE.net, and were actually having problems keeping our Enterprise systems cool enough (they have really cheap cabinets). When you put enough Dell 2650s in a rack, they generate a LOT of heat. We lost 4 disks in a couple day period (and we're talking about 146gb 10k and 36gb 15k SCSI Seagates) due to the heat. So it was getting to the point where it needed to happen in a hurry. I mean, it was so bad that one of our guys had to put a freekin' house fan in the rack to keep the cirulation good enough! Mae West is heads and shoulders above the old facility, and we've been able to build out the racks to spec, so systems stay cool. We're also sitting on something like 8 substations (aka, it's rediculous how many) so power hasn't been an issue. We did move to daily backups ongoing, so the corruption we experienced will never happen again. We bought a NAS system just for backing up billing data offsite. I'm really sorry you had such bad experiences. I hope that knowing why would make a difference to you. I really hate to read this kind of stuff in the forums -- it's bad for business, but more importantly, you guys were good customers (and yes, we have definitely had some bad customers), and it's personally sad that we lost your business. At the end of the day, we've had some growing pains. We're really struggled to find competent staff members to service our client base, which is a little bit high-end of a hosting crowd. We've built our system out right, and had to make some tough choices, and deal with some tough technological hurdles, along the way. Right now, I'm finally feel confident in our infrastructure. FireBright is actually focused on expanding operations right now. We're going to be announcing new eZ Publish products and services towards the end of January. Pete, if you guys want to come back at any point, get in touch. I will make it worth your time, and do whatever I can to take care of you. Stuart, one thing I'm confused about is that you site is still on our servers. Do get in touch with me if you're still unhappy, and I'll compensate you for your troubles with a free month of service on the house. We're a small company, focused almost entirely on this product (eZ), and we really, really, really, really want to keep our customers happy. We cannot survive without you! Most Sincerely, Jonathan Dillon-Hayes
---------
FireBright provides advanced eZ deployment with root access
http://www.FireBright.com/
|