Rackspace Handled the Outage Well (IMO)
I should probably state for the record that I might be a little bias on this one because I currently serve as a member of the Client Advisory Board at Rackspace and provide feedback on various aspects of their hosted email applications. But, as I am sure a lot of you realized today, there was a disturbance in the force. Rackspace experienced an outage today and as a result, a lot of websites and email accounts experienced downtime. While Pleth doesn’t host any websites at Rackspace, but we do however house some of our clients email solutions there.
Usually when there is a major NOC failure like this I like to find a nice tight hole and crawl into it, fortunately these situations rarely happen. When they do it’s a helpless feeling for those of us whose connectivity is affected by it. Normally we immediately call into our datacenter to see what news we can get, only to find that the call center is experiencing a severe meltdown thanks to the high call volume of resellers calling in to report the obvious. At the end of the day all we want to know is the same thing, what is the ETA for everything being back online? Well, thanks to Twitter, that’s all changed…
Today Rackspace did a great job getting updates out to their customers via Twitter, and I wanted to commend Cameron Nouri, the Rackspace Apps Evangelist who manages the @RackApps for doing an excellent job keeping us updated. I know it sounds weird to give a company Kudos on a day that they experienced an outage, but let’s face it, these things are going to happen. When these things do happen, keeping clients informed should be one of the top priorities for a provider in my opinion…
I know that there are going to be thousands of people out there that were affected that aren’t as calm about the outage as I am, and I promise I understand where you are coming from. I just wanted to point out how well they did on keeping us all in the loop.
Also, here’s a list of system status updates from the outage…
Critch on VMWare ESXi 4.0 Migration
We recently made the decision to upgrade our network and virtualization environment. Thanks to Matt’s careful engineering and configuration, Greg has been able to migrate some serious client data. One of the things that I think is cool about our setup now is that we have the ability to move IP addresses around that might be assigned to some of our clients websites.
As I have mentioned before, Matt Critcher is one of the most brilliant server administrators / engineers that I have ever had the opportunity to work with. He recently blogged about our migration to VMWare ESXi 4.0 on his blog, I have posted excerpts and links to both of his posts below…
VMWare ESXi 4.0 migration
As I posted last time, we decided to move over to ESXi and so far, its been pretty smooth. ThePlanet installed ESXi 3.5 on our servers, which I quickly upgraded into 4.0. When you install the vSphere Client there is an option to install the host update utility. Run it, point it to the zip file you’ve download from VMWare’s website, and wait a bit. It works like a charm (put the machine into maintenance mode first!!). Since the servers had no clients running on them, I did it during the day (which let me sleep last night! lol!!) I’ve been copying over the VM’s from our VMWare Server machine with good ol’ scp and using the vmkfstools command on the ESXi box to convert them into ESXi format. Takes about 30-40 min per server for the whole process, which isn’t exactly quick, but we’re moving low-traffic boxes in very off hours. I moved the server that this website runs on during lunch today….;o)
2 days later Matt added another post regarding the migration…
VMWare ESXi 4.0 Migration, Part Deux
As I wrote about last time, Pleth’s move from VMWare Server to VMWare ESXi has been very successful thus far, but in the process we’ve discovered a couple of "neat tricks" and have proven to ourselves that the technology choices we made a few years back were indeed the right ones.
When you copy a .vmdk (vmware disk image) over from a VMWare Server machine, you have to convert it over to ESXi format. This process makes the resulting disk image the whole size that you’ve allocated. This isn’t necessarily a bad thing, but if you had it set to thin provisioning in VMWare Server your disk usage just went up. WAY up.
We were working off of a template that we could clone into a new VPS rather quickly, so we settled on a default VPS size of 20gb. There are definite benefits to provisioning the entire disk at creation, but when you sell several to different customers using thin provisioning allows you to minimize the total size of the datastore because most of them are never going to use the whole 20gb we’ve allocated. In fact, most never use more than a few hundred megabytes that their website actually takes up. We’ve got plenty of space available even if all were fully allocated because we don’t believe in overselling disk space even if some other providers do. Having smaller disk images makes moving the machines from one ESXi server to another much easier and faster if the need ever arose, thereby limiting downtime. Now, you can avoid all of this by running your VMware environment on a SAN with tons of disk space and using vCenter and VMotion but given our small size, budget, and very small number (7) of VM’s we just can not justify the extreme yearly costs associated with it.
So how do you get around this? You leverage the technology you have to the fullest extent. We spent a lot of time evaluating different products and have tried to make the decisions that allow us to provide the absolute best service given our budgetary constraints. This led us to purchase a private rack at ThePlanet, an entire class C of IP addresses, and lease some extremely powerful machines for our hosting environment. We’ve chosen great software to help manage our hosting customers, and probably our best decision was to invest in a product called R1Soft CDP. It has the ability to make multiple block-level snapshots of servers per hour and can Bare-Metal Restore one in just a few minutes. I can not say enough how well R1’s CDP works. Over the past two years it has saved us HOURS of downtime and tons of headaches more times than I can mention. The ability to make so many snapshots so quickly lets you look like a rockstar when a customer calls and says "I accidentally deleted all changes I made this morning" and you can put them back in 30 seconds.
So how did I use R1Soft to pull this off? I created a new Virtual Machine with 20gb of thin partitioned space. I then assigned the CD-ROM to a ISO image of the R1Soft Restore Disk and booted the machine. We shut all services off on the old VMWare Server VM, made a quick backup with R1Soft CDP, and shut off the VM. On the newly created ESXi VM, I did a bare-metal restore of the old one. It took about 10 min to restore about 4gb of data. One quick reboot, and voila! It’s running again. We could have used VMware vCenter Converter to do this job. It will convert almost any disk image into almost any other disk image format. It also has some distinct advantages over our chosen method namely in creating virtual appliances, but the problem for us to use it was that we didn’t have a server in our rack we could dedicate to the task — only a VPS. We would have to import the data (ie copy it to the machine running Converter) and export it to the new server, which is basically copying it twice (ESX/i to ESX/i doesn’t require this step). Our chosen method meant no copying over the data, doing a conversion, and then recreating the image. Just 10 min and it’s done (which also means only 10 minutes of downtime). Not only that, we got thin-provisioned disks back on the machines we needed it on, while the other servers got to keep their fully provisioned disks. I love technology….
VMWare ESXi 4.0 migration | www.mcritch.com
Project Management w/ Basecamp for Web Developers
Now that Pleth has 3 locations (Batesville, Conway, and Jonesboro) being able to manage multiple projects at the same time has grown way past the enormous whiteboard at our Batesville office, and being somewhat OCD, I have always looked at a whiteboard in my office as more of a hindrance than a help anyway. In the past our internal project management has always been up to whoever was heading up a project. Each one of us have our own unique way of management. I am a strong Outlook user and have used Tasks to manage projects that I had going on in the past but my partners have always relied on their whiteboard pretty heavily. We have also used a pretty robust intranet solution from Vialect (which I also highly recommend) to store our client proposals, track hourly time, etc.
A few times in the past I have collaborated or freelanced with other companies on various projects and on a few occasions I have had the opportunity to use Basecamp before, so it was totally unfamiliar to me when we first started using it this past week. Our reasoning for using it was we have found ourselves collaborated with a few outside agencies and providers that we partner with from time to time on projects and it was more or less their recommendation. Being a minimum of 2 hours away from the rest of my team I am open to anything that will help us collaborate better on projects. I am also a big fan on archiving past work, I literally have every file I have ever coded or created for a client since I got into this business about 10 years ago. Basecamp has some excellent archival methods built in, which is pretty cool, but here are some of the things that I am really getting hooked on…
- Universal – Odds are that when we involve freelancers in the future on projects they will already be familiar with Basecamp because it is extremely popular within our industry. Even though it is a really straightforward solution, it’s nice to know that we don’t have to spend that extra hour of time getting someone up to speed on our system.
- Customizable! I have no idea why this is as important for me, but it is. As I mentioned earlier, I am a bit OCD when it comes to tools and things I work with and for me, just being able to work inside a clean, well laid out environment. I know that this probably doesn’t matter to a whole lot of people as long as the solution works right? But trust me it does affect my level of participation.
- Dashboard – 37 Signals has done a great job w/ the layout of the dashboard on Basecamp, and honestly I don’t remember it being this user friendly the few times I logged my work in the past but it might also have something to do with the fact that I understand project management a little bit better now. There also some obvious things you notice about the Dashboard too, for instance, late items appear in red at the top of the list as well anything that is due within the next 2 weeks. It’s also pretty cool to see what all we have assigned to each member of our team, not that we are consumed by performance or efficiency, our concerns center more around quality than anything else.
- To-do Lists – Being a power Outlook user I am extremely familiar with Tasks. I have tracked every client project that I have ever worked on using Tasks. The To-do Lists in Basecamp pretty much serve the same purpose, but with a little more flexibility in that where I would normally track each project as a task of their own, I can now break down the elements of a project into stages (ex. Pre-Flight, Development, Testing, Launch, Bugfixes). Furthermore, each of these elements can be assigned to a different member of our team, therefore eliminating the need to pass tasks back and forth to each other. There is also some sort of endorphin high, or adrenaline rush when you check off a task from your list, or at least there is for me. Another pretty neat feature of the to-do lists that I like is the fact that you can add items that are only visible to “need to know” members of your team, this will come in handy I am sure when we are outsourcing elements of projects.
- File Sharing – In our industry we often find ourselves working with a wide variety of files (ex. Artwork, PDF’s, Documents, and Compressed Files). Being able to retrieve these files 6 months to a year after a project is launched is nice, because believe me, this does happen from time to time. I had to retrieve a vector logo file I created for a client about 4 years ago this past week and being able to locate it in about 20 minutes from my previously mentioned archival system was nice. Having this flexibility companywide is going to be nice.
- Message Boards – I know that email is still the killer application but sometimes shooting interoffice emails back and forth is not the most effective way to communicate. With the message boards inside of Basecamp all messages are displayed in reverse chronological order and the best part, files can be attached to messages in the message board, so this actually makes this part of the application even more collaborative. You can even categorize the messages inside the message board area.
- Milestones – When I speak to a client on the front end they always want to know 2 things upfront, costs and timeframe. Sometimes the costs aren’t near as important to them as the timeframe, especially if they are trying to rush along a product release or capitalizing on a promotion. I like to give my clients 3 tentative milestones when we get our initial payment for the project. The first milestone will be the start date, since we often times manage multiple projects, sometimes it’s not feasible for us to start on a new project the day we get our clients deposit, so we have to give them some sort of tentative start date. The next milestone is the test date, and of course a lot of this has to do with when I have all of the required assets from the client to get started, when clients drag their feet providing me with logos, content, etc., this can move the testing date back later than originally projected. Another milestone is the launch date, and this is sometimes hard to call and really dependant on the first two milestones and how quickly they come together. With Basecamp I can project these milestones and stay on track and prioritize my time a lot better than ever before. Also, Basecamp allows you to subscribe to your Milestones in iCalendar format, and they are even color-coded inside of Basecamp, which also really helps me visually to see the big picture.
- Time Tracking – For us, our billable time is our bottom line, it is for this reason that time tracking is so critical. With Basecamp we can log our time on project elements and see a full log of our time entered on a project, therefore giving us another valuable look at the big picture. Tracking time against to-do list items might be one of the biggest advantages to this software for a company like ours.
- Project Overview – Speaking of getting the big picture on a project, the overview section for each projects shows you everything, including milestones (late items are listed in red), and things that are due in the next 2 weeks are listed in the mini-calendar area. There is also an RSS Feed for each project which is also an added plus. With the project overview you can also see who is assigned what and it’s a great motivator, especially if the ball is in your court and there are others waiting on something you have to provide.
- Comments on Messages – I really like the way to communicate to specific members of our team on components that require their involvement. For instance, if I get as far as I can in the development process and I need Matt or Greg to do something server side before I can continue, I can add them to that particular element and send them a message letting them know that the ball is in their court. They can also provide comments back in the event they should have questions, etc.
- Expandable – There are a lot of add-ons and extras out there that are available as add-ons for Basecamp. These extras and add-ons can do everything from subversion to accounting. Since we are adjusted to our internal accounting solution we probably won’t jump into any of these anytime soon, it’s still pretty nice to know that these products are already on the market and mature in their feature sets and stability should we ever decide to transition.
Granted, I know that overall standardization for project management in our industry is non-existent, that’s probably due to the fact that we are all more or less pioneers since the Internet hasn’t been around all that long, but when a lot of organizations the size of our company and freelancers alike all start embracing tools like Basecamp, we can lay the foundation toward standardization and efficiency.
I speak from the perspective of a partner in a web development firm, but I can honestly see where Basecamp could also be a very useful tool for other industries out there, even further than Graphic Designers and Content Writers, but also wedding and event planners, teachers, consultants, etc. If you are like me and always on the lookout for ways to improve your business, take a look at Basecamp and see what it can do for your business. Here are a few companies that utilize Basecamp already in their operations:
Critch on VMware, Apache, PHP/MySQL
I am happy (bordering on giddy) that our server engineer / administrator Matt Critcher is now blogging, dude is probably one of the sharpest guys I have ever met and he is an all around cool guy to hang with too, but beware of the fancy cheese he brings to dinner parties because you could find yourself in the emergency room on New Years Eve thanks to a long-standing penicillin allergy.
As some of you might know we made the transition to Virtualization a while back and have been extremely happy with the versatility it has brought us with our managed hosting and vps products that it has allowed us to bring to our clients, but with growth there can also be growing pains, it is for this reason that I am so glad we have Matt in our corner, dude knows his stuff and he can get to the bottom of an issue better than anyone I have ever worked with.
Lately we have been transitioning to VMware and have had some issues w/ websites that are slow to respond via browsers, but yet they still ping out okay. It’s been a weird week or so, here’s a post that Matt put together the other night about the issues, I thought maybe someone else could benefit from his findings down the road:
I posted a few weeks back that Pleth had transitioned some of their equipment over to VMware Server and for the most part it’s been a very smooth process. But, as of late we’ve ran into some slowdowns, especially on the VPS with Plesk (which happens to host several of our websites). After doing a bunch of research and spending many a late hour digging through tons of mpstat and other sysutils data I think I found the culprit(s).
VMware Server, unlike the ESX/ESXi products, does not run in a Type 1 Hypervisor. This means that the underlying OS (in our case Red Hat Enterprise Linuxwas tuned out of the box for a general all-purpose server. This configuration isn’t always optimal for a Type 2 Hypervisor. It works just fine as long as things are "normal," but as the new VMware server got a larger load (in terms of I/O and CPU) performance went downhill.
One of the major problems has to do with how VMware Server uses disk-backed memory files (*.vmem). There is great debate on the web whether or not you should disable them, but one thing that is clear — when a site is busy, the file will be updated with memory information to reflect the changing memory of the VPS in question. This is where the problem lies — servers with unga-bunga hardware RAID solutions with 15K RPM disks and tons of spindles have a less of a problem with it but moderate quad-core Xeon and SAS disks in a RAID1 configuration like we and most other webhosts our size have it is a bigger issue. All those writes causes a wait-state in the CPU and therefore a backlog of transactions to be processed causing said server slowdown.
One way to deal with this is to modify the /etc/sysctl.conf to add (or modify) the following parameters:
vm.dirty_background_ratio
vm.dirty_ratioI set my vm.dirty_background_ratio = 2 and vm.dirty_ratio = 85
Basically what these 2 parameters do is dictate the percentage of memory that can be "dirty" before it begins to flush (background_ratio) and the percentage of memory that can be "dirty" before a forced flush begins. When these files are updated, we can either have them done in the background (hence the low number for background ratio) with pdflush which allows other processes to continue to run, or we can have them queued up and wait for a synchronous (forced) write causing the iowait states (hence the large number for dirty_ratio). The big gap between background writes and synchronous is to try to keep the background writes coming consistently and avoid the synchronous writes as much as possible. You’ll have to play around with these figures to see what works best for you. See this page about half-way down for a little more in-depth explanation of these two parameters.
I also made some configuration changes to PHP and Apache to try to get a tad bit more performance out of each of them. I had written out a whole list of stuff that I’d modified to post here, and as I was looking for websites to help explain the modifications, I stumbled upon this website from IBM that lists pretty much every change that I made to Apache and PHP.
If you want to tune your MySQL database, this website is invaluable. It explains almost every parameter that you can possibly adjust and how to adjust them. One that it doesn’t really get into though is
innodb_flush_log_at_trx_commit
Setting this to "2" will force the system to write out any changes to the transaction log when the commit occurs but will only cause a flush of this data from memory to disk once every second (which gets stuck in the scheduler and is handled in the background by pdflush). The default setting of "1" will write out to file and flush this data from memory every time a commit happens. On really busy servers with InnoDB tables, this can cause slowdowns if your server really isn’t designed to handle a heavy DB load (most webservers aren’t). The drawback to this is that if the system crashes, you could lose 1 second of writes. Depending on what you are doing, this might be acceptable. Setting this to 0 will cause the write every second, but if the server crashes you might lose a ton of data because nothing is done at transaction commit. Scary, but fast (to me, scary outweighs speed in this case).
None of these changes should be taken without first thinking about what might happen. We have a test box in our office that basically mirrors our production server that I could test on beforehand. The Apache and PHP config changes are easy — no server reboots required, and you’ll know almost immediately if you mess them up. If you modify sysctl.conf incorrectly, the server might not boot. Better test a few things out (a VMware VM is a perfect testbed for these settings) BEFORE you have downtime.
VMware, Apache, MySQL, and PHP Performance Tuning | www.mcritch.com
Turkish Hackers Break Into US Army Servers
Speaking from experience, Turkish hackers are probably some of the most underestimated and resourceful hackers on the planet. My partners and I have been around the block w/ some Turkish hackers in the past and even involved the FBI once during a pretty persistent onslaught and I walked away from that experience pretty impressed with their hacking talents.
Defacing websites and planting rootkits on commercial servers is one thing but hacking into anything belonging to the United States Military is another story altogether. This is an embarrassment and it should make some people drawing a government salary a little bit on edge today. Our government should not stop looking into this breach until they have first apprehended the hacker cell completely and cut off their arms (they are in Turkey after all, this should be okay there), and secondly they should put into place a team of established hackers like Kevin Mitnick, and some of the better ones that have never been caught, and pay them the big bucks to just sit around and try to wiggle their way into our stuff everyday and provide intelligence as to how these things go down.
Here is the story from WHIR about the breach: (and by the way, yes this is the same group that defaced the United Nations website back in 2007)
(WEB HOST INDUSTRY REVIEW) — An anti-American group of hackershave broken into at least two of the US Army’s critical web servers, according to an exclusive report by InformationWeek. Despite the advanced security and antivirus software the Defense department’s has in place, the hackers were able to breach the servers.
The hackers are based in Turkey, which is known to have ties to the al-Qaida network. However, it is still unclear if the group is affiliated in any way with the notorious terrorist organization. The attacks are currently being investigated by the Department of Defense and the US Army’s Judge Advocate General’s Office and Computer Emergency Response Team.
The group, who call themselves the "m0sted", broke into servers at the Army’s McAlester Ammunition Plant in McAlester, Oklahoma on January 26, and previously at the US Army Corps of Engineers’ Transatlantic Center in Winchester, Virginia on September 19, 2007.
In the case of the McAlester Ammunitions plant breach, visitors who were trying to access the plant’s website found themselves redirected to a page that featured a m0sted-led protest against climate change. In the Army Corps of Engineers’ attack, the hackers sent website vistorsto www.m0sted.net, which at the time contained anti-American and anti-Israeli messages and images.
The site is currently a parked domain page with airline reservation links. It is still not clear as to whether the hackers managed to steal any sensitive data from the Army’s servers.
So far, officials have followed through with records search warrants against Microsoft, Yahoo, Google, as well as other Internet and email service firms in their ongoing efforts to discover the hackers’ true identities.
According to officials, the hackers broke into the web servers by using an SQL injection where they successfully exploited a security vulnerability in Microsoft’s SQL Server database.
In the past, the hackers performed similar attacks on many other websites, including an attack in July 2008 against a site operated by international computer security firm Kaspersky Lab.
Critch on ModSecurity…
Matt Critcher, our server admin, posted this on his blog the other day about ModSecurity and I thought that it was worthwhile to repost. We implemented ModSecurity a while back and hardened all of our servers to help guard us against a lot of the threats that are out there today. Looking back it was probably one of the smartest things we ever did. Over the years I have worked with a lot of server administrators, but I have never worked with one that has as good a grasp of ModSecurity as Matt.
From a Developers perspective ModSecurity can be a little frustrating on the front end because it will by nature shutdown or cause some elements of your applications to “break” until you get ModSecurity configured correctly and all of these core functions added as includes, but trust me, once you get everything configured correctly it sure does help you sleep better at night knowing that some hacker in India isn’t setting up a rootkit on your server through a hole in one of your applications.
Like Matt says in his post, Security is an ongoing thing, and part of that ongoing process also has to include keeping all of your open source software patched and up to date. Here’s Matt’s post:
Since I’m back, I’ve got a few days worth of log files to dig through. A couple of years ago an old legacy PHP script Pleth was running wasn’t very secure, but was critical to the operations of a particular customer. It got hacked (well, they used it to upload a C99Shell) a couple of times before the vendor released an update. Scouring the internet for a solution, I learned of Mod Mod Security, an application firewall of sorts. It runs as a module in your Apache configuration and uses a set of user-configurable rules files to detect and prevent a number of attacks against a website. The rules list has a huge community backing, and people have written rules for about every vulnerability out there. Open Source is good no? Anyway, as I was digging through those files today it kinda shocked me to see just how much stuff mod_sec blocked. The internet is a dangerous place…..
Among the same lines, you can further protect your server by making a few small php.ini changes as well. Look for the line in yours that says
1: disable_functions = "........and make sure you add
1: shell_exec,escapeshellargto the list there. This will prevent PHP from operating as a shell, which you really don’t need anyway (well, you shouldn’t in my opinion). There’s about a million different things you can actually disable, but some of them are needed.
Another PHP trick is open_basedir, which is a php configuration directive that sorta "jails" the scripts to whatever directories are listed in the open_basedir directive for that particular domain.
From the manual page:
When a script tries to open a file with, for example, fopen() or gzopen(), the location of the file is checked. When the file is outside the specified directory-tree, PHP will refuse to open it. All symbolic links are resolved, so it’s not possible to avoid this restriction with a symlink.
It’s not the most friendly option on the planet, but it does work and takes a bit of careful configuration to get it working right. For a site that might be considered risky, it’s worth the effort.
Just don’t be fooled into thinking that these fixes are the end-all-do-all. Security is a never-ending process. PHP is just one aspect of it.
Mod Security is good for you! | www.mcritch.com
Business Continuity Plan
My business partners and i have been working with consultants for the past few months to help us assess or corporate infrastructure and it has been a valuable process. We have been extremely blessed as a company, and as a result we felt that it was important to have some “outside eyes” come in and take a look at our infrastructure and give us their opinion as to some things we could improve upon. In addition to consulting us on things like Life Insurance Investments, Performance Bonding, and Operating Agreements, they have also prompted us to look at our Business Continuity Plan.
This is something that I honestly hadn’t given a lot of thought until we started discussing yesterday afternoon what would happen if one of us crashed a plane or choked on a lobster tail. Granted it would be a very tragic occurrence but it’s something that we need to definitely keep in mind for the welfare of our company. Since I have two partners we each more or less take care of managing our own ends of the business, and from day one we have always worked very well together and served the needs of our clients extremely well, but without one of us there, how would the other two pick up the slack and keep things running smoothly? This is where a Business Continuity Plan comes into play…
I sat down this morning and mapped out my home network which consists of 2 development machines and a test server. I outlined all of the critical processes that I have running from my office location and how to operate those processes. I also outlined where all of my client project data is stored from the past 11 years and sent copies of all of this to my partners, they have done the same thing on their end.
In addition, I sent a copy of my instructions to my survivors including my wife, parents, and in-laws, stating that in the event that I should become incapacitated or dead that it would be imperative that these systems and files be immediately turned over to my partners not only for the well-being of our company but also for our clients that have entrusted us with their business over the years.
The entire document only took a few minutes to put together, but it’s very detailed as to what all critical data I have at my office and it will save anyone who has to come behind me to retrieve any of this data. Maybe we were behind the curve a little bit by not already having a plan like this in place, I personally had never seen a plan like this before in the 10 years or so I have been involved with this industry, it definitely makes a lot of sense to me though and I thought that it was something that might be worthy of considering if you haven’t already got one. Another way of looking at this whole thing, if not just to give your business partners and employees the peace of mind knowing that if you should die that they would not be in a bad position moving forward, would be to think about the peace of mind you could provide your clients moving forward as well…
Just random thoughts…
Kudos to the Planet
Yesterday The Planet Datacenter experienced a widespread outage that affected literally thousands of clients around the world. Since our network is housed at The Planet our clients experienced a brief outage. The details of the outage still aren’t for certain, we know that it had to do something with a router that was not functioning correctly.
The timing for this outage could not have been any worse, our server administrator was boarding an airplane headed for France and my other partner Stephen just so happened to be in Florida. For the past 2 weeks we have been doing some gradual client migrations and phasing out some old equipment. I scrambled and called my partner Greg who was at his sons baseball game and he scrambled into the office so we could put our heads together to see what was going on.
Initially we thought that the issue may have been on our end given the fact that we have been doing a lot of migrations but about the time Greg and I got connected I glanced at my Twitter window and noticed several other hosting providers talking about their systems were down. I put two and two together and figured that a lot of these providers were probably all housing their infrastructure at the planet as well. A few minutes later Kevin from Planet updated his status to let us know what was going on with the outage.
Since the phone lines coming into the support center were busy, just having this twitter account out there giving us periodic updates about the situation really did give my partners and I peace of mind knowing that the Planet had the situation under control.
Kudos to the Planet for using Twitter to communicate effectively to your clients and providing real time updates.
The Planet – Kevin (theplanetdotcom) on Twitter
Google Container Data Center Tour
I thought that it was pretty interesting that Google posted these tours of their new container data center storage facilities on YouTube. If you are a nerd like me and you look at a Data Center Tour like others would a trip to Disneyland, you will love this…
I got a kick out of how advanced this data center is laid out, and all of the research and planning that went into it, pretty much under the radar screen of a lot of us in the industry. Way to go Google:
Extremist Websites Using US Hosts
This is a practice that has been going on for a while now but is just starting to get some attention. Apparently some of these Muslim extremist websites that are touting anti-American sentiments are actually using web hosting infrastructures located here in the United States.
I personally know of at least two other incidents similar to this where websites were being hosted that were in support of Al Qaida. This is something that is very hard to police, but I think that as an industry we should do our best to make sure our clients are all on the up and up. Before any clients gain hosting access onto any of our web servers, my partners and I always take a close look at the project and if there is anything we aren’t sure about we always try to error on the side of caution.
Honestly I hate it that this happened to the Planet, but at the end of the day there really isn’t a whole lot that they can do to prevent it from happening again, especially given the high volume of hosting accounts that they house in their data centers.
(WEB HOST INDUSTRY REVIEW) — After taking credit for a deadly March 25 attack on coalition forces, a Taliban website claiming to be the voice of the "Islamic Emirate of Afghanistan" was found to be hosted by Texas web host The Planet (www.theplanet.com).
Unbeknownst to the US web host, according to a Washington Post report Wednesday, the militant group used The Planet’s hosting services for more than a year "to rally its followers and keep a running tally of suicide bombings, rocket attacks and raids against US and allied troops." As one of the world’s largest hosting providers, The Planet was unaware of its Taliban connections until a blogger noticed the connection last week and The Planet pulled the plug on it.
The Post article connected The Planet’s incident with a greater trend of extremist groups using US hosting because of its reliability, ease-of-use, and terms of use that allow for anonymity. At around $70 a month, payable by credit card, Site Intelligence Group co-founder and extremist Muslim communication researcher Rita Katz told the Post that low-cost, high-quality US hosting has resulted in many extremist groups willing to pay US companies to host their anti-US messages.
Using Free Web Town (www.freewebtown.com), run by Atlanta-based Tulix Systems (www.tulix.com), Tora Bora Front (www.toorabora.com), another Taliban Web site, continues to post regularly about attacks on US-led coalition forces and other inflammatory material.
Such incidents fuel the debate on whether US webhosts should shut down these websites or let them continue to operate, giving US intelligence access to clues about the leadership and structure of terrorist groups. This also reminds web hosts of the importance of being aware of the companies with whom they are dealing.
