As some of you might be aware, we (Pleth Networks, LLC) have a redundant backup datacenter located in Houston, Texas.  This is not our primary datacenter, our primary NOC is located in Dallas, Texas.  Our RDP Backup Solution runs from the Houston datacenter. 

For those of you who are unfamiliar with the hosting industry and how it operates, here’s a little clarification. We basically lease server rackspace from the Planet in both locations (Dallas and Houston) and in turn the Planet provides us with around the clock service and support as well as one of the largest redundancy fiber backbones in the world, if not the largest in the world.  We have some serious carriers when it comes to backbone providers!

With Hurricane Ike making his way toward Galveston and suspected landfall sometime this weekend I started doing a little research as to what the Planet was doing to insure no downtime during the hurricane and tropical storm.  I have to admit that I was really impressed with Kevin Hazzard’s post on the Planet’s customer forum.

For data center power, we have confirmed our fuel levels and have brought in an additional 10,000 gallons as backup,” Hazard writes. “Backup generators at both data centers and our Houston headquarters are prepared to shoulder the power needs of the data centers and our downtown offices. We pay premium fees year-round with several providers to guarantee that we are at the front of the line if we need additional fuel. With our current fuel supply, we can continuously power our facilities for up to a week without any required backup.

I also found out that according to Houston’s building codes, the datacenter is rated to withstand tropical storm winds already.  I think that they have taken a few additional steps to button down the hatch over the past week as well.  In terms of onsite personnel for support and uptime assurance I also found out that they are bringing in some key folks from the Dallas datacenter to stay throughout the storm.  Here’s another quote from Kevin…

“A set of our top technicians have volunteered to be part of the storm ride-out team,” wrote data center supervisor Eric Rice. “We wanted them to make sure their families were safe before coming in. Personally, I’m very glad to be in such a secure and well provided-for facility in events like this.”

All in all I feel even better about our network infrastructure.  Keep in mind that all of these measures are being put into place to insure that our “backup processes” run correctly, this isn’t our primary datacenter.  This is part of the reason why Pleth’s managed hosting rates are considerably higher than those of other providers out there.

Just an update about the Explosion / Fire at The Planet‘s H1 Datacenter in Houston, Texas. The staff was called into work this morning following the incident along with representatives from many of the planet’s key providers.

At last report from 1-866-325-0045, the Planet’s information line, partial power should be restored by 5pm this afternoon.  Again, fortunately none of Pleth’s primary infrastructure was affected by this outage.  Only our secondary backup infrastructure is housed at the H1 datacenter.  I want to commend the Planet on their extraordinary efforts of getting this issue resolved for their clients and for keeping everyone abreast of what’s going on.  Another update should be available at 2:30pm today.

I received a Tweet this morning at 4:55 that there was an explosion at at The Planet’s H1 Data Center this morning in Houston.  This primarily affects Legacy EV1 / Rackshack clients.  Hopefully they can get everything up and running as quickly as possible, I know that they have a pool of vendors on standby at the data center as well as the entire staff called into work.

Here is a link for status updates: H1 Data Center Offline – Status Updates Available Here – The Planet Forums.  Fortunately none of Pleth’s clients are affected by this outage at this time however there might be intermittent issues this afternoon, it’s still unknown.  Our servers are located in Dallas and our redundancy backup solution is in Houston.

Here’s a quote from Planet CEO Doug Erwin:

said by The Planet CEO Doug Erwin :
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact.

With most hosting companies the only way to get technical support is through a ticketing system.  This method requires customers to login to their online account at their hosting providers website and report their support issues via form generated email that is archived into a ticketing system.  With technical users this is usually just fine.  In fact, our NOC, The Planet, utilizes a ticket system for most support related issues, (we do however have a direct support line that we can call should the need arise).  At Pleth, we have an online ticket system that our clients can utilize but since a vast majority of our clients are non-technical users they generally prefer to pick up the phone and give us a call to discuss their support issues.  By allowing our clients this level of access to our team I fee like it’s a huge reason for our success. 

Below, John Dunsmore from the Planet posts an entry on his blog about Ticket’s in the FastLane, I thought that it was a good hosting industry topic so I decided to copy it here for your review…  I like to check the Planet Blogs from time to time, they are usually filled with some great insight and info…

Tickets in the Fast Lane

December 30th, 2007 by John Dunsmore, QA in Tech Stuff, The Planet

Non-verbal communication is becoming more and more a way of life in the customer service industry. In customer contact centers once dominated by phone calls, companies now communicate with customers via web chat, e-mail and ticketing systems.

Obviously, the technology industry has been at the forefront of this type of direct communication, with customers often placing more trust in these alternate methods than the traditional “give ‘em a call” approach.

Resolving problems with a ticketing system creates an entirely different support management paradigm than answering a phone call. Issues can be tracked and researched via the written record that exists in a ticket, but tickets don’t have the immediate feedback of a phone call. Because the vast majority of our support is handled through tickets, here’s a glimpse at the inner-workings of our support process.

When The Planet receives a customer ticket, the first thing that we do is conduct a “triage” to determine the problem that has been described and which department should take ownership. We have a staff of triage experts who do their best to insure that a ticket is routed to the right department as soon as it is received.

To avoid being “stuck in the slow lane,” give us as much detail as possible when you submit a ticket. While we can work with “my bandwidth is wrong” or “my server isn’t working,” advising us that different software showed a different amount of bandwidth traffic or that you can’t connect to your server will help us route tickets to the right department much more quickly. Give us as much detail as you can about the problem, and chances are we can resolve the problem more quickly.

Sometimes, ticket processing can be slowed when we request additional information from you by updating the ticket. If you have an open ticket, it’s important to keep an eye on it to see if we’ve asked for new information that will help us keep your account secure and resolve the issue. For us, there is nothing worse than missing one last piece of the puzzle, so we are often as anxious to hear back from you as you are to hear back from us.

When you do submit a ticket, there is a level of trust involved that we ARE working on it. It’s true that some take longer than others to process; it’s the nature of the customer service in technology. Resolving an incorrect server charge on your account will probably take less time than figuring out why a server won’t reboot, and it is our responsibility to keep you in the loop, so we have multiple systems in place to do so. Each department escalates tickets to higher priority levels when a ticket has been open too long without an update. We strictly adhere to these limits and take action when ticket times exceed our set warning levels. Our support managers are constantly updated in real time on the pending time to process all tickets we receive, and our support reps work diligently to get responses and resolutions to every ticket as quickly as possible. In the most complex tickets, cross-departmental communication or the need for additional research may keep us from resolving the individual tickets as quickly as we would like, but without these behind-the-scenes steps, we would not be giving you the best answer.

Moral of the Story: To keep your ticket humming along in the fast lane, please include as much detail as possible in your initial ticket and keep an eye out for requests for additional information. Please be patient as we work to resolve more difficult problems, and always feel free to request updates. We pay attention to comments made by our customers regarding the ticketing system, and we are always looking for ways to improve our work-flow and enhance the customer experience.

Keep this post in mind when (or should I say “if”?) you need to submit a new ticket so you can keep your tickets in the fast lane!

-John

Tickets in the Fast Lane » The Planet Blog

I blogged on this topic yesterday because I thought that it was probably one of the most bizarre things I had heard in a long time, if ever before, in the hosting industry.  I was first alerted when I saw some angry posts on the message boards out there from CI Host Customers.  Thank goodness we don’t have to worry about a situation like this with our datacenter!!!  Thank Goodness!  This is nonsense!  I am not for sure who is really to blame, and thank goodness it’s not my job to figure that out but my goodness, this is outside of my realm of understanding…

Here is an excerpt from an article on the WHIR where they (CI Host) responds to all that has been going on:

“Bottom line is, no-one likes to hear what we have to say,” says Eckles. “We can offer services or discounts, but people want money. They want compensation. But we don’t have money to give them. We’re just as victimized as our customers. They came to us because we offered them cheap colocation services. They think because we’re a corporation we have lots of money, but we make our money through volume. If we had the money, we would give it to them.”

Source: Web Host Industry News | C I Host Responds to Robbery Reports