UNIX System Monitoring

Matt Critcher recently posted a review of Monit on his blog.  We have been trying it out as a monitoring solution on our servers at Pleth.  I get the impression from Matt’s post that he likes the software pretty well, I know he’s not a big fan of any kind of bloated software.

I checked Monit’s website this morning and was very impressed to see that they also have a free iPhone application that will allow you to track your systems from your phone.  How cool is that?    The software itself is also pretty cool. Monit can start a process, restart a process if it does not respond and stop a process if it uses too much resources.

Being a developer, one thing that I can see might be useful is that Monit also allows you to monitor files, directories and filesystems for changes, such as timestamp changes, checksum changes or size changes. There are several other solutions out there that will all you to track remote hosts also but they don’t have all of the functionality that monit has I don’t think. With Monit you can monitor remote hosts; and ping a remote host and can check TCP/IP port connections and server protocols.

I can remember a few years back I installed a similar solution to this on one our first servers for the purpose of tracking uptime and posting it on our website, it was a nice solution but it had so many security revisions with it that we eventually decided the risk wasn’t worth it in the long run.  I can’t remember the name of the software we were using but I am sure it’s gone away by now…  Here’s what Matt had to say about Monit on his blog:

When you maintain several servers each running several virtual machines and have anything else at all to do, it is impossible to keep your eyes on them 100% of the time. There are a handful of software packages out there that can do this for you, but most are either too bulky, too complicated, or lack the features that you want.

Scott Pinkston referred me to one the other day called Monit, which is the first one of these programs that I actually liked. Most of the others just have too much. This one is short, and to the point. Monit installs very quickly, and runs in the background as a daemon.

One very simple config file holds the configuration data for the services you want to monitor, and if you so choose, you can have a secure webpage display stats on your services. It can also be configured to email you (or call the beeper, etc…) if certain events occur, and best of all, is super lightweight. It won’t tell you if your version of apache needs updated and it won’t impress you with tons of buttons, whistles, and flash animations. But it will let you know if some process is running away, and it will kill it for you (and restart it if you have it configured)

So far, we have NO complaints. It’s licensed under the GPL, and they also sell a version (which we haven’t tried) that can monitor multiple servers from one central login for around $135.

Here’s a screenshot of Monit that I found on their website, looks pretty straightforward to me…

UNIX System Monitoring | www.mcritch.com

Focusing on MODx Hosting Solutions

I recently did a blog post for our corporate website about our MODx hosting solutions.  As I have said in the past, we aren’t your typical hosting company, we very easily could be though given our advanced infrastructure, but for us that’s just not where our heart is.  We tend to take more of a developmental approach to hosting, this is probably because we are developers first and foremost. The fact that we don’t spend as much time promoting our hosting solutions as we do our development services could probably be viewed as an injustice to those that are familiar with our infrastructure.

When it comes to our hosting solutions, the fact is that we probably have one of the most secure infrastructures on the market, with possibly the most value added services you will find anywhere else.  We also rank very high when it comes to bandwidth and our storage rates are quite comparable to those offered on a national level.  We can even provide developers and small agencies that are not interested in managing their own hosting infrastructure a viable and affordable alternative with our VPS (virtual private server) solutions.

Even though we don’t typically pursue hosting clients, we will occasionally notice a segment of the landscape that we would really like to reach, one of those segments is the MODx developer community.  Here’s an excerpt from the post I did on MODx hosting solutions that we provide.  (you can read the entire post here…) Given the fact that we know the software relatively well, possibly better than a lot of the larger, bargain basement providers out there, we feel as if we can do a better job hosting it for our clients.

One service that we have gotten very good at over time is hosting particular software, primarily WordPress and MODx. My partner Greg and I almost work exclusively with these two products and we know the inner workings of these applications and can pretty much troubleshoot any problems that may arise with little cause for concern.  Also, Matt Critcher has devoted a lot of time toward securing these applications and making sure that they run at optimal performance.  In the case of wordpress, there are a lot of hosting providers out there that can provide one click wordpress installations through their control panels (us included), but with MODx, there aren’t a lot of hosting providers out there that have worked with MODx in comparison to some other CMS solutions that are out there such as Joomla or Drupal.  This sort of is a good thing for us, it gives us an advantage over a vast majority of the hosting companies out there because we already have a good working knowledge of the software, and can provide straight to the point support on any issues our clients should have running the software. In fact, this site runs on MODx.

With this in mind, my partners and I are going to focus a lot of our attention toward the niche MODx managed hosting crowd that just wants to deploy their website from a developmental and management perspective and not have to be bothered with the actual hosting itself.  We can provide one of the best hosting environments on the planet for MODx CMS as well as some of the most knowledgeable technical support for the software you will find.  Again, we know this because we also deploy MODx solutions for our clients and manage them on a daily basis.

If you are interested in discussing your MODx hosting needs, please don’t hesitate to contact us and we will be glad to provide you with a free quote for managed or dedicated hosting.

If you have a need for MODx hosting, whether it’s 1 or 300 installations, please don’t hesitate to contact us if you think we can be of any assistance.  We enjoy speaking with other developers who utilize this platform.

Pleth, LLC | MODx Hosting Solutions from Pleth

System Administrator Appreciation Day

At Pleth, we will be celebrating System Administrator Appreciation Day!  What an awesome idea. I know that this holiday is probably not on those calendars you get for Christmas from your family each year, but it’s a pretty cool thing to add to it.  Friday, July 31, is system administrator day across the country, it’s celebrated on the last Thursday in July each year.  If you have a system administrator be sure to let them know how much you appreciate what they do behind the scenes.  And if you have never had a conversation with your system administrator when you weren’t frantic about something, you should take it even a step further and take them to lunch or something because he / she probably thinks you are a turd… 

I feel blessed because I have worked with a lot of system administrators over the years and can honestly say that Matt Critcher is quite possibly one of the best ones out there, and I have the pleasure of working alongside him putting out fires.  Of course it’s usually him spraying the water and me pointing at the fire with a dazed look in my eye, and also sometimes denying any responsibility for the fire whatsoever!

So What is a System Administrator??

A sysadmin unpacked the server for this website from its box, installed an operating system, patched it for security, made sure the power and air conditioning was working in the server room, monitored it for stability, set up the software, and kept backups in case anything went wrong. All to serve this webpage.

A sysadmin installed the routers, laid the cables, configured the networks, set up the firewalls, and watched and guided the traffic for each hop of the network that runs over copper, fiber optic glass, and even the air itself to bring the Internet to your computer. All to make sure the webpage found its way from the server to your computer.

A sysadmin makes sure your network connection is safe, secure, open, and working. A sysadmin makes sure your computer is working in a healthy way on a healthy network. A sysadmin takes backups to guard against disaster both human and otherwise, holds the gates against security threats and crackers, and keeps the printers going no matter how many copies of the tax code someone from Accounting prints out.

A sysadmin worries about spam, viruses, spyware, but also power outages, fires and floods. When the email server goes down at 2 AM on a Sunday, your sysadmin is paged, wakes up, and goes to work. A sysadmin is a professional, who plans, worries, hacks, fixes, pushes, advocates, protects and creates good computer networks, to get you your data, to help you do work — to bring the potential of computing ever closer to reality.

So if you can read this, thank your sysadmin — and know he or she is only one of dozens or possibly hundreds whose work brings you the email from your aunt on the West Coast, the instant message from your son at college, the free phone call from the friend in Australia, and this webpage.

System Administrator Appreciation Day

Critch on Drupal Security / PCI Compliance

drupal-logo Resident Server Administrator and all around Linux Guru, Matt Critcher, recently posted an entry on his blog about Drupal Security and PCI compliance.  Matt has been running Drupal on his site for a while now and it seems to be working out well for him.  I always look to Matt for security issues because he has an enormous knowledgebase between his ears when it comes to that sort of thing.

Making your website secure is one thing, but going the extra mile and making it PCI compliant is another thing.  In the past I have recommended PCI Compliance only to our clients that do e-commerce, or gather sensitive client data, but it’s rapidly becoming a buzzword in the industry.  I first learned about it in 2006 at a conference we attended in Las Vegas.

My business partners and I even partnered with HackerSafe, now owned by Mcafee, to sell PCI compliance solutions and certification to our clients back in 2006.  If you are interested in learning more about PCI compliance or securing your website, be sure to give us a shout.

This site is running in a CMS called Drupal. It, like most CMS systems, allows users to easily create, edit, and delete content and manage many features of a website. But, like most, it is not without a few security flaws. Me, being a geek, and having more than a passing interest in security, decided to try to make this site a little more secure, and possibly even PCI Compliant.

It is possible to make Drupal PCI Compliant, but it takes a little work. Now, for the record I don’t have nor do I collect data that falls under this standard, but some people do, and some run Drupal. There’s not much information about the subject on the net, so I figure it’s worth writing about. But be warned that there is a trade-off. By default, Drupal is set up to be more convenient for its users. Putting these modifications in place will make you login EVERY time you close your browser window. To me, that’s not a problem. I actually prefer that to be the case. Others, well, you may not like it as much. YMMV.

First thing that you need to do is to force Drupal to use HTTPS for login. There are tutorials all over the net on how to install mod_ssl or Apache-SSL and configure it for HTTPS traffic, which is a pre-requisite for this. There is currently no drupal module that does just this, but you can get around it using .htaccess. In the root of your website, put the following somewhere in the .htaccess file

You can read the rest of Matt’s post here: Making Drupal More Secure | www.mcritch.com

Rackspace Handled the Outage Well (IMO)

rackspacelogoI should probably state for the record that I might be a little bias on this one because I currently serve as a member of the Client Advisory Board at Rackspace and provide feedback on various aspects of their hosted email applications.  But, as I am sure a lot of you realized today, there was a disturbance in the force.  Rackspace experienced an outage today and as a result, a lot of websites and email accounts experienced downtime.  While Pleth doesn’t host any websites at Rackspace, but we do however house some of our clients email solutions there. 

Usually when there is a major NOC failure like this I like to find a nice tight hole and crawl into it, fortunately these situations rarely happen. When they do it’s a helpless feeling for those of us whose connectivity is affected by it.  Normally we immediately call into our datacenter to see what news we can get, only to find that the call center is experiencing a severe meltdown thanks to the high call volume of resellers calling in to report the obvious.  At the end of the day all we want to know is the same thing, what is the ETA for everything being back online?  Well, thanks to Twitter, that’s all changed…

Today Rackspace did a great job getting updates out to their customers via Twitter, and I wanted to commend Cameron Nouri, the Rackspace Apps Evangelist who manages the @RackApps for doing an excellent job keeping us updated.  I know it sounds weird to give a company Kudos on a day that they experienced an outage, but let’s face it, these things are going to happen.  When these things do happen, keeping clients informed should be one of the top priorities for a provider in my opinion…

I know that there are going to be thousands of people out there that were affected that aren’t as calm about the outage as I am, and I promise I understand where you are coming from.  I just wanted to point out how well they did on keeping us all in the loop. 

Also, here’s a list of system status updates from the outage…

Critch on VMWare ESXi 4.0 Migration

vmware_logo We recently made the decision to upgrade our network and virtualization environment.  Thanks to Matt’s careful engineering and configuration, Greg has been able to migrate some serious client data.  One of the things that I think is cool about our setup now is that we have the ability to move IP addresses around that might be assigned to some of our clients websites. 

As I have mentioned before, Matt Critcher is one of the most brilliant server administrators / engineers that I have ever had the opportunity to work with.  He recently blogged about our migration to VMWare ESXi 4.0 on his blog, I have posted excerpts and links to both of his posts below…

VMWare ESXi 4.0 migration

As I posted last time, we decided to move over to ESXi and so far, its been pretty smooth. ThePlanet installed ESXi 3.5 on our servers, which I quickly upgraded into 4.0. When you install the vSphere Client there is an option to install the host update utility. Run it, point it to the zip file you’ve download from VMWare’s website, and wait a bit. It works like a charm (put the machine into maintenance mode first!!). Since the servers had no clients running on them, I did it during the day (which let me sleep last night! lol!!) I’ve been copying over the VM’s from our VMWare Server machine with good ol’ scp and using the vmkfstools command on the ESXi box to convert them into ESXi format. Takes about 30-40 min per server for the whole process, which isn’t exactly quick, but we’re moving low-traffic boxes in very off hours. I moved the server that this website runs on during lunch today….;o)

Read the Entire Post…

2 days later Matt added another post regarding the migration…

VMWare ESXi 4.0 Migration, Part Deux

As I wrote about last time, Pleth’s move from VMWare Server to VMWare ESXi has been very successful thus far, but in the process we’ve discovered a couple of "neat tricks" and have proven to ourselves that the technology choices we made a few years back were indeed the right ones.

When you copy a .vmdk (vmware disk image) over from a VMWare Server machine, you have to convert it over to ESXi format. This process makes the resulting disk image the whole size that you’ve allocated. This isn’t necessarily a bad thing, but if you had it set to thin provisioning in VMWare Server your disk usage just went up. WAY up.

We were working off of a template that we could clone into a new VPS rather quickly, so we settled on a default VPS size of 20gb. There are definite benefits to provisioning the entire disk at creation, but when you sell several to different customers using thin provisioning allows you to minimize the total size of the datastore because most of them are never going to use the whole 20gb we’ve allocated. In fact, most never use more than a few hundred megabytes that their website actually takes up. We’ve got plenty of space available even if all were fully allocated because we don’t believe in overselling disk space even if some other providers do. Having smaller disk images makes moving the machines from one ESXi server to another much easier and faster if the need ever arose, thereby limiting downtime. Now, you can avoid all of this by running your VMware environment on a SAN with tons of disk space and using vCenter and VMotion but given our small size, budget, and very small number (7) of VM’s we just can not justify the extreme yearly costs associated with it.

So how do you get around this? You leverage the technology you have to the fullest extent. We spent a lot of time evaluating different products and have tried to make the decisions that allow us to provide the absolute best service given our budgetary constraints. This led us to purchase a private rack at ThePlanet, an entire class C of IP addresses, and lease some extremely powerful machines for our hosting environment. We’ve chosen great software to help manage our hosting customers, and probably our best decision was to invest in a product called R1Soft CDP. It has the ability to make multiple block-level snapshots of servers per hour and can Bare-Metal Restore one in just a few minutes. I can not say enough how well R1′s CDP works. Over the past two years it has saved us HOURS of downtime and tons of headaches more times than I can mention. The ability to make so many snapshots so quickly lets you look like a rockstar when a customer calls and says "I accidentally deleted all changes I made this morning" and you can put them back in 30 seconds.

So how did I use R1Soft to pull this off? I created a new Virtual Machine with 20gb of thin partitioned space. I then assigned the CD-ROM to a ISO image of the R1Soft Restore Disk and booted the machine. We shut all services off on the old VMWare Server VM, made a quick backup with R1Soft CDP, and shut off the VM. On the newly created ESXi VM, I did a bare-metal restore of the old one. It took about 10 min to restore about 4gb of data. One quick reboot, and voila! It’s running again. We could have used VMware vCenter Converter to do this job. It will convert almost any disk image into almost any other disk image format. It also has some distinct advantages over our chosen method namely in creating virtual appliances, but the problem for us to use it was that we didn’t have a server in our rack we could dedicate to the task — only a VPS. We would have to import the data (ie copy it to the machine running Converter) and export it to the new server, which is basically copying it twice (ESX/i to ESX/i doesn’t require this step). Our chosen method meant no copying over the data, doing a conversion, and then recreating the image. Just 10 min and it’s done (which also means only 10 minutes of downtime). Not only that, we got thin-provisioned disks back on the machines we needed it on, while the other servers got to keep their fully provisioned disks. I love technology….

Read the Entire Post…

VMWare ESXi 4.0 migration | www.mcritch.com

Project Management w/ Basecamp for Web Developers

basecamp-logo Now that Pleth has 3 locations (Batesville, Conway, and Jonesboro) being able to manage multiple projects at the same time has grown way past the enormous whiteboard at our Batesville office, and being somewhat OCD, I have always looked at a whiteboard in my office as more of a hindrance than a help anyway.  In the past our internal project management has always been up to whoever was heading up a project.  Each one of us have our own unique way of management. I am a strong Outlook user and have used Tasks to manage projects that I had going on in the past but my partners have always relied on their whiteboard pretty heavily.  We have also used a pretty robust intranet solution from Vialect (which I also highly recommend) to store our client proposals, track hourly time, etc.  

A few times in the past I have collaborated or freelanced with other companies on various projects and on a few occasions I have had the opportunity to use Basecamp before, so it was totally unfamiliar to me when we first started using it this past week.  Our reasoning for using it was we have found ourselves collaborated with a few outside agencies and providers that we partner with from time to time on projects and it was more or less their recommendation.  Being a minimum of 2 hours away from the rest of my team I am open to anything that will help us collaborate better on projects.  I am also a big fan on archiving past work, I literally have every file I have ever coded or created for a client since I got into this business about 10 years ago.  Basecamp has some excellent archival methods built in, which is pretty cool, but here are some of the things that I am really getting hooked on…

  • Universal – Odds are that when we involve freelancers in the future on projects they will already be familiar with Basecamp because it is extremely popular within our industry.  Even though it is a really straightforward solution, it’s nice to know that we don’t have to spend that extra hour of time getting someone up to speed on our system.
  • Customizable!  I have no idea why this is as important for me, but it is.  As I mentioned earlier, I am a bit OCD when it comes to tools and things I work with and for me, just being able to work inside a clean, well laid out environment.  I know that this probably doesn’t matter to a whole lot of people as long as the solution works right?  But trust me it does affect my level of participation.
  • Dashboard – 37 Signals has done a great job w/ the layout of the dashboard on Basecamp, and honestly I don’t remember it being this user friendly the few times I logged my work in the past but it might also have something to do with the fact that I understand project management a little bit better now.  There also some obvious things you notice about the Dashboard too, for instance, late items appear in red at the top of the list as well anything that is due within the next 2 weeks.  It’s also pretty cool to see what all we have assigned to each member of our team, not that we are consumed by performance or efficiency, our concerns center more around quality than anything else.
  • To-do Lists – Being a power Outlook user I am extremely familiar with Tasks.  I have tracked every client project that I have ever worked on using Tasks.  The To-do Lists in Basecamp pretty much serve the same purpose, but with a little more flexibility in that where I would normally track each project as a task of their own, I can now break down the elements of a project into stages (ex. Pre-Flight, Development, Testing, Launch, Bugfixes).  Furthermore, each of these elements can be assigned to a different member of our team, therefore eliminating the need to pass tasks back and forth to each other.  There is also some sort of endorphin high, or adrenaline rush when you check off a task from your list, or at least there is for me.  Another pretty neat feature of the to-do lists that I like is the fact that you can add items that are only visible to “need to know” members of your team, this will come in handy I am sure when we are outsourcing elements of projects.
  • File Sharing – In our industry we often find ourselves working with a wide variety of files (ex. Artwork, PDF’s, Documents, and Compressed Files).  Being able to retrieve these files 6 months to a year after a project is launched is nice, because believe me, this does happen from time to time.  I had to retrieve a vector logo file I created for a client about 4 years ago this past week and being able to locate it in about 20 minutes from my previously mentioned archival system was nice.  Having this flexibility companywide is going to be nice.
  • Message Boards – I know that email is still the killer application but sometimes shooting interoffice emails back and forth is not the most effective way to communicate.  With the message boards inside of Basecamp all messages are displayed in reverse chronological order and the best part, files can be attached to messages in the message board, so this actually makes this part of the application even more collaborative.  You can even categorize the messages inside the message board area.
  • Milestones – When I speak to a client on the front end they always want to know 2 things upfront, costs and timeframe.  Sometimes the costs aren’t near as important to them as the timeframe, especially if they are trying to rush along a product release or capitalizing on a promotion.  I like to give my clients 3 tentative milestones when we get our initial payment for the project.  The first milestone will be the start date, since we often times manage multiple projects, sometimes it’s not feasible for us to start on a new project the day we get our clients deposit, so we have to give them some sort of tentative start date.  The next milestone is the test date, and of course a lot of this has to do with when I have all of the required assets from the client to get started, when clients drag their feet providing me with logos, content, etc., this can move the testing date back later than originally projected.  Another milestone is the launch date, and this is sometimes hard to call and really dependant on the first two milestones and how quickly they come together.  With Basecamp I can project these milestones and stay on track and prioritize my time a lot better than ever before.  Also, Basecamp allows you to subscribe to your Milestones in iCalendar format, and they are even color-coded inside of Basecamp, which also really helps me visually to see the big picture.
  • Time Tracking –  For us, our billable time is our bottom line, it is for this reason that time tracking is so critical.  With Basecamp we can log our time on project elements and see a full log of our time entered on a project, therefore giving us another valuable look at the big picture.  Tracking time against to-do list items might be one of the biggest advantages to this software for a company like ours.
  • Project Overview – Speaking of getting the big picture on a project, the overview section for each projects shows you everything, including milestones (late items are listed in red), and things that are due in the next 2 weeks are listed in the mini-calendar area.  There is also an RSS Feed for each project which is also an added plus.  With the project overview you can also see who is assigned what and it’s a great motivator, especially if the ball is in your court and there are others waiting on something you have to provide.
  • Comments on Messages – I really like the way to communicate to specific members of our team on components that require their involvement.  For instance, if I get as far as I can in the development process and I need Matt or Greg to do something server side before I can continue, I can add them to that particular element and send them a message letting them know that the ball is in their court.  They can also provide comments back in the event they should have questions, etc. 
  • Expandable – There are a lot of add-ons and extras out there that are available as add-ons for Basecamp.  These extras and add-ons can do everything from subversion to accounting.  Since we are adjusted to our internal accounting solution we probably won’t jump into any of these anytime soon, it’s still pretty nice to know that these products are already on the market and mature in their feature sets and stability should we ever decide to transition. 

Granted, I know that overall standardization for project management in our industry is non-existent, that’s probably due to the fact that we are all more or less pioneers since the Internet hasn’t been around all that long, but when a lot of organizations the size of our company and freelancers alike all start embracing tools like Basecamp, we can lay the foundation toward standardization and efficiency.

I speak from the perspective of a partner in a web development firm, but I can honestly see where Basecamp could also be a very useful tool for other industries out there, even further than Graphic Designers and Content Writers, but also wedding and event planners, teachers, consultants, etc.  If you are like me and always on the lookout for ways to improve your business, take a look at Basecamp and see what it can do for your business.  Here are a few companies that utilize Basecamp already in their operations:

conpnbq

Critch on VMware, Apache, PHP/MySQL

I am happy (bordering on giddy) that our server engineer / administrator Matt Critcher is now blogging, dude is probably one of the sharpest guys I have ever met and he is an all around cool guy to hang with too, but beware of the fancy cheese he brings to dinner parties because you could find yourself in the emergency room on New Years Eve thanks to a long-standing penicillin allergy.

As some of you might know we made the transition to Virtualization a while back and have been extremely happy with the versatility it has brought us with our managed hosting and vps products that it has allowed us to bring to our clients, but with growth there can also be growing pains, it is for this reason that I am so glad we have Matt in our corner, dude knows his stuff and he can get to the bottom of an issue better than anyone I have ever worked with.

Lately we have been transitioning to VMware and have had some issues w/ websites that are slow to respond via browsers, but yet they still ping out okay.  It’s been a weird week or so, here’s a post that Matt put together the other night about the issues, I thought maybe someone else could benefit from his findings down the road:

I posted a few weeks back that Pleth had transitioned some of their equipment over to VMware Server and for the most part it’s been a very smooth process. But, as of late we’ve ran into some slowdowns, especially on the VPS with Plesk (which happens to host several of our websites). After doing a bunch of research and spending many a late hour digging through tons of mpstat and other sysutils data I think I found the culprit(s).

VMware Server, unlike the ESX/ESXi products, does not run in a Type 1 Hypervisor. This means that the underlying OS (in our case Red Hat Enterprise Linuxwas tuned out of the box for a general all-purpose server. This configuration isn’t always optimal for a Type 2 Hypervisor. It works just fine as long as things are "normal," but as the new VMware server got a larger load (in terms of I/O and CPU) performance went downhill.

One of the major problems has to do with how VMware Server uses disk-backed memory files (*.vmem). There is great debate on the web whether or not you should disable them, but one thing that is clear — when a site is busy, the file will be updated with memory information to reflect the changing memory of the VPS in question. This is where the problem lies — servers with unga-bunga hardware RAID solutions with 15K RPM disks and tons of spindles have a less of a problem with it but moderate quad-core Xeon and SAS disks in a RAID1 configuration like we and most other webhosts our size have it is a bigger issue. All those writes causes a wait-state in the CPU and therefore a backlog of transactions to be processed causing said server slowdown.

One way to deal with this is to modify the /etc/sysctl.conf to add (or modify) the following parameters:

vm.dirty_background_ratio
vm.dirty_ratio

I set my vm.dirty_background_ratio = 2 and vm.dirty_ratio = 85

Basically what these 2 parameters do is dictate the percentage of memory that can be "dirty" before it begins to flush (background_ratio) and the percentage of memory that can be "dirty" before a forced flush begins. When these files are updated, we can either have them done in the background (hence the low number for background ratio) with pdflush which allows other processes to continue to run, or we can have them queued up and wait for a synchronous (forced) write causing the iowait states (hence the large number for dirty_ratio). The big gap between background writes and synchronous is to try to keep the background writes coming consistently and avoid the synchronous writes as much as possible. You’ll have to play around with these figures to see what works best for you. See this page about half-way down for a little more in-depth explanation of these two parameters.

I also made some configuration changes to PHP and Apache to try to get a tad bit more performance out of each of them. I had written out a whole list of stuff that I’d modified to post here, and as I was looking for websites to help explain the modifications, I stumbled upon this website from IBM that lists pretty much every change that I made to Apache and PHP.

If you want to tune your MySQL database, this website is invaluable. It explains almost every parameter that you can possibly adjust and how to adjust them. One that it doesn’t really get into though is

innodb_flush_log_at_trx_commit

Setting this to "2" will force the system to write out any changes to the transaction log when the commit occurs but will only cause a flush of this data from memory to disk once every second (which gets stuck in the scheduler and is handled in the background by pdflush). The default setting of "1" will write out to file and flush this data from memory every time a commit happens. On really busy servers with InnoDB tables, this can cause slowdowns if your server really isn’t designed to handle a heavy DB load (most webservers aren’t). The drawback to this is that if the system crashes, you could lose 1 second of writes. Depending on what you are doing, this might be acceptable. Setting this to 0 will cause the write every second, but if the server crashes you might lose a ton of data because nothing is done at transaction commit. Scary, but fast (to me, scary outweighs speed in this case).

None of these changes should be taken without first thinking about what might happen. We have a test box in our office that basically mirrors our production server that I could test on beforehand. The Apache and PHP config changes are easy — no server reboots required, and you’ll know almost immediately if you mess them up. If you modify sysctl.conf incorrectly, the server might not boot. Better test a few things out (a VMware VM is a perfect testbed for these settings) BEFORE you have downtime.

VMware, Apache, MySQL, and PHP Performance Tuning | www.mcritch.com

Turkish Hackers Break Into US Army Servers

armylogo Speaking from experience, Turkish hackers are probably some of the most underestimated and resourceful hackers on the planetMy partners and I have been around the block w/ some Turkish hackers in the past and even involved the FBI once during a pretty persistent onslaught and I walked away from that experience pretty impressed with their hacking talents.

Defacing websites and planting rootkits on commercial servers is one thing but hacking into anything belonging to the United States Military is another story altogether.  This is an embarrassment and it should make some people drawing a government salary a little bit on edge today.  Our government should not stop looking into this breach until they have first apprehended the hacker cell completely and cut off their arms (they are in Turkey after all, this should be okay there), and secondly they should put into place a team of established hackers like Kevin Mitnick, and some of the better ones that have never been caught, and pay them the big bucks to just sit around and try to wiggle their way into our stuff everyday and provide intelligence as to how these things go down.

Here is the story from WHIR about the breach: (and by the way, yes this is the same group that defaced the United Nations website back in 2007)

(WEB HOST INDUSTRY REVIEW) — An anti-American group of hackershave broken into at least two of the US Army’s critical web servers, according to an exclusive report by InformationWeek. Despite the advanced security and antivirus software the Defense department’s has in place, the hackers were able to breach the servers.

The hackers are based in Turkey, which is known to have ties to the al-Qaida network. However, it is still unclear if the group is affiliated in any way with the notorious terrorist organization. The attacks are currently being investigated by the Department of Defense and the US Army’s Judge Advocate General’s Office and Computer Emergency Response Team.

The group, who call themselves the "m0sted", broke into servers at the Army’s McAlester Ammunition Plant in McAlester, Oklahoma on January 26, and previously at the US Army Corps of Engineers’ Transatlantic Center in Winchester, Virginia on September 19, 2007.

In the case of the McAlester Ammunitions plant breach, visitors who were trying to access the plant’s website found themselves redirected to a page that featured a m0sted-led protest against climate change. In the Army Corps of Engineers’ attack, the hackers sent website vistorsto www.m0sted.net, which at the time contained anti-American and anti-Israeli messages and images.

The site is currently a parked domain page with airline reservation links. It is still not clear as to whether the hackers managed to steal any sensitive data from the Army’s servers.

So far, officials have followed through with records search warrants against Microsoft, Yahoo, Google, as well as other Internet and email service firms in their ongoing efforts to discover the hackers’ true identities.

According to officials, the hackers broke into the web servers by using an SQL injection where they successfully exploited a security vulnerability in Microsoft’s SQL Server database.

In the past, the hackers performed similar attacks on many other websites, including an attack in July 2008 against a site operated by international computer security firm Kaspersky Lab.

Hackers Break Into US Army Servers – Web Hosting Industry News | Daily Web Hosting News and Web Host Interviews

Critch on ModSecurity…

Matt Critcher, our server admin, posted this on his blog the other day about ModSecurity and I thought that it was worthwhile to repost.  We implemented ModSecurity a while back and hardened all of our servers to help guard us against a lot of the threats that are out there today.  Looking back it was probably one of the smartest things we ever did.  Over the years I have worked with a lot of server administrators, but I have never worked with one that has as good a grasp of ModSecurity as Matt.

From a Developers perspective ModSecurity can be a little frustrating on the front end because it will by nature shutdown or cause some elements of your applications to “break” until you get ModSecurity configured correctly and all of these core functions added as includes, but trust me, once you get everything configured correctly it sure does help you sleep better at night knowing that some hacker in India isn’t setting up a rootkit on your server through a hole in one of your applications.

Like Matt says in his post, Security is an ongoing thing, and part of that ongoing process also has to include keeping all of your open source software patched and up to date.  Here’s Matt’s post:

Since I’m back, I’ve got a few days worth of log files to dig through. A couple of years ago an old legacy PHP script Pleth was running wasn’t very secure, but was critical to the operations of a particular customer. It got hacked (well, they used it to upload a C99Shell) a couple of times before the vendor released an update. Scouring the internet for a solution, I learned of Mod Mod Security, an application firewall of sorts. It runs as a module in your Apache configuration and uses a set of user-configurable rules files to detect and prevent a number of attacks against a website. The rules list has a huge community backing, and people have written rules for about every vulnerability out there. Open Source is good no? Anyway, as I was digging through those files today it kinda shocked me to see just how much stuff mod_sec blocked. The internet is a dangerous place…..

Among the same lines, you can further protect your server by making a few small php.ini changes as well. Look for the line in yours that says

   1: disable_functions = "........

and make sure you add

   1: shell_exec,escapeshellarg

to the list there. This will prevent PHP from operating as a shell, which you really don’t need anyway (well, you shouldn’t in my opinion). There’s about a million different things you can actually disable, but some of them are needed.

Another PHP trick is open_basedir, which is a php configuration directive that sorta "jails" the scripts to whatever directories are listed in the open_basedir directive for that particular domain.

From the manual page:

When a script tries to open a file with, for example, fopen() or gzopen(), the location of the file is checked. When the file is outside the specified directory-tree, PHP will refuse to open it. All symbolic links are resolved, so it’s not possible to avoid this restriction with a symlink.

It’s not the most friendly option on the planet, but it does work and takes a bit of careful configuration to get it working right. For a site that might be considered risky, it’s worth the effort.

Just don’t be fooled into thinking that these fixes are the end-all-do-all. Security is a never-ending process. PHP is just one aspect of it.

Mod Security is good for you! | www.mcritch.com