Yes, the pun IS intended :)
This weekend, Hurricane Irene roared up the East Coast of the United States and severa lessons were learned, as it relates to secuing your IT infrastructure and data. I want to share the following two e-mail messages that Jenaly sent to all our valued clients, both before the storm and after. I hope you will be able to find some value within these messages that may help you survive any future event like this...
The following message was our preparatory e-mail, which we sent out on Thursday:
It appears likely that Hurricane Irene may make a direct hit on New England over the upcoming weekend. The latest reports suggest the wee hours of Monday morning and into the day. We want you to know that Jenaly is actively monitoring the forecasts and taking several steps to assist our clients in the event of any impact.
Please review the following list of 8 recommendations and share it with the appropriate members of your team. We obviously hope that none of us will suffer any impact, personally or professionally, from this storm. One of my favorite sayings is "Better to have a plan and not need it than need a plan and not have it", so hopefully the simple step of reading this e-mail and taking these recommendations seriously, will help you weather the storm (yes, the pun is intended) without issue.
1. Prepare For a Power Outage:
It is highly likely that you will lose power during this storm. As a result, we are strongly recommending that all non-essential equipment be powered down. All personal computers, printers, copiers, phone systems, scanners, etc. should be powered off and unplugged from the wall outlets for the weekend. All your servers are protected by battery backup power (UPS), however, it is highly likely that you may endure a power outage that will outlast the ability of your battery backups to keep your systems up and running. This will result in the UPS properly shutting down your servers when battery power runs low. If you are in an area that is known for power outages or spiking during storms like this, we also recommend you consider shutting down your servers as well, to ensure maximum safety. Obviously if you power down your servers, you may lose the ability to send and receive e-mail during this time, so you want to evaluate what the impact of not having e-mail during this time may be. For those of our clients that subscribe to our RADAR-Lite 30 Day E-mail Archive or our RADAR Unlimited E-mail Archive, you will always have the ability to send and receive throughout the storm using the Cloud hosted RADAR servers athttps://jenaly-radar.reflexion.net. All your users will need is their VSA E-mail Security username and password. Anyone who does not know this information may request it be re-sent to them. All Jenaly clients will have e-mail protection throughout the storm as all mail will be held for delivery to any server that may drop offline during the storm, either planned or unplanned. You may rest assured you will not lose any e-mail.
2. Remove Backups Off-Site:
If you are not presently using an automated off-site backup, to ensure that your backups are safely outside of your office, you will want to be certain that you have a complete copy of your most recent backup taken off-site Friday evening.
3. Find Out What Your Building Management Office Plans Are:
The loss of power is expected when a hurricane hits. Some building management groups have turned off power on the complete building before the storm hits. If these are their plans, please shut down your equipment in order to avoid abrupt shutdowns. This includes computers, servers, phone systems, fax machines, copiers, UPS's, etc.
4. Remove Equipment from Floor & Move All Equipment Away from Windows:
If your office has a risk of flooding, this can obviously create significant issues with the computer equipment. Please make sure that all equipment is at least 12" off the ground. Make sure to move all equipment at least 6 feet away from windows.
5. Document ALL Emergency & Support Numbers:
It is important to document all emergency and technical support numbers and take them with you off-site. As an example, your Internet Service Provider and telephone line provider. If you do not have access to your office, you will need to contact these providers quickly to make other arrangements. Please make sure you have this documentation on paper since computers may not be accessible. Emergency contact details for Jenaly are included at the bottom of this e-mail.
6. Take Pictures of Your Offices and Equipment:
Digital pictures can be stored on multiple pieces of equipment and backed up on your server as well. These pictures will be very handy if your office were to be effected by the hurricane.
7. Make Arrangements to Use Other Vehicles of Communication:
If your server and/or ISP are not up and running, you should be able to communicate through a different vehicle. Please remember that cell phone service should remain up, but may be interrupted. Texting or free e-mail services like Gmail may provide a short term way to communicate should your office lose power for an extended period of time. Again, Jenaly clients using our RADAR services will have these capabilities throughout the storm.
8. Business Continuity Plan (BCP):
While today is not the time, you should periodically dust off your Business Continuity Plan (BCP) plan and see how well it still applies. If possible, go through a test and make sure all of the required personnel and systems are still available and aware of their roles. Your BCP should include how to contact fellow employees and clients in case the office is unavailable, where to meet and what you will need to function as a company. Distribute cellular numbers as well as home numbers to key team members. Assign an out of area contact to be available to receive information from your employees in case of emergency.If you don’t have a BCP, you should. Once we get through this Hurricane, please feel free to contact me to discuss getting a BCP in place for your company so you will always have that plan that we all hope you will never need.
To access our emergency support during the storm, please call the main office number at 603-431-7864 and select extension 301. A live operator will ask you for some specifics related to your emergency and then page our team who will call you right back. Should our office phone system be impacted, our lines will auto forward directly to our answering service (which is located outside of New England for redundancy).
All of us at Jenaly hope you will weather this storm without issue and that your family, friends and co-workers will all remain safe and sound. If you have any questions or concerns about preparing for this weekend, please feel free to call our office at (603) 431-7864 and talk with a member of our team. We will be here throughout, with all hands monitoring the storm and any potential impact on you, our valued clients.
We next followed-up with this e-mail, this morning:
I wanted to take a brief moment to e-mail you this morning and let you know how thankful we are that your office has come through Irene without issue. We appreciate you taking our advice last week and performing the recommended steps to keep your IT infrastructure safe through the storm.
Your Jenaly team has completed our initial, post Irene, triage and we are thrilled to see all but two clients online and functional. Those two clients are currently experiencing an Internet outage, so at least they are able to work internally until this is resolved. If you are experiencing any issues this morning that may concern you, please do not hesitate to call the office 603-431-7864. We are maintaining a proactive, all hand on-deck, stance this morning and throughout the day until we are 100% confident that all our clients are working at normal capacities.
Myself, that’s a slightly different story. Last Thursday and Friday I was in San Diego for a technical training on Cloud computing (something I’ll be communicating with you about more after Labor Day). As a treat for me, my wife and son came along so we could visit some family in the area and have a little mini-vacation over the weekend before school starts this week. As you might imagine, our flight home was cancelled. We were supposed to come back on a Red Eye last night. Believe it or not, the first flight home they can get us on is Friday September 2nd! Yes, you read that correctly. We’re hoping they will add capacity as we’re one of many who are stuck, but if you have to be stuck somewhere, there could definitely be worse places. Fortunately, our girls and our dogs are safe and sound and well taken care of. I’ll be working East Coast hours all week while my wife and son get to explore the area a little more than planned. Feel free to call my mobile or e-mail anytime between 8:30 and 5 as that’s when I’ll definitely be online and working, as usual. Just for fun, I thought you might get a kick of my office view this morning, East Coast time (can’t work in the room at that hour and wake the sleepy heads :).
This first picture is at 5 AM Pacific, 8 AM Eastern and the second is an hour later at 9 AM Eastern. I never envisioned a California office for Jenaly, but as that famous saying goes, “When life gives you lemons, make lemonade”, so it appears this will be my office for the week. Fortunately, we are completely configured to work remotely and securely, so while I won’t be able to meet with you in person this week, I am able to interact with you just as if I was sitting in my office. I’m going to be the ultimate remote worker this week!
If you are not setup to work remotely and securely, you need be, just in case anything like this ever happens to you.I hope your home and family all came through the hurricane just as well as your office. Happy Monday!
As I said, I am so relieved that our staff and our clients have come through this storm easily. The two minor issues we had to deal with were ISP outages and are resolving themselves without major impact. It could have been considerably worse. The storm took a more westerly track and as you can see from this picture, the western part of our state was not so lucky.
This is a building in western New Hampshire, near the Vermont border. The servers were within 3 inches of going underwater when they were rescued.
If this is not a striking example of why it is so important to have a business continuity plan and automated, secure and monitored remote backup, I'm not sure what is. Had the data all been on-site, and the techs been just a few minutes later to rescue those servers, this could have been truly catastrophic. Instead, the servers and data are safely off-site and the staff is working remotely, either from home, or if they don't have power and Internet, from a safe location where they do, so that their operations are able to continue, even while they deal with a very serious facility issue.
As I mentioned above, for me, secure and reliable remote access have saved my day. We were not allowed to rebook our flight in advance of the hurricane, at least not without paying rediculously high change fees. Our flight was finally cancelled Friday evening and jetBlue is really struggling to get their capacity back in shape, as are all the airlines. Because US airlines run so lean these days, they just don't have the available aircraft to put on extra flights to get stranded people home. They have to insert all these people into what unsold seats may be left on the flights they regularly schedule. That's why my family and I don't have seats until Friday, September 2nd. The airlines really need some motivation to do a better job dealing with situations like this and those that do have seats available, should not be aloud to price gouge to the tune of $695 and $1,275 per ticket. Yes, you read that right, PER TICKET, for the only two flights we could find between now and our scheduled return on the 2nd. That just not right and I'm not too thrilled about that, but at least my family can enjoy the sites and attractions around beautiful San Diego, CA and I can sit by the poolside with power and high speed Internet and securely connect to the Jenaly office and work with our team and our clients. The only thing I'm not able to do is meet with a client in person, but under the circumstances, I'm sure they will understand. I do have to give kudos to the Marriott Marquis and Marina in San Diego for helping us deal with our unexpected stranding. Not only have they allowed us to remain in our same room, they have given us a steeply discounted rate for the extra time, nearly matching our group rate that we had during our planned stay. Their willingness to go the extra mile is truly appreciated and one of the reason why Marriott is my preferred hotel brand. Thank you!
So, there you have it. The key messages here are to prepare and be proactive. Having all your emergency contact information assembled and your data safely backed up and off-site is critically important. Being able to work remotely and securely, without notice is equally important if you want to stay productive. Sure, I could have just taken a forced vacation, but being the person that I am, I want to work and keep our business and our clients projects moving forward, without having to work around my situation as an inconvenience or worse. Because of the time difference, I get to knock off at 2 PM local time here on the West Coast and still enjoy a few hours of play each day. It just means getting up a little earlier than usual to be online for our normal East Coast starting time :)
I truly hope you and yours came through the hurricane without any problems and I hope I have shared some good information and tips that will help you now and in the future.