Business Plan for Virtualization and Storage

It's been year since I walked into my new job and noticed to my surprise that we were still pairing a single APP to one OS to one Server (hardware).  The IT department was rolling out thick clients and the project was a mess.  This would have been a great oportunity to start to implement cloud services with thin clients (VDI).  Since my new job including SDLC (system development life-cycle) planning I was in the perfect position to propose the technical solution and accompanying business case for moving to a more "cloud"-focused architecture in our enterprise.

The pitch included a hypervisor based on VMware (although if I had my way I would have conducted a feasibility study and proved out a KVM-Linux based hypervisor...thus reducing costs).  Also included in the business case was a 3-year refresh of our servers and how the upfront costs of a virtualized system would more than pay for itself.  We also had requirements for security appliances such as firewalls, IDS/IPS, VPN and packet capture devices that could be virtualized.  Add to that the upcoming trend in network virtualization (whether NFV or SDN, you choose) and the savings is obvious.

Now that we have everyone on-board, the first task was the upgrade our EMC SAN from an older CX to a newer VNX platform.  Which brings me to this posting.  I've been managing the team and acting as techinical advisor on this upgrade.  The VNX is a complex platform allowing for a number of flexible configurations.  This leads one to a number of possibilities when planning for the network and primary/backup DR sites.  Just look at how many "processing engines" comprise the VNX series and the number of connections coming out of these guys.

Good news is we're almost there.  Most of the design considerations and site planning is complete and we're about ready to deploy.  Transition planning involves replication from old to new SAN and arranging for a failover to our DR site before cutting over.

After the SAN is done, we move forward with building out our VMware hypervisor environment at the primary and backup site with full VMware high availability, site recovery and vMotion capability ensuring 99.9999% uptime.  Then on to the OS/APP projects where we will stand up a new security system based on the Nessus scanner and passive packet analyzers as well as a SPLUNK SIEM.

Note of interest.  We interviewed a company named Nutanix.  They have a very interesting proposition with "virtualized storage" utilizing a "pay as you grow" model.  I personally would have conducted a trial with their system had it not been for the fact that we have been with EMC for a number of years.  Keep an eye on these guys, they are on FIRE!

Lab Testing - Slow and Low DDoS Attack using Switchblade4

A typical Disributed Denial-of-Service (DDoS) attack requires the direct control of a massive amount resources (usually bot-infected machines of unsuspecting victims) in order to create resource exhaustion issues on a victim machine by initiating a large number of connection or file upload/download requests to that machine from all of the hosts.

When referring to a DoS attack using a single machine and the desired end result being the same there is a more stealth-like tactic utilizing a "low-and-slow" or "slow-rate" method.  This method is preferred over other approaches such as a SYN-Flood because it requires EXTREMELY LITTLE RESOURCES from the host.

The DDoS approach described above is analogous to a teacher being asked a unique question by a million students simultaneously.  The teacher would not able to answer all of those questions or any new student questions that came in.

The DOS approach is analogous to a person being told that a single person will ask them a hundred questions and the answers will have to wait until all questions are asked.  The catch being that each question is being asked slooooooowly.  Again, the teacher would be tied up waiting for the questions to finish so the answers can be given so no new students will be able their questions asked until all one hundred are asked and answered.

A low and slow tool that's come out recently and I've been wanting to try out on my lab servers is Switchblade4.  I finally have some free time to check it out and found within just minutes that it does what it says it does.  It took down both my windows 2008 server running IIS as well as my CentOS 6.5 server running apache.  Next I'll be researching and implementing mitigating controls to monitor and protect the servers.

The Beauty of Simplicity - Arch Linux | ARM

After years of working with Red Hat/CentOS, Ubuntu/Debian and BSD I've found myself in a situation that warranted a different type of Linux - one built for the "ARM hobbyist".  As CPU and Memory increase exponentially (Moore's Law), Operating System minimum requirements (especially Linux) stay relatively small in comparison leaving us with smaller and smaller form factor computing devices, such as the Cubox-i and Raspberry Pi, with which we can run full-featured Operating Systems and applications on with little to no lag.

The requirement began a few months back with a packet generator and tester that could be deployed easily by technicians in the field.  It would be lightweight and low cost ($99, 2oz, 2"x2" cube) but we had one problem...there would be no monitor/keyboard/mouse to determine 1) if the local device actually could see the far-end IP/Device, 2) if the test started or hung/died and 3) when the test ended and what the results of the test were.  We solved the results question by writing the results to a micro-USB (small form-factor) plug.  The other 3 would involve coming up with some sort of visual cue for the tech.  We found a System-on-Chip LED next to the RF receiver on the little box.  Problem was Ubuntu could not access it.  Only ARM could!

Once we figured out how to get the ARM built on top of the micro-SD card in the Cubox we were up and running and found the folder/file that controlled the LED.  Created a bash script containing a do-loop and sleep commands in between turning the LED on and off and wallah...we had morse-code-like capabilities!  We could signal information to the techs now.  Arch is similar to other Linux ditros but I like the simplicity and minimal environment in which to build what you want and need.  More to come!

Software Defined Networking (SDN) and it's Potential for Real-Time Defense

Working with the team on the day job researching and soon-to-be-testing SDN controllers in the lab.  Had a stimulating conversation on the "state of networking and security" with someone with whom I trust in our company...it was an intellectually stimulating conversation.  He is on the SDN working group with other prestigious fellows and I he brought up a paper, a PhD dissertation, that touched on at a high level the topic of "Active Network Defense using SDN".  He didn't have any more information but it got me and the team thinking...Yes...the north and southbound interfaces all exist to do this.  If we had to design it the architecture might look something like this:

 

OpenFlow rules can be written to match on any combination of 12 items:

Ingress Port
Ethernet Source (Layer 2 source)
Ethernet Destination (Layer 2 Destination)
Ether Type
VLAN ID
VLAN Priority
IP Source
IP Destination
IP Protocol
IP ToS
TCP/UDP Source Port
TCP/UDP Destination Port

An example flow table entry for a layer2 port would look like this…

Rule:
     In Port:  e1/4
     In VLAN:  Untagged
     Source Mac:  0024:2214:a5ba
     Destination Mac:  0024:2025:2a44
Action:  FORWARD
     Out Port:  e1/2

This essentially says that any Ethernet frames coming into ingress port ethernet1/4 with no vlan ID with a specific source and destination MAC address, forward it out port 1/2.  Reminds me of a route-map at layer 3 but instead it's happening at layer 2 instead.

Security Compliance Monitoring in a Virtual "Cloud" tied to Risk Management Framework

I've been wanting to do this for some time now.  Build a "security compliance monitoring solution in-a-box".  A virtual box that is.  My VMware lab server has about 8 virtual machines running all sorts of clients and servers, of several OS types, and I thought it would be great to design and deploy an entire security solution based on the MONITOR aspect of regulatory compliance frameworks.  I came up with the drawing below:

 

The challenge I've given myself is:  Can I do this all with opensource software and can it be monitored/managed easily with as many automation functions using custom scripting where possible.  Within 10 minutes I had four VMs fired up installing (Simultaneously) a network IDS/IPS server, a SIEM, a Vulnerability Scanner and an all-in-one Firewall/Router/Proxy/web-content-filter appliance.

 

Next I'm looking into a DLP (Host and Network) solution.  Then on to the exciting part, to INTEGRATE THEM ALL!  I'll be installing agents on the clients I already have (Windows XP, 7, Ubuntu/Debian and Redhat/Ubuntu) in the virtual environment.

One of my former SNRians is off to Abu Dhabi

I found Stan Chua at the University during his last year there in the fall of 2009.  He was young, shy, and unsure of his future but willing to push hard and eager to learn all he could about security.  It's been five years and we've long since parted with SNR but we've kept in touch and today he mentioned he was moving to Abu Dhabi to be part of a cybersecurity team.  He's one of the best engineers in Hawaii and these islands are just too small for him.  I know he'll do well since the way we continue to develop our wings is to constantly keep jumping off of cliffs!

Consultative Selling

A dear friend called last night and asked if I could help out with one of their mid-sized healthcare accounts.  He wanted me to act as an "advisor" in matters related to security controls and HIPAA.  I'm familiar with the HIPAA administrative, physical and technical security requirements and thought it would be great to be part of the team.

It became apparent to me after about two minutes sitting with the "team" that not only had they brought me there to play as if I was their Information Security Officer, but they were pushing the client for a hard sell on a storage system to meet one of the encryption requirements for PII/ePHI.  I couldn't help myself as I saw the client in a semi-dazed and confused trance as the "team" bombarded them with technical babble and regulatory draconian speak...I had to jump in!
I found my chance...there was a pause from the team where they were seeking a reply from the client, the doctor and nurses were still trying to figure out what "data at rest" meant, and I jumped in.... "Excuse me, can I ask a question?".  The client looked at me with a stare that I couldn't make out, either it was a "yeah what do you want? I'm busy trying to reconcile all the data here in my head" or "please save us".  I wasn't sure which it was.

I asked the client if they had ever taken a HIPAA self-assessment survey.  Aha!  It was as if that was the missing link...they got it!  The excitement grew as I held their hand and took them on a journey of possibilities that would ensue once they had all the answers to their questions, and it was interesting to see how the "team" was dumbfounded.  The "team" looked like I had just stolen their lunch money, but I knew they saw value in what was being presented.  We closed up the meeting with the client asking for a follow up next week after they finish the assessment (I had the "team" take that one for action) and as we all walked to our vehicles, I explained to the "team" that although they had not walked away with a signed contract this day, they would see the value in what we are doing as it paves the way for becoming a STRATEGIC PARTNER in the long-run and will lead to not only this single box solution sale, but many others that they cannot begin to imagine occurring over the next two years.

Data Visualization - Hacking around the World

Watch hacking attempts unfold before your very eyes.  A friend in D.C. sent this my way.  Best viewed in google chrome.

http://map.ipviking.com/

For added pleasure, minimize your browser and bring it back up after about a minute to see all of the queued attempts happen simultaneously.

Noticed a lot of attacks coming from China and Russia with a target of Kirksville, MO.

Fake Virus Alert Website with Interesting artifacts

Got a call from my wife's friend today saying she was given a 1-800 "Microsoft" number to call as shown on her laptop screen.  Said the virus alert warning wouldn't go away.  Of course my response was "No, you didn't call the number did you" to which she responded YES!  They asked for her credit card information so they could bill her for...now get this... a total of four firewalls and two antivirus programs.  Total $199 USD.  I'm glad she didn't fall for it and hung up right way.  But she did say someone got access to her laptop and was "poking around saying uhuh...hmmm..yes you're infected alright!".

As I looked through her laptop for artifacts of the suspected compromised I came up with the site she visited where the virus alert image showed up...pretty amateur if you ask me.

I ran the page on my workstation in a sandbox to analyze it.  Noticed some references/redirects to a few other pages one of which included an "update" for adobe flash which tried to install on my machine.  Another site was non-functional.  Ran a few tools and found a Trojan and other malware artifacts.  Recommended a reimage of her machine.

NIST Risk and Vulnerability Management

Lots of activity at work around NIST Cybersecurity Framework and related NIST publications.  Working an Incident Response Plan to 800-61, Configuration Management Policy, Security Awareness and Employee Training Policy, and paving the way for Risk Management and Vulnerabilty Management policies all mapped back to NIST.  Still need asset/inventory list so we can complete the long-term goal of a holistic vulnerability and risk management framework.

Smartgrid Server Online

Another project of mine: Smartgrid Server.  The objective is to create a simple interface for monitoring, reporting and remediating security events on Linux devices such as my Dropgrid appliance.  Used a technique that hackers use to infiltrate networks and create backdoors - Reverse SSH Tunnels - to connect to my smartgrid server and send logs and activity messages over a secure channel.  Preplaced public-private keys for automatic tunnel creation.  I am also building a small Linux device that could be used to conduct pentest via Metasploit or at a minimum vulnerability assessments using NMAP.  Also have an initial version of arachni web apps scanner on the server to do external vulnerability assessments just for added value (since my Dropgrid server utilizes apache).

Actual screenshot of the smartgrid server GUI.  Written in PHP using MySQLi backend.

Dropgrid Pilot doing well

Been getting feedback on the pilots of my Dropgrid beta server (www.dropgrid.com).  Salesperson at a media company says he uses it every day and people are asking him where he got it from.  They like the fact that it's his own private server and he can access devices from the road through mobile means.  A tester in Korea is using it to upload images from homes for viewing/editing later on his PC.  Other use cases involve transmitting sensitive information between a client/company and their outside attorney firm, real estate agents taking field photos with instant upload to the server and contractors looking to pull real-time files like autocad drawings while in the field over 4G.

I have been using my server every day.  I've got about 500GB on the 4TB server.  RAID still going strong.

Next Generation, Service Provider "Mesh" Packet Architecture

After almost six months of requirements definition writing, research, vendor interviews and a rigorous RFI process we've gotten the results back from the vendors.  Comes down to three vendors.  It'll be interesting to see what each came up with as we work to get our customer their next generation "mesh" network for cloud and private VPN services to customers on the island.

Work also involves heavy Quality-of-Service testing including prioritization via policing and queuing techniques.  Pretty standard stuff in today's carrier packet equipment.  Interesting to note that optical manufacturers and enterprise packet manufacturers are coming at the next-gen packet story a bit different from two different sides of the market.  One from the layer1 to 2 approach and the other from the layer 3 to 2.  Layer 1 to 2 seems a bit more inflexible but only time will tell as we conduct our feasibility studies.

Excited to see some feedback on our request for Software Defined Networking (SDN)-Capable solutions.  Especially since we've been working on building contrail (Juniper) and floodlight (BigSwitch) virtual machines in our lab to test out SDN.

Low-cost packet generator

Just completed the low-cost packet generator for Comspec Engineering LLC.  The requirement was for a mini form factor device that would generate packets at different sites on a wide area network and test both throughput and goodput.

I used a device called a Cubox that was sourced from Israel.  Installed Ubuntu 12.4 (would have preferred Redhat 6.4 but doesn't support the hardware).  Leveraged iperf for the packet generator.  Initially I had added a GUI written in PHP for ease of configuration and management but decided to use a CONF file on a microSD card instead.  Unintended consequence of the SD card was it's red LED light that flickers when you write to it.  I intend to use that as a "visual cue" for the technicians to know when something has occurred.  Think Morse Code.  For instance, write several small files to the SD card with 1 second spacing between them to signal a "dot-pause-dot-pause-dot-pause" which could mean the client does not see the server (no layer 1/2/3 connectivity).  Or use a large file write for a looooooong red light to signal the fact that connectivity was established and the data was sent and received (uni- or bi-directionally) and that stats were written to the SD card.  User would pull SD card and retrieve stats from it for review.

Future additions may include ostinato to be able to custom craft packets with TOS/DiffServ, COS (802.1p), VLAN (802.1q) among other things.  Also considering adding TCPreplay with PCAPs of voice (G.711 and G.729) and video with several threads activating to simulate real-world triple-play traffic generation on the wire.