Home networking has Enterprise-grade network segmentation and security "out of the box"

For some time now enterprises have exercised best practices for wireless security by separating wireless users into their own zone/network and interrogating them with firewall rules before they access the internal wired network(s).  I have not seen this in wireless home products...until now!

First some background:  I purchased FTTH by our local telephone company and noticed they came into my home with an Alcatel-Lucent GPON ONT which split data and POTS on the ONT.  The data then went to a video-over-IP-to-Coax converter with built-in firewall/routing and switch ports.

I have an internal LAMP server (linux/apache web) running downstairs that connected to a LAN port.  I turned on wireless on the provided router since I figure "why not"!  Laptop connected but I couldn't access the LAMP server.  Ooohhhhh!!!  Troubleshooting!  I love it!

Arp table on my laptop (windows) showed no MAC when I tried to ping the server.  I know the server is still functioning as it was for over a year previously...so most likely not the server, but then again anything could happen right?  Turn on TCPdump on the server, not seeing any ARP requests and of course no replies.  Don't have hosts.deny being used so the packets should make their way up the stack to TCPdump utility (this bit me in the butt in the past when trying to troubleshoot syslogD).  Verified IPtables entries, everything looked good (turned it off just in case).  Set SElinux (this is redhat/centos box) to passive.  Still nothing in tcpdump.  Packets not even making it to the server.

Next step:  Turn on wireshark on laptop.  Okay, ARP requests going out, nothing coming back.  Then...light bulb goes on...I'll add a static IP-MAC arp entry in windows (DOS prompt) using NetSH command (windows 7 won't allow you to use arp -s anymore, access denied error, even in elevated admin mode).  Okay, check.  At this point, I've bypassed the whole initial arp stage, my laptop should see that 1) The server IP is in the same subnet as mine, thus negating the need to arp/send to the gateway router and 2) construct the packet in its entirety using the destination MAC/IP of the server.

Started with a ping, nothing.  ICMP doesn't make it to the server.  Next HTTP request, nada!  See the SYN go out, no ACK back.  Server doesn't even see the SYN.  That's when I'm convinced, the wireless router must be the culprit!  Plug into the LAN physical ethernet port and IT WORKS!!!  There are not settings on the router configuration page to disable this firewall functionality so I've since disabled the router wifi feature and placed a cisco AP off of one of the LAN ports, works great!

SDN and NFV - Virtualizing the Network | Automating Security

My SDN lab is finally coming together.  Previously I wrote about using SDN technologies to support pro-active network defense mechanisms and I'm one step closer.  It's interesting (and exciting) to be working with the tools that make up SDN as I have a background building automated provisioning tools since 1999.  This involved careful review of the skills of our operations folk and what types of functionality should be included in the application.  Today this act of having one leg in operations and one in development is referred to as DEVops.
Back to SDN.
I've tested several opensource controllers and settled on opendaylight (ODL) Hydrogen (first release) versus Helium (lastest, second release) or floodlight (Big Switch Neworks).  I plan to check out Juniper's OpenContrail SDN controller shortly, but decided to keep moving forward with ODL and experimenting with python scripts to try and create some automated flow writes.  I'll also be trying to migrate from my VMware hypervisor to Linux KVM or Openstack.

Once python scripts are built for flow writes, the next step will be to trigger on ingested data of any kind.  For our network defense solution I'm going to set triggers against security logs.  Splunk, a great opensource (paid version for larger ingest rates) is a great tool to allow us to pick up on certain events, set thresholds, and launch the python/openflow scripts.  Here are some of the things I'll be attempting to trigger on:

Denial of Service Attacks
- Large volume attacks (network floods)
- Low and Slow attacks (usually host/thread based)
Response: Redirect the traffic to a packet capture device

Insider Threats
- Unsuccesful logins or access attempts
- Unauthorized escalation of priviledges
- Movements of or attempts to access large amounts of files/data
Response: Network disconnect or limiting network throughput.  ID any staging servers being used.

Malware
- Infection points
- lateral movements
Response: Interrogate host by initiating a credentialed scan to check for deviations from baseline configurations.  Check whitelisted hashes of applications for unauthorized changes.  Place host into an isolated network segment/VLAN.  Launch honeypot/fake services and/or place in sandbox and monitor behavior of malware in order to classify and determine indicators of compromise (IOCs).

VPN Access and Network Segment Monitoring
- Close monitoring of "tollgates" accessed by VPN users.
- Follow cross-network segment access
Response: TBD

Next Generation Firewall
- Attempts to "tunnel" one application through another.
- Reverse shells
Response: TBD

Business Plan for Virtualization and Storage

It's been year since I walked into my new job and noticed to my surprise that we were still pairing a single APP to one OS to one Server (hardware).  The IT department was rolling out thick clients and the project was a mess.  This would have been a great oportunity to start to implement cloud services with thin clients (VDI).  Since my new job including SDLC (system development life-cycle) planning I was in the perfect position to propose the technical solution and accompanying business case for moving to a more "cloud"-focused architecture in our enterprise.

The pitch included a hypervisor based on VMware (although if I had my way I would have conducted a feasibility study and proved out a KVM-Linux based hypervisor...thus reducing costs).  Also included in the business case was a 3-year refresh of our servers and how the upfront costs of a virtualized system would more than pay for itself.  We also had requirements for security appliances such as firewalls, IDS/IPS, VPN and packet capture devices that could be virtualized.  Add to that the upcoming trend in network virtualization (whether NFV or SDN, you choose) and the savings is obvious.

Now that we have everyone on-board, the first task was the upgrade our EMC SAN from an older CX to a newer VNX platform.  Which brings me to this posting.  I've been managing the team and acting as techinical advisor on this upgrade.  The VNX is a complex platform allowing for a number of flexible configurations.  This leads one to a number of possibilities when planning for the network and primary/backup DR sites.  Just look at how many "processing engines" comprise the VNX series and the number of connections coming out of these guys.

Good news is we're almost there.  Most of the design considerations and site planning is complete and we're about ready to deploy.  Transition planning involves replication from old to new SAN and arranging for a failover to our DR site before cutting over.

After the SAN is done, we move forward with building out our VMware hypervisor environment at the primary and backup site with full VMware high availability, site recovery and vMotion capability ensuring 99.9999% uptime.  Then on to the OS/APP projects where we will stand up a new security system based on the Nessus scanner and passive packet analyzers as well as a SPLUNK SIEM.

Note of interest.  We interviewed a company named Nutanix.  They have a very interesting proposition with "virtualized storage" utilizing a "pay as you grow" model.  I personally would have conducted a trial with their system had it not been for the fact that we have been with EMC for a number of years.  Keep an eye on these guys, they are on FIRE!

Lab Testing - Slow and Low DDoS Attack using Switchblade4

A typical Disributed Denial-of-Service (DDoS) attack requires the direct control of a massive amount resources (usually bot-infected machines of unsuspecting victims) in order to create resource exhaustion issues on a victim machine by initiating a large number of connection or file upload/download requests to that machine from all of the hosts.

When referring to a DoS attack using a single machine and the desired end result being the same there is a more stealth-like tactic utilizing a "low-and-slow" or "slow-rate" method.  This method is preferred over other approaches such as a SYN-Flood because it requires EXTREMELY LITTLE RESOURCES from the host.

The DDoS approach described above is analogous to a teacher being asked a unique question by a million students simultaneously.  The teacher would not able to answer all of those questions or any new student questions that came in.

The DOS approach is analogous to a person being told that a single person will ask them a hundred questions and the answers will have to wait until all questions are asked.  The catch being that each question is being asked slooooooowly.  Again, the teacher would be tied up waiting for the questions to finish so the answers can be given so no new students will be able their questions asked until all one hundred are asked and answered.

A low and slow tool that's come out recently and I've been wanting to try out on my lab servers is Switchblade4.  I finally have some free time to check it out and found within just minutes that it does what it says it does.  It took down both my windows 2008 server running IIS as well as my CentOS 6.5 server running apache.  Next I'll be researching and implementing mitigating controls to monitor and protect the servers.

The Beauty of Simplicity - Arch Linux | ARM

After years of working with Red Hat/CentOS, Ubuntu/Debian and BSD I've found myself in a situation that warranted a different type of Linux - one built for the "ARM hobbyist".  As CPU and Memory increase exponentially (Moore's Law), Operating System minimum requirements (especially Linux) stay relatively small in comparison leaving us with smaller and smaller form factor computing devices, such as the Cubox-i and Raspberry Pi, with which we can run full-featured Operating Systems and applications on with little to no lag.

The requirement began a few months back with a packet generator and tester that could be deployed easily by technicians in the field.  It would be lightweight and low cost ($99, 2oz, 2"x2" cube) but we had one problem...there would be no monitor/keyboard/mouse to determine 1) if the local device actually could see the far-end IP/Device, 2) if the test started or hung/died and 3) when the test ended and what the results of the test were.  We solved the results question by writing the results to a micro-USB (small form-factor) plug.  The other 3 would involve coming up with some sort of visual cue for the tech.  We found a System-on-Chip LED next to the RF receiver on the little box.  Problem was Ubuntu could not access it.  Only ARM could!

Once we figured out how to get the ARM built on top of the micro-SD card in the Cubox we were up and running and found the folder/file that controlled the LED.  Created a bash script containing a do-loop and sleep commands in between turning the LED on and off and wallah...we had morse-code-like capabilities!  We could signal information to the techs now.  Arch is similar to other Linux ditros but I like the simplicity and minimal environment in which to build what you want and need.  More to come!

Software Defined Networking (SDN) and it's Potential for Real-Time Defense

Working with the team on the day job researching and soon-to-be-testing SDN controllers in the lab.  Had a stimulating conversation on the "state of networking and security" with someone with whom I trust in our company...it was an intellectually stimulating conversation.  He is on the SDN working group with other prestigious fellows and I he brought up a paper, a PhD dissertation, that touched on at a high level the topic of "Active Network Defense using SDN".  He didn't have any more information but it got me and the team thinking...Yes...the north and southbound interfaces all exist to do this.  If we had to design it the architecture might look something like this:

 

OpenFlow rules can be written to match on any combination of 12 items:

Ingress Port
Ethernet Source (Layer 2 source)
Ethernet Destination (Layer 2 Destination)
Ether Type
VLAN ID
VLAN Priority
IP Source
IP Destination
IP Protocol
IP ToS
TCP/UDP Source Port
TCP/UDP Destination Port

An example flow table entry for a layer2 port would look like this…

Rule:
     In Port:  e1/4
     In VLAN:  Untagged
     Source Mac:  0024:2214:a5ba
     Destination Mac:  0024:2025:2a44
Action:  FORWARD
     Out Port:  e1/2

This essentially says that any Ethernet frames coming into ingress port ethernet1/4 with no vlan ID with a specific source and destination MAC address, forward it out port 1/2.  Reminds me of a route-map at layer 3 but instead it's happening at layer 2 instead.

Security Compliance Monitoring in a Virtual "Cloud" tied to Risk Management Framework

I've been wanting to do this for some time now.  Build a "security compliance monitoring solution in-a-box".  A virtual box that is.  My VMware lab server has about 8 virtual machines running all sorts of clients and servers, of several OS types, and I thought it would be great to design and deploy an entire security solution based on the MONITOR aspect of regulatory compliance frameworks.  I came up with the drawing below:

 

The challenge I've given myself is:  Can I do this all with opensource software and can it be monitored/managed easily with as many automation functions using custom scripting where possible.  Within 10 minutes I had four VMs fired up installing (Simultaneously) a network IDS/IPS server, a SIEM, a Vulnerability Scanner and an all-in-one Firewall/Router/Proxy/web-content-filter appliance.

 

Next I'm looking into a DLP (Host and Network) solution.  Then on to the exciting part, to INTEGRATE THEM ALL!  I'll be installing agents on the clients I already have (Windows XP, 7, Ubuntu/Debian and Redhat/Ubuntu) in the virtual environment.

One of my former SNRians is off to Abu Dhabi

I found Stan Chua at the University during his last year there in the fall of 2009.  He was young, shy, and unsure of his future but willing to push hard and eager to learn all he could about security.  It's been five years and we've long since parted with SNR but we've kept in touch and today he mentioned he was moving to Abu Dhabi to be part of a cybersecurity team.  He's one of the best engineers in Hawaii and these islands are just too small for him.  I know he'll do well since the way we continue to develop our wings is to constantly keep jumping off of cliffs!

Consultative Selling

A dear friend called last night and asked if I could help out with one of their mid-sized healthcare accounts.  He wanted me to act as an "advisor" in matters related to security controls and HIPAA.  I'm familiar with the HIPAA administrative, physical and technical security requirements and thought it would be great to be part of the team.

It became apparent to me after about two minutes sitting with the "team" that not only had they brought me there to play as if I was their Information Security Officer, but they were pushing the client for a hard sell on a storage system to meet one of the encryption requirements for PII/ePHI.  I couldn't help myself as I saw the client in a semi-dazed and confused trance as the "team" bombarded them with technical babble and regulatory draconian speak...I had to jump in!
I found my chance...there was a pause from the team where they were seeking a reply from the client, the doctor and nurses were still trying to figure out what "data at rest" meant, and I jumped in.... "Excuse me, can I ask a question?".  The client looked at me with a stare that I couldn't make out, either it was a "yeah what do you want? I'm busy trying to reconcile all the data here in my head" or "please save us".  I wasn't sure which it was.

I asked the client if they had ever taken a HIPAA self-assessment survey.  Aha!  It was as if that was the missing link...they got it!  The excitement grew as I held their hand and took them on a journey of possibilities that would ensue once they had all the answers to their questions, and it was interesting to see how the "team" was dumbfounded.  The "team" looked like I had just stolen their lunch money, but I knew they saw value in what was being presented.  We closed up the meeting with the client asking for a follow up next week after they finish the assessment (I had the "team" take that one for action) and as we all walked to our vehicles, I explained to the "team" that although they had not walked away with a signed contract this day, they would see the value in what we are doing as it paves the way for becoming a STRATEGIC PARTNER in the long-run and will lead to not only this single box solution sale, but many others that they cannot begin to imagine occurring over the next two years.

Data Visualization - Hacking around the World

Watch hacking attempts unfold before your very eyes.  A friend in D.C. sent this my way.  Best viewed in google chrome.

http://map.ipviking.com/

For added pleasure, minimize your browser and bring it back up after about a minute to see all of the queued attempts happen simultaneously.

Noticed a lot of attacks coming from China and Russia with a target of Kirksville, MO.

Fake Virus Alert Website with Interesting artifacts

Got a call from my wife's friend today saying she was given a 1-800 "Microsoft" number to call as shown on her laptop screen.  Said the virus alert warning wouldn't go away.  Of course my response was "No, you didn't call the number did you" to which she responded YES!  They asked for her credit card information so they could bill her for...now get this... a total of four firewalls and two antivirus programs.  Total $199 USD.  I'm glad she didn't fall for it and hung up right way.  But she did say someone got access to her laptop and was "poking around saying uhuh...hmmm..yes you're infected alright!".

As I looked through her laptop for artifacts of the suspected compromised I came up with the site she visited where the virus alert image showed up...pretty amateur if you ask me.

I ran the page on my workstation in a sandbox to analyze it.  Noticed some references/redirects to a few other pages one of which included an "update" for adobe flash which tried to install on my machine.  Another site was non-functional.  Ran a few tools and found a Trojan and other malware artifacts.  Recommended a reimage of her machine.

NIST Risk and Vulnerability Management

Lots of activity at work around NIST Cybersecurity Framework and related NIST publications.  Working an Incident Response Plan to 800-61, Configuration Management Policy, Security Awareness and Employee Training Policy, and paving the way for Risk Management and Vulnerabilty Management policies all mapped back to NIST.  Still need asset/inventory list so we can complete the long-term goal of a holistic vulnerability and risk management framework.

Smartgrid Server Online

Another project of mine: Smartgrid Server.  The objective is to create a simple interface for monitoring, reporting and remediating security events on Linux devices such as my Dropgrid appliance.  Used a technique that hackers use to infiltrate networks and create backdoors - Reverse SSH Tunnels - to connect to my smartgrid server and send logs and activity messages over a secure channel.  Preplaced public-private keys for automatic tunnel creation.  I am also building a small Linux device that could be used to conduct pentest via Metasploit or at a minimum vulnerability assessments using NMAP.  Also have an initial version of arachni web apps scanner on the server to do external vulnerability assessments just for added value (since my Dropgrid server utilizes apache).

Actual screenshot of the smartgrid server GUI.  Written in PHP using MySQLi backend.

Dropgrid Pilot doing well

Been getting feedback on the pilots of my Dropgrid beta server (www.dropgrid.com).  Salesperson at a media company says he uses it every day and people are asking him where he got it from.  They like the fact that it's his own private server and he can access devices from the road through mobile means.  A tester in Korea is using it to upload images from homes for viewing/editing later on his PC.  Other use cases involve transmitting sensitive information between a client/company and their outside attorney firm, real estate agents taking field photos with instant upload to the server and contractors looking to pull real-time files like autocad drawings while in the field over 4G.

I have been using my server every day.  I've got about 500GB on the 4TB server.  RAID still going strong.

Next Generation, Service Provider "Mesh" Packet Architecture

After almost six months of requirements definition writing, research, vendor interviews and a rigorous RFI process we've gotten the results back from the vendors.  Comes down to three vendors.  It'll be interesting to see what each came up with as we work to get our customer their next generation "mesh" network for cloud and private VPN services to customers on the island.

Work also involves heavy Quality-of-Service testing including prioritization via policing and queuing techniques.  Pretty standard stuff in today's carrier packet equipment.  Interesting to note that optical manufacturers and enterprise packet manufacturers are coming at the next-gen packet story a bit different from two different sides of the market.  One from the layer1 to 2 approach and the other from the layer 3 to 2.  Layer 1 to 2 seems a bit more inflexible but only time will tell as we conduct our feasibility studies.

Excited to see some feedback on our request for Software Defined Networking (SDN)-Capable solutions.  Especially since we've been working on building contrail (Juniper) and floodlight (BigSwitch) virtual machines in our lab to test out SDN.

Low-cost packet generator

Just completed the low-cost packet generator for Comspec Engineering LLC.  The requirement was for a mini form factor device that would generate packets at different sites on a wide area network and test both throughput and goodput.

I used a device called a Cubox that was sourced from Israel.  Installed Ubuntu 12.4 (would have preferred Redhat 6.4 but doesn't support the hardware).  Leveraged iperf for the packet generator.  Initially I had added a GUI written in PHP for ease of configuration and management but decided to use a CONF file on a microSD card instead.  Unintended consequence of the SD card was it's red LED light that flickers when you write to it.  I intend to use that as a "visual cue" for the technicians to know when something has occurred.  Think Morse Code.  For instance, write several small files to the SD card with 1 second spacing between them to signal a "dot-pause-dot-pause-dot-pause" which could mean the client does not see the server (no layer 1/2/3 connectivity).  Or use a large file write for a looooooong red light to signal the fact that connectivity was established and the data was sent and received (uni- or bi-directionally) and that stats were written to the SD card.  User would pull SD card and retrieve stats from it for review.

Future additions may include ostinato to be able to custom craft packets with TOS/DiffServ, COS (802.1p), VLAN (802.1q) among other things.  Also considering adding TCPreplay with PCAPs of voice (G.711 and G.729) and video with several threads activating to simulate real-world triple-play traffic generation on the wire.

Raspberry Pi

Working on the Pi today.  Loaded Ubuntu 12.4 LTS and the usual "must have" apt packages.  Been wanting to create a rsync or unison personal server with external public storage somewhere else.  Trying to figure out the best way to do this.  Wanted another way to create "Dropgrid" server but in a micro-instance manner using 512MB RAM.

PFsense Linux Firewall

Needed a firewall with Client-Site VPN access with private/public keys for added security.  Had to keep costs down and found PFsense.  I had dabbled with M0nowall some time ago but it was a bit crude and difficult to implement and manage.  Not so with PFsense.  Great firewall.  Session-based firewall too!  Not next-gen, but for the price you can't beat it.

I implemented it in a virtual environment and customer also had a need for a syslog server so I set up with Splunk.  Lastly, we needed a way to introduce jitter, delay and packet loss into a network for application stress testing and found a handy utility (opensource of course) called NETEM for Linux.  Great application and I used two of the four intel pro Gig-E ports on their own private virtual switch to provide that functionality.

The Art of Deception by Kevin Mitnick

I've had this one on my "To Read" list for some time now. Just never had the cycles to commit to the book.  Got through the first few chapters and I've got to hand it to Mr. Mitnick on this one.  If the stories contained within the book are true then my hat off to him.

It's amazing how the seemingly benign information he gathers in the beginning of each story can be used to gain access to the pot of gold at the end.  I find it fascinating how a high degree of IQ/EQ when used in the right way can accomplish such amazing results.

Next up on the reading list are:
"The Hacker Playbook: Practical Guide To Penetration Testing"
"Security Testing with Kali Linux"
"Red Team Field Manual"
"The Balanced Scorecard - Translating Strategy into Action"

Infographic Resume

My wife needed help with her resume and as I researched what the "new, fresh" approach to writing CVs were, I came across some exicting resume infographics!  I thought I'd give it a try out for myself and whipped out photoshop to do this one.  Pretty cool if you ask me!