Category Archives: Business Continuity

No End in Sight: Cyber Security and the Digital Maginot Line


Yesterday my spouse was informed by a laboratory company where she was having some blood work done that she needed to provide them a credit card number that they could put on file in case our insurance company could not pay or did not pay the bill for the lab costs. This after showing our insurance card and providing proof that we are insured. Having lived with me the last 7 years she asked the woman at the counter for a copy of the InfoSec strategy asking them to “please include information on encryption ciphers, key lengths as well as information on how authentication and authorization is managed by their system and if her credit card information would be encrypted at rest”. Needless to say, they had no idea what she was talking about much to the exasperation of the people waiting behind her in line as well as the front office staff. She ended up getting her tests done but was told she would not be welcomed back if she was going to continue to be unwilling to surrender her Credit Card number to their front office for them to, digitally, keep on file.

Between the two of us, we have replaced 4 or 5 cards in the last 3 years due to various breaches, I have had to replace two and, I believe, she has had to replace 3 of them. In my case, each incident cost me around $800 that I had to wait weeks to get back and only after I went into the bank and filled out forms to attest that I did not make the charges. Each incident was about 4 hours of my time by the time all was said and done. Yes, there were lawsuits and lawyers were paid six figure sums as a result and I am sure they deserved it but at the end of the day, I was without my $800-$1600 for an extended period of time and I had to run through a regulatory maze just to get back what I had lost. No…..I never got any settlement money, I hope they spent it well. Fortunately for me, I am 46 years old now and have a great job, if this had happened to 26 year-old (still a screw-up) John, it would have been utterly devastating as I likely would have been evicted from my apartment and had bill collectors calling me. I can’t imagine calamity this creates for some folks.

I am somewhat dumbfounded that any company at any level would seek to get people to surrender their information digitally given the egregious levels of retail breaches that have plagued the industry the last few years. Forget that consumer advocacy is non-existent, while some retailers have been very forward in understanding the impact to their consumers, I simply do not see things getting better, EVER. The current method by which Cyber Security is practiced today is broken and there seems to be no motivation to fix it. This in spite of extremely costly settlements and damage to brands, the way we practice security today is deeply flawed and it’s not the Security team’s fault. Until system owners start taking some responsibility for their own security, these breaches will simply never end.

Bitching about the lack of responsibility of system owners isn’t new to me, my first “documented” rant on it was back in early 2010. As a system owner, I, almost compulsively, logged everything that went on and wrote the metrics to a centralized console. In a way, it was a bit of a poor-man’s DevOps endeavor. In doing so, I was able automate reporting so that when I came into work each morning, I would spend 15 minutes sipping my coffee and looking at all of the non-standard communications that went on the previous day (basically all internet traffic that did not use a web browser and all traffic outside the US). No, it wasn’t full IDS/IPS production but on two separate occasions, I was able to find malware before several seven figure investments in malware detection software. That is two instances in four years or 2/1000 mornings (approximately 4 years’ worth of work minus vacations, holidays etc.) where I noted actionable intelligence. That may not have been a lot but if you are one of the dozens of retailers who have had breaches in the last few years, I think it is plausible to assume the systems teams could have had an impact on the success of a breach had they been a little more involved in their own security. Don’t underestimate the value of human observation.

Why the INFOSEC is not enough?
Short of a crystal ball, I am not sure how we expect INFOSEC teams to be able to know what communication is acceptable and what communications are not. In the last few years “sophisticated persistent advanced super-duper complex malware” generally means that someone compromised a set of credentials and ran amuck for months on end stealing the digital crown jewels. Even if a policeman is parked outside my house, if they see someone walk up, open the door with a key and walk out with my safe, 60 TV (Actually, I don’t have a 60 inch TV) and other valuables how the hell are they supposed to know they should or should not be doing that. In most cases, this is the digital equivalent of what is happening in some of these breaches accept that digitally, I am sitting at my couch while all of this is going on in front of me. If an attacker has gotten credentials or has compromised a system and is stealing, expecting the security team to see this before extensive damage is done is unrealistic. With some of the social engineering techniques that exist and some of the service accounts used with elevated privileges, you don’t always have the 150 login failures to warn you. If I am actually paying attention, I can actually say, “Hey, what the hell are you doing, put that TV down before I call the cops!” (Or, my step-daughter is a foodie and she has some cast iron skillets that could REALLY leave a lump on someone’s head).

The presence of an INFOSEC team does not absolve system owners of their own security any more than the presence of a police department in my city means I don’t have to lock my doors or pay attention to who comes and goes from my house.

Police: “911 operator what is your emergency?”

Me: “I’ve been burgled, someone came into my house and stole from me”

Police: “When did this happen? Are they still in your house?”

Me: “It happened six months ago but I don’t know if they are still in my house stealing from me or not”

Police: “Ugh!!”

If someone has made a copy of the keys to my house it is not the police’s fault if they don’t catch them illegally entering my home in the same manor that the police cannot be everywhere, all the time, you INFOSEC team cannot inspect every digital transaction all the time.

Thought Exercise:
If someone has compromised a set of credentials or, say a server in your REST/SOAP tier and they are running ad hoc queries against your back end database, let’s evaluate how that would look to the system owner vs. the INFOSEC practitioner.

To the INFOSEC Practitioner: They see approved credentials over approved ports, since they are not the DBA or the Web Systems owner so this, likely, does not trigger any responses because the INFOSEC resource is not privy to the day to day behavior or design.
The DBA: The DBA should notice that the types of queries have changed and fall out of their chair.
Web Properties team: They should have a similar “WTF!?!?” moment as they note that the change from what is normally stored procedures or even recognizable SQL statements to custom ad hoc queries of critical data.

In this scenario, one in which I covered on in May of 2014, it is obvious that the INFOSEC professional is not as well positioned to detect the breach as he or she does not manage the system on a day to day basis and while several processes have INFOSEC involved during the architecture the idea that your INFOSEC team is going to know everything about every application is neither practical or reasonable. It is imperative that system owners take part in making sure their own systems are secure by engaging in a consistent level of intelligence gathering and surveillance. In my case, it was 15 minutes of every morning. Ask yourself, do you know every nonstandard communication that sourced from your server block? Will you find out within an hour, 8 hours, a single day? These are things that are easily accomplished with wire data or even log mongering but to continue to be utterly clueless of who your systems are talking to outside of normal communications (DNS, A/D, DB, HTTP) to internal application partners is to perpetuate the existing paradigm of simply waiting for your company to get breached. While we give the INFOSEC team the black eye, they are the least likely group to be able to see an issue in spite of the fact that they are probably going to be held accountable for it.

There are services from companies like FireEye and BeyondTrust that offer innovative threat analytics and offer a number of “non-charlatan” solutions to today’s security threats. I’ve struggled to avoid calling Cyber Security an abject failure but we are reaching the point where the Maginot line was more successful than today’s Cyber Security efforts. I am not a military expert and won’t pretend to be one but as I understanding, the Maginot line, the French solution to the German invasion during WWI, was built on the strategies of the previous war (breach) and was essentially perimeter centric and the enemy simply went around it (sound familiar?). So perimeter centric was it that apparently upon being attacked from behind they were unable do defend themselves as the turrets were never designed to turn all the way around. The thought of what to do once an enemy force got inside was apparently never considered. I find the parallels between today’s Cyber Security efforts and the Maginot line to be somewhat surprising. I am not down on perimeter security but a more agile solution is needed to augment perimeter measures. One might even argue that there really isn’t a perimeter anymore. The monitoring of peer-to-peer communications by individual system owners is an imperative. While these teams are stretched thin already (don’t EVEN get me started on morale, workload and all around BS that exists in today’s Enterprise IT) what is the cost of not doing it? In every high profile breach we have noted in the last three years, all of these “sophisticated persistent threats” could have been prevented by a little diligence on the part of the system owners and better integration with the INFOSEC apparatus.

Cyber Insurance Policies could change things?
Actually, we are starting to see insurance providers force companies to purchase a separate rider for cyber breach insurance. I can honestly say, this may bring about some changes to the level of cyber responsibility shown by different companies. I live in Florida where we are essentially the whipping boys for the home owners insurance industry and I have actually received notification that if I did not put a hand rail on my back porch that they would cancel my policy. (The great irony being that I fell ass over teakettle on that very back porch while moving in). While annoyed, I had a hand rail installed post haste as I did not want to have my policy cancelled since, at the time, we only had one choice for insurance in Florida and it was the smart thing to do.

Now imagine I call that same insurance company with the following claim:
“Hello, yes, uh, I am being sued by the Girl Scouts of America because one of them came to my door to sell me cookies and she fell through my termite eaten front porch and landed on the crushed beer bottles that are strewn about my property cutting herself and then she was mauled by my five semi-feral pit bulls that I just let run around my property feeding them occasionally”.

Sadly, this IS Florida and that IS NOT an entirely unlikely phone call for an adjuster to get, however, even more sad is the fact that this analogy likely UNDERSTATES the level of cyber-responsibility taken by several Enterprises when it comes to protecting critical information and preventing a breach. If you are a Cyber Insurance provider and your customer cannot prove to you that they are monitoring peer-to-peer communications, I would think twice about writing the policy at all.

In the same manor that insurance agents drive around my house, expect auditors to start asking questions about how your enterprise audits peer-to-peer communications. If you cannot readily provide a list of ALL non-standard communications within a few minutes, you have a problem!! These breaches are now into the 7-8 digit dollar amounts and those companies who do not ensure proper diligence do so at their own peril.

As an IT professional and someone who cares about IT Security, I am somewhat baffled at the continued focus on yesterday’s breach. I can tell you what tomorrow’s breach will be, it will involve someone’s production system or servers with critical information on them having a conversation with another system that it shouldn’t. This could mean a compromised web tier server running ad hoc queries; this could be a new FTP Server that is suddenly stood up and sending credit card information to a system in Belarus. This could be a pissed of employee emailing your leads to his gmail account. The point is, there ARE technologies and innovations out there that can help provide visibility into non-standard communications. While I would agree that today’s attacks are more complex, in many cases, they involve several steps to stage the actual breach itself. With the right platform, vigilant system owners can spot these pieces being put into place before they start or at least maybe detect the breach within minutes, hours or days instead of months. Let’s accept the fact that we are going to get breached and build a strategy on quelling it sooner. As a consumer who looks at his credit card expiration date and thinks to himself “Yeah right!” basically betting it gets compromised before it expires. I see apathy prevailing and companies who really don’t understand what a pain in the ass it is when I have to, yet again, get another Debit or Credit card due to a breach and while they think it is just their breach, companies need to keep in mind that your breach may be the 3rd or 4th time your customer has had to go through this and it is your brand that will suffer disproportionately as a result. Your consumers are already fed up and companies need to assume that the margin of error was already eaten up by whichever vendor previously forced your customers through post-breach aftermath. I see system owners continuing to get stretched thin and kept out of the security process and not taking part in the INFOSEC initiatives at their companies, either due to apathy or workload. And unfortunately, I see no end in sight….

Thanks for reading

John M. Smith













Advanced Persistent Surveillance: Re-thinking Lateral Communications with Wire Data Analytics

Several high profile compromises, involving breaches of trusted systems working over trusted ports, has – once again – raised the issue of lateral communications between internal hosts.  Breaches will continue as hackers evolve and learn to work around existing countermeasures that are, at times, overly based on algorithms and not based enough on surveillance.

So what is an infosec practitioner to do?

How practical is monitoring lateral communications?

Do we assign a person to look at every single build-up and tear-down?

Do we set all of our networking equipment to debug level 7 and pay to index petabytes of logs with a big data platform?

Do we assign a SecOps resource to watch every single conversation on our network?

Answer: Maybe…or maybe not.

Most of our critical systems (Cardholder Data Environment, CRM Databases, EMRs and HIS) are made up of a group of systems, some are client-server some are tiered with web services or MTS (Microsoft Transaction Services) acting as middleware and some are legacy socket driven solutions.  All of them have a common set of expected communications that can be monitored.

What if we could separate the millions of packets, and hopefully lion’s share, of expected communication from that communication which is unexpected?

What if we could do it at layer 7?

Using ExtraHop’s Wire Data Analytics Platform INFOSEC teams and application owners are positioned to be able to see non-standard lateral communications that would otherwise go unnoticed by incumbent IPS/Anti-malware/Anti-Virus tools.  The fact is, while we need the existing tools set, today’s complicated breaches tend to hide in the shadows communicating over approved ports and using trusted internal hosts.  ExtraHop shines light on this behavior leaving them exposed and positioning teams to “get their ‘stomp’ on” and stamp out these threats like cockroaches.

How we do it: 
Most INFOSEC practitioners have worked with Wire Data before though their IPS and IDS systems. ExtraHop’s platform is similar in that we work off of a span but instead of looking for specific signatures we observe and rebuild Layer 4-7 flows supporting speeds of up to a sustained 20 Gb per second. We also use a technology called triggers to support specific conditions we want to monitor and alert on (such as anomalies in lateral communications) This is a contrast from most of our perimeter defenses that scale into the megabit/single gigabit range, we are able to work up to the tens of gigabits range. The same innovation that allows us to collect Operational Intelligence and Performance metrics directly off a span port can be used to provide layer 4-7 digital surveillance of critical systems within your or your customer’s infrastructure.

While it is not practical to monitor every single transaction between your critical systems and other components of your enterprise applications, it is now possible, to monitor the unexpected traffic.  For instance, if we have a tiered application that has a web front end, RESTFUL web service tier and a back end database tier.  We can define the expected traffic that we should see between the tiers (what should be the lions share) and ignore that traffic while reporting on the traffic that is NOT expected.

Figure 1: We can write a trigger that ignores expected communications and reports on unexpected communications.



We can accomplish this task by writing two triggers, the first trigger is a basic Layer 4 detail surveillance trigger where we white-list selected hosts and protocols and only report on communications outside our expected conversations.  The second trigger is a layer 7 trigger that leverages ExtraHop’s layer 7 database fluency of SQL Server communications.  The layer 7 trigger will white-list specific stored procedures that are run from our REST tier that make up our Layer 7 expected communications.  Anything outside of these will be accounted for.

The first trigger makes the following assumptions:
The Web/Public tier comes in from hosts  and the REST tier is at  The approved ports are 8000, 8080 and 1433.

The first trigger monitors client communications from specific clients and over specific ports.  Unlike NAC (Network Access Control), we are simply monitoring unexpected communications and we are not blocking anything.  In many cases we white list the Active Directory Controllers (if a Windows environment) and we would likely white list the WSUS server for Windows environments.  With the trigger below, you would be alerted of every RDP connection, any direct CIFS access or any exfiltration attempts that utilize ports not previously approved.  This simple trigger could have warned any number of breach victims of the staging that was going on prior to data being exfiltrated.  This trigger took less than one day to write (Don’t be intimidated by the javascript, I knew none when I started and we have people who can help you with it)

Leveraging layer 7 fluency:
Layer 7 surveillance is also a critical part of preventing tomorrow’s sophisticated breaches. In the trigger below, we are watching for expected layer 7 communications. A common occurrence in many high profile breaches is the compromise of a trusted system that is allowed to traverse sensitive networks. In the example above, if the REST tier were to become compromised it is particularly dangerous due to it’s being a trusted host of the Database environment (likely an open ACL). Using the trigger below, we can monitor which stored procedures we should be seeing connecting to the database. states that there is a 156 day laps between the time a computer is compromised and the time it is detected. That is nearly six months that, in the event of a SOAP/REST tier breach, they have to run ad hoc queries against my back end database. For this reason, properly identifying anomaly’s at layer 7 (What SQL Statements are being run?) will be key in preventing/mitigating data loss and just might keep you out of the news.

So using the trigger we have created, if I run the following command (Example: emulate a breached Middleware server)

We are able to increment the counters in the ExtraHop dashboard:

Clicking on the “3” shows us the Offending IP and the Query that they ran:


And here we see that we can, in fact, send the data to Splunk or any other SIEMS product:

Digital Evidence:

You can also assign a precision packet capture to a trigger that will create a pcap file that you can use as digital evidence after the fact.


Sample Scenario: Middleware Breach
To show how a ExtraHop can detect a middleware breach (see Figure 2) using the two triggers above you would first catch the rogue queries being run with the layer 7 surveillance while ignoring the common stored procedures.  ExtraHop’s Wire Data Analytics will also catch the communications with the Exfiltration/staging server because the communications are outside of those we set in the trigger.    ExtraHop sees this communication and implements the steps in the triggers, from the first trigger it sees the REST tier communicating with an unknown host over an unknown port.  In the second trigger it sees the ad hoc queries mixed in with the normal stored procedures

Figure 2: SOAP/REST Tier Breached:
In figure 2 below you see how an ExtraHop Wire Data analytics appliance can be written to ignore expected traffic and only report on unexpected traffic.



There has been a lot of talk about the need to monitor lateral communications but there has been little practical information on how to do it. Your digital surveillance strategy with Wire Data Analytics will be a living process that in which you periodically update and evaluate your connectivity profile. This is a crucial process often overlooked in INFOSEC.

Using ExtraHop’s platform to build out a surveillance strategy makes the once daunting prospect of monitoring lateral communications a workable process that can provide peace of mind.   We need to accept the fact that we cannot keep all Malware from getting inside; there are two types of systems, breached and about-to-get breached.  I think we need to look at our INFOSEC practice from the WHEN perspective and not the IF perspective.

As infiltration attempts become more sophisticated the ability to spot the building blocks of a data breach will become increasingly valuable.  ExtraHop can provide that extra set of eyes to do the heavy lifting for you while reporting on communications beyond those you expect, reducing the burden of additional auditing.  The reporting/surveillance framework also allows for application owners and shared services teams to get involved in INFOSEC by reviewing the periodic reports and, hopefully in most cases, deleting them.

Wire Data Analytics with the ExtraHop platform can provide your organization with a daily, weekly and monthly digital police blotter giving you the information you need to stay vigilant in the face of this new breed of threat.

Thank you for reading


John M. Smith