Advanced Persistent Surveillance: Re-thinking Lateral Communications with Wire Data Analytics

Several high profile compromises, involving breaches of trusted systems working over trusted ports, has – once again – raised the issue of lateral communications between internal hosts.  Breaches will continue as hackers evolve and learn to work around existing countermeasures that are, at times, overly based on algorithms and not based enough on surveillance.

So what is an infosec practitioner to do?

How practical is monitoring lateral communications?

Do we assign a person to look at every single build-up and tear-down?

Do we set all of our networking equipment to debug level 7 and pay to index petabytes of logs with a big data platform?

Do we assign a SecOps resource to watch every single conversation on our network?

Answer: Maybe…or maybe not.

Most of our critical systems (Cardholder Data Environment, CRM Databases, EMRs and HIS) are made up of a group of systems, some are client-server some are tiered with web services or MTS (Microsoft Transaction Services) acting as middleware and some are legacy socket driven solutions.  All of them have a common set of expected communications that can be monitored.

What if we could separate the millions of packets, and hopefully lion’s share, of expected communication from that communication which is unexpected?

What if we could do it at layer 7?

Using ExtraHop’s Wire Data Analytics Platform INFOSEC teams and application owners are positioned to be able to see non-standard lateral communications that would otherwise go unnoticed by incumbent IPS/Anti-malware/Anti-Virus tools.  The fact is, while we need the existing tools set, today’s complicated breaches tend to hide in the shadows communicating over approved ports and using trusted internal hosts.  ExtraHop shines light on this behavior leaving them exposed and positioning teams to “get their ‘stomp’ on” and stamp out these threats like cockroaches.

How we do it: 
Most INFOSEC practitioners have worked with Wire Data before though their IPS and IDS systems. ExtraHop’s platform is similar in that we work off of a span but instead of looking for specific signatures we observe and rebuild Layer 4-7 flows supporting speeds of up to a sustained 20 Gb per second. We also use a technology called triggers to support specific conditions we want to monitor and alert on (such as anomalies in lateral communications) This is a contrast from most of our perimeter defenses that scale into the megabit/single gigabit range, we are able to work up to the tens of gigabits range. The same innovation that allows us to collect Operational Intelligence and Performance metrics directly off a span port can be used to provide layer 4-7 digital surveillance of critical systems within your or your customer’s infrastructure.

While it is not practical to monitor every single transaction between your critical systems and other components of your enterprise applications, it is now possible, to monitor the unexpected traffic.  For instance, if we have a tiered application that has a web front end, RESTFUL web service tier and a back end database tier.  We can define the expected traffic that we should see between the tiers (what should be the lions share) and ignore that traffic while reporting on the traffic that is NOT expected.

Figure 1: We can write a trigger that ignores expected communications and reports on unexpected communications.

 

 

We can accomplish this task by writing two triggers, the first trigger is a basic Layer 4 detail surveillance trigger where we white-list selected hosts and protocols and only report on communications outside our expected conversations.  The second trigger is a layer 7 trigger that leverages ExtraHop’s layer 7 database fluency of SQL Server communications.  The layer 7 trigger will white-list specific stored procedures that are run from our REST tier that make up our Layer 7 expected communications.  Anything outside of these will be accounted for.

The first trigger makes the following assumptions:
The Web/Public tier comes in from hosts 192.168.1.10/20  and the REST tier is at 192.168.1.30.  The approved ports are 8000, 8080 and 1433.

The first trigger monitors client communications from specific clients and over specific ports.  Unlike NAC (Network Access Control), we are simply monitoring unexpected communications and we are not blocking anything.  In many cases we white list the Active Directory Controllers (if a Windows environment) and we would likely white list the WSUS server for Windows environments.  With the trigger below, you would be alerted of every RDP connection, any direct CIFS access or any exfiltration attempts that utilize ports not previously approved.  This simple trigger could have warned any number of breach victims of the staging that was going on prior to data being exfiltrated.  This trigger took less than one day to write (Don’t be intimidated by the javascript, I knew none when I started and we have people who can help you with it)

Leveraging layer 7 fluency:
Layer 7 surveillance is also a critical part of preventing tomorrow’s sophisticated breaches. In the trigger below, we are watching for expected layer 7 communications. A common occurrence in many high profile breaches is the compromise of a trusted system that is allowed to traverse sensitive networks. In the example above, if the REST tier were to become compromised it is particularly dangerous due to it’s being a trusted host of the Database environment (likely an open ACL). Using the trigger below, we can monitor which stored procedures we should be seeing connecting to the database. Stopthehacker.com states that there is a 156 day laps between the time a computer is compromised and the time it is detected. That is nearly six months that, in the event of a SOAP/REST tier breach, they have to run ad hoc queries against my back end database. For this reason, properly identifying anomaly’s at layer 7 (What SQL Statements are being run?) will be key in preventing/mitigating data loss and just might keep you out of the news.

So using the trigger we have created, if I run the following command (Example: emulate a breached Middleware server)

We are able to increment the counters in the ExtraHop dashboard:

Clicking on the “3” shows us the Offending IP and the Query that they ran:

 

And here we see that we can, in fact, send the data to Splunk or any other SIEMS product:

Digital Evidence:

You can also assign a precision packet capture to a trigger that will create a pcap file that you can use as digital evidence after the fact.

 

Sample Scenario: Middleware Breach
To show how a ExtraHop can detect a middleware breach (see Figure 2) using the two triggers above you would first catch the rogue queries being run with the layer 7 surveillance while ignoring the common stored procedures.  ExtraHop’s Wire Data Analytics will also catch the communications with the Exfiltration/staging server because the communications are outside of those we set in the trigger.    ExtraHop sees this communication and implements the steps in the triggers, from the first trigger it sees the REST tier communicating with an unknown host over an unknown port.  In the second trigger it sees the ad hoc queries mixed in with the normal stored procedures

Figure 2: SOAP/REST Tier Breached:
In figure 2 below you see how an ExtraHop Wire Data analytics appliance can be written to ignore expected traffic and only report on unexpected traffic.

 

Conclusion:

There has been a lot of talk about the need to monitor lateral communications but there has been little practical information on how to do it. Your digital surveillance strategy with Wire Data Analytics will be a living process that in which you periodically update and evaluate your connectivity profile. This is a crucial process often overlooked in INFOSEC.

Using ExtraHop’s platform to build out a surveillance strategy makes the once daunting prospect of monitoring lateral communications a workable process that can provide peace of mind.   We need to accept the fact that we cannot keep all Malware from getting inside; there are two types of systems, breached and about-to-get breached.  I think we need to look at our INFOSEC practice from the WHEN perspective and not the IF perspective.

As infiltration attempts become more sophisticated the ability to spot the building blocks of a data breach will become increasingly valuable.  ExtraHop can provide that extra set of eyes to do the heavy lifting for you while reporting on communications beyond those you expect, reducing the burden of additional auditing.  The reporting/surveillance framework also allows for application owners and shared services teams to get involved in INFOSEC by reviewing the periodic reports and, hopefully in most cases, deleting them.

Wire Data Analytics with the ExtraHop platform can provide your organization with a daily, weekly and monthly digital police blotter giving you the information you need to stay vigilant in the face of this new breed of threat.

Thank you for reading

 

John M. Smith

 

 

 

Leave a Reply