Monthly Archives: April 2017

The Case for Wire Data: GONE in 60….”err”…8 seconds! (Counter-punching DNS jackassery)

A few days ago SANS wrote an article about the importance of tracking DNS Query length stating emphatically “Size Does Matter”. It’s an excellent article and certainly worth a read, you can find it here

The article demonstrates how easy it is to exfiltrate a file using the DNS Channel. They ran a script that encoded the /etc/passwd file into base32 chunks and exfiltrated the file to an external DNS Server. Since the subdomain limit is 63 characters or less they used all 63 characters to append an encoded text string onto the subdomains allowing them to push the data externally using the internal corporate DNS server as the mule in the process.

Just before showing us the Splunk Histograms they did something VERY unique, they showed you the following command:

# tcpdump -vvv -s 0 -i eth0 -l -n port 53 | egrep “A\? .*\.data\.rootshell\.be”

As you are aware, tcpdump is a wire analysis sniffing tool that shows you packets as they are being observed on the wire. If ONLY there were a way to take action against this behavior as it was being observed on the wire. What if I told you there was one? And you probably didn’t even know you could leverage it in your security practice!

Enter Wire Data:
In a way, the SANS article provided a Segway for me to demonstrate the power of wire data. The ExtraHop platform provides full, in flight stream analysis and has the capability to interact with external APIs that may be able to actually STOP DNS exfiltration instead of telling you that it had already happened.

So to do a similar test, I ran something similar attempting to use base64 to exfiltrate a much larger filer called blockedIPs.txt using the following script named I then used an online stopwatch to count the time from when I executed the script to when it finished. The entire text file was exfiltrated in 8.01 seconds. Basically by the time it is rendered in Splunk, the data has already been exfiltrated. That doesn’t mean that the splunk scenario isn’t valuable but leveraging wire data we can do A LOT more than just tell your SOC that they have been breached.

Below you see that the script was able to exfiltrate the entire file in 8.01 seconds.


So in looking at the time it took to exfiltrate the entire BlockedIPs.txt file, 8.01 seconds isn’t really a lot to work with as your SOC does not have a crystal ball.  BUT, in the world of wire data analytics where you deal in microseconds, seconds are hours! Below is a diagram of how we have set up the ExtraHop appliance to alert us when DNS exfiltration takes place. Since I don’t have an active OpenDNS account I am using the Slack API to demonstrate how the ExtraHop platform can integrate with intelligent services. For this test we set the following criteria using ExtraHop’s Application Inspection Triggers.

  • DNS length of greater 63 or greater
  • Not part of B2B partners’, or Internal Namespace
  • Not a common CDN like Akamai or

There will always be SOME white listing that will need to occur to avoid digital collateral damage. If the site is a .ru, .cn or in this case, a .be and I am a hospital in Minneapolis than I doubt that I have an abundance of business with those name spaces and I feel pretty comfortable bit-bucketing them via OpenDNS or another next generation DNS product.


So upon executing the script we begin to see results populating our Slack channel within a few milliseconds. This could just as easily be an API call to your DNS service to “black hole” the suspect domain.

Transaction speed: In the two images below, you can note the performance of the slack transactions. You can see an average round trip time of 41ms with an average processing time of 58ms. I would expect an API call to OpenDNS to be similarly fast basically meaning that you have plenty of time to stop a file from being fully transferred using DNS exfiltration. The point here is, unlike many SIEM based solutions, you are well positioned to counter-punch using ExtraHop and Wire Data Analytics.

Slack Performance Metrics: Taken from the ExtraHop Appliance

The article also made a point to suggest that you also monitor the subdomain counts (they are one of the few to do that, tip of the hat to ya!). Using the ExtraHop platform, we also keep track of the number of subdomains by Client/Domain combo. If you look below, you see the number of lookups for a specific domain from a single IP. Unless it is a NATed address with a few hundred users behind it, it is pretty safe that a large number of the metrics below are NOT human metrics but some sort of program doing the lookups. Even the fastest internet users cannot perform more than a few DNS queries in a 30 second period.

Also noted in the SANS article was the need to pay attention to query lengths. Here we have written a trigger to give you the average query length by root domain. This can, as well as the subdomain count, metrics can be leveraged with an API to actually orchestrate a response via OpenDNS or another product.

There is a big push to add automation and orchestration to most security practices. This is more difficult than it reads/sounds. In the case of DNS Exfiltration, many SIEM based products, while still quite valuable, lack the shutter-speed to successfully observe and react to DNS based exfiltration. However, when you leverage the power of ExtraHop’s Wire Data Analytics and the Open Data Stream (ODS) technology allowing you to orchestrate responses the job becomes almost trivial. As I stated, in our world, seconds are hours and minutes are days. The number of orchestration products hitting the cyber security market are going to make show floors look like a middle-school science fair where practitioners are going to feel like they are looking at a myriad of baking-soda volcanos! Orchestration is only as good as the visibility integrated into it, good visibility starts with wire data, good visibility starts with the ExtraHop platform.

*PS anyone running OpenDNS and familiar with the API, I’d LOVE to try counter punching using the techniques described here!!

The Money is in the Mash-Up: RESTFUL mash-ups to help under-staffed INFOSEC teams

In this post, we will couple ExtraHop’s wire data analytics, Anomali STAXX, a leading threat intelligence solution and Slack, a cloud-based collaboration platform to demonstrate how we can use orchestration and automation in a manner that helps today’s under-(wo)manned security teams meet today’s threats with the level of agility needed!

I was fortunate enough to be selected to speak at RSAC 2017 and it was surely a career highlight for me. As several analysts pointed out post-show, automation and orchestration seemed to be the flavor of the year. Over the last 36 months it has become glaringly obvious that we simply cannot keep bad actors and malicious software off of our networks. I have been preaching the folly of perimeter (only) based security since 2010. The speed with which systems are now compromised and the emergence of the “human vector” through phishing has all but assured us that the horde is behind the wall and needs to be directly engaged. The reliance on logs, SIEM products will give you a forensic view of what is going on but will do little to be effective against today’s threats where a system could be compromised by the time the log is written.

While the idea of automation and orchestration is a great one, there are issues with it and will not be the first time “self-defending networks” have been brought to market. Bruce Schneier makes a very good point in his “Schneier on Security” blog post when he states the following:

You can only automate what you’re certain about, and there is still an enormous amount of uncertainty in cyber security”. He also makes one of the greatest quotes in INFOSEC history when he states “Data does equal information and information does not equal understanding”.

Perhaps the battle here is to get to a place of certainty, I too was once an advocate of “log everything and sort it out later” but the process of sorting through the data become extremely tedious and the amount of work it took to get to “certainty” I believe, gave bad actors time to operate while I wrote SQL queries, batch processes and parsing scripts for my context-starved data sets.  Couple this with the fact that teams are digitally bludgeoned to death with alerts and warnings that the “INFOSEC death sentence” starts to take root as people get desensitized to the alerts.

So where do we find certainty and how do we use it?
While the industry is still developing, there have been great strides in Threat Intelligence. ISACs around the world are working together to build shared intelligence around specific threats and making the information readily available via TAXII, STIXX and CIF. There is even a confidence level associated with each record that we are able to use as a guide to determine if a specific action is needed. The challenge with good threat intelligence is how we make it usable. Currently most threat Intel is leveraged in conjunction with a SIEM or logging product. While I certainly advocate for logs, there are some limitations with them.

  • Not everything logs properly (IoT Systems normally have NO logging at all)
  • You have a data gravity issue (you have to move the data into the cloud to be evaluated or you have to store petabytes of data to evaluate)
  • In some cases, only a small portion of the log is usable (but you pay to index the entire log with most platforms)
  • Their use is largely forensic with many of today’s threats

The case for Wire Data Analytics:
The key difference that I want to point out here is that using Wire Data Analytics with ExtraHop you can perform quite a bit of analysis in flight. ExtraHop “takes” data off the wire and is not dependent on another system to “give” the data to it. The only prerequisite for ExtraHop is an IP address. Examples of how I have made a SIEM more effective using wire data include:

  • Reducing Logging by 5000% by looking at logins by IP and calculating the total THEN sending a syslog message to the SIEM for those IPs with more than 100 logins vs. sending tens of thousands of logs per minute to the SIEM and checking on the back end
  • Checking an EGRESS transaction to against threat intelligence THEN sending the syslog if there is a match
  • In an enterprise with tens of thousands of employees, rather than logging EVERY failed login, aggregate records into five minute increments then send those with more than 5 login failures to the SIEM.

The point here is that you can deliver some context when you leverage wire data analytics with your SIEM workflows. Using SIEM-only, you must achieve context by aggregating the logs and looking at them after they are written. Using ExtraHop with your SIEM, you are able to achieve context (and more importantly, get closer to Mr. Schneier’s certainty) BEFORE sending the data to the SIEM. You can keep all the workflows that are tied to the incumbent SIEM system, you are just getting better, and fewer, logs. Should I disable an account that has 50 login failures in the last five minutes (Locked out or not)…..HELL YES! While I don’t think that automation and orchestration are a panacea, I think there are SOME cases where the certainty level is high enough to orchestrate a response. Also, I believe that automation and orchestration is not just for responding but can be used to make your SOC more effective.

Now that I have, hopefully, established the merits of using Wire Data Analytics, let’s keep in mind orchestration does NOT have to be a specific action or response. Orchestration can also be used to make your team more agile and hopefully, more effective. Most security teams I come across have at least one, two and in some cases, three open positions. The fact is, at a time when threats are becoming more complex, finding people with the needed skills to confront them is harder than ever. The situation has gotten so bad that the other day I typed “Human Capital Crisis” in Google and it auto-filled “in cybersecurity”. The job is getting tougher and there are fewer of us doing it, what I am going to show you in this post will never replace a human being but it might ease some of the heavy lifting that goes into achieving situational awareness.

Anyone who has ever been phished or worked in an organization that is experiencing a phishing/spear phishing campaign has felt exactly as the section title says.  Lets have a look at how we can help our security teams get better data by leveraging the API’s of three unique platforms to warn them when a known phishing site has been accessed.

For those of us who are working too hard to bring context to the deluge of data, my suggestion…get some REST!!! Below I am going to walk you through how I can monitor activity to known phishing sites by doing a mash up of three technologies using the RESTFUL API of all three platforms.

Solution Roster:

  • ExtraHop Discovery/Explorer appliance
    ExtraHop provides wire data analytics and surveillance by working from a mirror of the traffic. Think of it as a CCTV for packets/transactions.
  • Anomali STAXX Virtual Machine
    Anomali STAXX provides me lists of current threat intelligence. Think of this as equipping the CCTV operator with a list of suspicious characters to look for.
  • Slack Collaboration Community
    Slack provides me a community at where my #virtualsoc team operations from anywhere in the world.
  • A python peer (Windows or Linux)
    This is the peer system that accesses the threat intelligence and pulls it off of the STAXX system and uploads the threat intelligence to the ExtraHop appliance.

How it works:
As you can see in the drawing below, the Linux peer uses the REST API to get a list of known phishing sites then executes a Python script to upload the data into the memcache on the ExtraHop appliance equipping it with the threat intelligence it needs. The ExtraHop appliance uses an application inspection trigger that checks every outgoing URI to see if it is a known phishing site. If there is a match, an alert is sent to Slack, Email/SMS in addition to being logged on their own internal dashboards and search appliance.



Click Image


What the final product looks like:
From my Linux box, (I don’t dare go to these sites on my Windows or Mac laptop) I do a “wget” on one of the known phishing sites and within milliseconds (Yes milliseconds, watch the video if you don’t believe me). We get the client IP, Server IP and the site that they went to. From here we can find out who owns that client machine and get them to change their password immediately as well as issue an ACL for the server in case this is a spear phishing campaign and they are targeting specific uses. Also, before you ask, “Yes” we can import the list of known malicious email addresses and monitor key executive recipients in case one of them gets an email from a known malicious address. We can also check HTTP referrers against the phish_url threat intelligence.

In the screenshot below, you see my “wget” command and the result at 11:23:53 and you can see that the Slack warning came in at 11:26.  If you watch the video you will see it takes milliseconds.

I believe that by using slack you can also color code certain messages and program in that awesome “WTF” emoji (if one exists) for specific messages ExtraHop sends. Also, if you are not comfortable with specific information being sent to slack, we can configure the appliance to send you a link to the LOCAL URI that ONLY you and your team can access.

While there is a lot of buzz around Orchestration and Automation I believe the pessimism around it is justified. Security teams have been promised a lot over the last few years and what we have found, especially lately, is that a lot of tried-and-true solutions either lack the shutter-speed or context to be effective. Here we are doing some orchestration and automation but we are doing so in order to give the HUMAN BEING better information. Our security director made a very good point to me the other day when he said the last thing a security team wants is more data. What we have hopefully shown in this post is that if you have open platforms like Anomali, SLACK and ExtraHop, you can craft an automation and orchestration solution that can actively help security teams in a manner that still leverages the nuance and rationalization that only exists in a human being. While there will be solutions that will effectively automatically block certain traffic, issue ACLs, Disable accounts, etc. We can also use automation to do some of the heavy lifting for today’s out(wo)manned security teams. To get where I think the Cyber Security space needs to be, it is going to take more than one product/tool/platform. If you have a solution that is closed and does not support any kind of RESTFUL API or open architecture, unless it fulfills a specific niche, get rid of it. If you are a vendor and you are selling a solution that is closed, do so at your own peril as I believe closed systems are destined to go the way of the dinosaur. By leveraging wire data with existing workflows, you can drastically reduce your TTWTF (time to WTF!??) and be better positioned to trade punches with tomorrow’s threats.

Thanks so much for reading, please watch the video.

John M. Smith