Category Archives: Wire Data

The Case for Wire Data: Security

During the 2nd week of February I had the honor to deliver two speaking sessions at the RSA Conference in San Francisco. One of them was on Ad Hoc threat intelligence and the 2nd was a Birds of a Feather round-table session called “Beyond Logs: Wire Data Analytics”. While it was a great conference, I found that you get some strange looks at a security conference when you are walking around with a badge that says “John Smith”. In both sessions, a key narrative was the effectiveness of wire data analytics and its ability to position security teams with the needed agility to combat today’s threats. In this post I would like to make the case for wire data analytics and demonstrate the effectiveness of using wire data as another tool along with your IDS/IPS and Log consolidation.

Wire Data Analytics:
Most security professionals are familiar with wire data already. Having used Intrusion protection and detection software for nearly 20 years now, concepts such as port mirroring and span aggregation are already native to them. INFOSEC professionals are some of the original wire data analytics professionals. We differ from IDS/IPS platforms in that we are not looking specifically at signatures rather we are rebuilding layer 2-7 flows. We have several areas where we can help INFOSEC teams by providing visibility into the SANS first two critical security controls, augmenting logs and increasing visibility as well as providing a catalyst for ongoing orchestration efforts.

SANS Top 2 Security Critical Controls:
From Wikipedia we have the following list making up the SANS top 20 Cyber Security Controls (Click image if you, like me, are middle aged and can’t see it)

In our conversations with practitioners we commonly hear that “if we could JUST get an inventory of what systems are on the network”. As virtualization and automation has matured over the years, the ability to mass-provision systems has made security teams’ job much harder as there can be as much as a 15% difference in the number of nodes on a single 24 bit CIDR block from one day to the next, hell from one hour to the next. Getting a consistent inventory with current technologies generally involves responding to an SNMP sweep, Ping response, WMI Mining or NMAP scan. As we have seen with IoT devices, many of them don’t have MIBs, WMI libraries and in most (all) cases logs. Most malicious doers will prefer to do their work in the dark, if detected, they will try to use approved or common ports to remain unseen.

“All snakes who wish to remain in Ireland, please raise your right hand….” Saint Patrick

The likelihood that a compromised system is going to respond to an SNMP walk, Ping, WMI connection or volunteer what they are doing may be about as likely as a snake raising their right hand.

How ExtraHop works with the top 2 SANS controls:
Most systems try to engineer down to this level of detail, technologies such as SNMP, Netflow, logs and the like to do a pretty good job of getting 80-90 percent of the hosts but there are still blind spots. When you are a passive wire data analytics solution, you aren’t dependent on someone to “give” data to you, we “take” the data off the wire. This means if someone shuts off logging, deletes /var/log/* they cannot hide. A senior security architect once told me, “if it has an IP address it can be compromised”. To that we at ExtraHop would answer “if it has an IP Address, it can’t hide from us”. I cannot tell you what the next big breach or vulnerability will be, but what I CAN say with certainty (and trust me, I coined the phrase “certainty is the enemy of reason”, I am NEVER certain) is that it will involve one host talking to another host it isn’t supposed to. With wire data, if you have an IP address and you talk to another node who ALSO has an IP address. Provided we have the proper surveillance in place….YOUR BUSTED!

ExtraHop creates an inventory as it “observes” packets and layer 7 transactions. This positions the security team to account for who is talking on their network regardless of the availability of agents, Netflow, MIBs or WMI libraries. To add to this, ExtraHop applies a layer of intelligence around it. Below, you see a collection of hosts and locations as well as a transaction count. What we have done is import a customer’s CIDR block mapping csv that will then allow us to geocode both RFC1918 addresses as well as external addresses so that you have a friendly name for the CIDR block. This is a process of reconciling which networks belong to which groups and/or functions. As you can see, we have a few IP Addresses, the workflow here is to identify every IP address and classify it’s CIDR block until you can fully account for who lives where. This takes a process of getting an accurate inventory from, what can be, a 3 month or longer task into a few hours. Once you have reconciled which hosts belong to which functions, you have taken the first step in building your Security Controls foundation securing the first control. The lack of this control is a significant reason why many security practices topple over. An accurate inventory is the foundation, to quote the defensivesecurity.org podcast “ya gotta know what you have first”.

Click Image:

SANS Control 2: Inventory of Authorized and Unauthorized Software:
While wire data cannot directly address this control, I tend to interpret (maybe incorrectly) this as being networked software. While something running on just one host could do significant damage to that one host. Most of us worry more about data exfiltration. This means that the malicious software HAS to do something on the Network. Here we look at both Layer 4 and Layer 7 to provide an inventory of what is actually being run on the systems that you have finally gathered an inventory for.

In the graphic below, you see one of our classified CIDR blocks. We have used the designation “Destination” (server) to get an accurate inventory of what ports and protocols are being served up by the systems on this CIDR block. (Or End Point Group if you are a Cisco ACI person). Given that I have filtered out for transactions being served up by our “Web Farm” the expected ports and protocols would be HTTP:8080, SSL:443, HTTP, etc. Sadly, what I am seeing below is someone SSHing into my system and that is NOT what I expected. While getting to this view too me only two clicks we can actually do better. We can trigger an alert letting the SOC or CSIRT know that there has been a violation. Later in this post, we will talk about how we could actually counter-punch this type of behavior using our API.

As far as SANS 2nd Control. If I look on a web server and I see that it is an FTP Client to a system in Belarus, I am generally left to conclude that the FTP is likely unauthorized. What ExtraHop gives you, in addition to an accurate inventory, is an accounting for what ports and protocols are in use by both the clients and servers using those segments. While this is not a literal solution for SANS 2nd control it does have significant value in that INFOSEC practitioners can see what is traversing their network and are positioned to respond with alerts or orchestrate remediation.

Layer 7 Monitoring:
In the video below, titled “Insider Hating”, you see our Layer 7 auditing capability. In this scenario we have set up an application inspection trigger to look for any queries of our EHR database. The fictitious scenario here is that we want to audit who is querying our EHR database to ensure that it is not improperly used or that someone does not steal PHI from it. When an attacker has stolen credentials or you have an insider, you now have an attack that is going to use approved credentials and approved ports/protocols. This is what keeps CIOs, CISOs and practitioners up at night. We can help and we HAVE helped on numerous occasions. Here we are setting up a L7 inspection trigger to look for any ad hoc like behavior. In doing so, we can position, not JUST the security team to engage in surveillance, but the system owners. This is an ABSOLUTE IMPARATIVE if we want to be able to stop insiders or folks with stolen credentials. We need to do away with the idea that security teams have a crystal ball. When someone runs a “Select * from ERH” from a laptop in the mail room, we can tell you that it came from the mail room and not the web server. We can also alert the DBA of this and get system owners to take some responsibility for their own security. This same query, to many security teams, will look like an approved set of creds using approved ports. This same information being viewed by the DBA or System owner may cause them to fall out of their chair and run screaming to the Security teams’ office. The of vigilance by system owners, in my opinion, is the single greatest reason breaches are worse than ever before in spite of the fact that we spend more money than ever.

 

Augmenting Logs:
I love logs, I love Splunk, LogRhythm and of course my old friend Kiwi!! But today’s threats and breaches happen so fast that using just logs positions you to operate in a largely forensic fashion. In many cases, by the time the log is written and noticed by the SOC the breach has already happened. Below you see a graphic from the Verizon DBIR that states that 93% of all compromises happen within minutes, 11% within seconds. Using just logs and batch processing to find these threats is great for rooting out patterns and malicious behavior but, as I stated previously, largely forensic. As a Wire Data Analytics platform we work/live in a world of microseconds and thus for us, seconds are hours and minutes are days. Current SIEM products, when not augmented with wire data analytics, simply don’t have the shutter speed to detect and notify or orchestrate a timely response.

 

Example:
I saw an amazing black-hat demo on how OpenDNS was using a hadoop cluster to root out C2 controllers and FastFlux domains. The job involved a periodic batch job using pic to extract domains with a TTL of 150. Through this process they were able to consistently root out “FastFluxy” domains to get a new block list.

We have had some success here collecting the data directly off the wire. I will explain how it works: (we are using a DNS Tunneling PCAP but C2 and Exfiltration will have similar behavior).

  • First we whitelist common CDNs and common domains such as Microsoft, Akamai, my internal intranet namespace, etc.
  • We collect root domains and we start adding the number of subdomains that we observe.
    • In the example below, we see pirate.sea and we start to increment each time we observe a subdomain
  • If a root domain has a count of over 50 subdomains within a 30 second period, we account for it. (thus the dashboard below)

The idea behind this inspection trigger is that if the root domain is NOT a CDN, not my internal namespace and not a common domain like Google or Microsoft, WHY THE HELL DOES THE CLIENT HAVE 24K lookups? Using logs, this is done via a batch process vs. using wire data, we uncover suspicious behavior in 30 seconds. Does that mean you don’t need logs or the ingenius work done by OpenDNS isn’t useful? Hell no, this is simply augmenting the log based approach to give you more agile tool to engage directly with an issue as it is happening. I am certain that even the folks at OpenDNS would find value in being able to get an initial screening within 30 seconds. In my experience, with good white listing, the number of positives is not overly high. Ultimately, if a single client makes 24500 DNS lookups for a domain that you don’t normally do business with, it’s worth investigating. We routinely see Malware, Adware as well as 3rd party, unapproved, apps that think they are clever by using DNS to phone home (yes YOU Dropbox) using this method.

Click Image:

SIEM products are a lynch pin for most security teams. For this reason, we support sending data to SIEM platforms such as LogRhythm and Splunk but we also provide a hand-to-hand combat tool for those SecOps (DevOps) focused teams who want to engage threats directly. In the hand-to-hand world of today’s threats, no platform gives you a sharper knife or a bigger stick than Wire Data Analytics with ExtraHop.

Automation and Orchestration (Digital counter-punching):
In an article in September of 2014 GCN asked “is automation security’s only hope?” With the emergence of the “human vector” what we have learned over the last 18 months is that you can spend ten million dollars in security software, tools and training only to have Fred in payroll open a malicious attachment and undo all of it within a few seconds. As stated earlier in this post, 11% of compromises happen within seconds. All, I hope, is not lost however, there have been significant improvements in orchestration and automation. At RSAC 2016 Phantom Cyber debuted their ability to counter-punch and won first prize in the innovation sandbox. You can go to my youtube channel and see several instances of integration with OctoBlu where we are using OctoBlu to query threat intelligence and warn us of malicious traffic. But we can go a lot further with this. I don’t think we have to settle for post-mortem detection (which is still quite valuable to restrict subsequent breach attempts) with logs and batched surveillance. Automation and orchestration will only be as effective as the visibility you can provide.

Enter Wire Data:
Using wire data analytics, keep in mind that ours is a world of microseconds, we have the shutter speed to observe and act on today’s threats and thread our observed intelligence into orchestration and automation platforms such as Phantom Cyber and/or OctoBlu and do more than just warn. ExtraHop Open Data Stream has the ability to securely issue an HTTP.post command whereby we send a JSON object with the parameters of who to block positioning INFOSEC teams to potentially stop malicious behavior BEFORE the compromise. Phantom Cyber supports REST based orchestration as does Citrix OctoBlu, most of your newer firewalls have API’s that can be accessed as does Cisco ACI. The important thing here to remember is that these orchestration tools and next generation hardware API’s need to partner with a platform that can not only observe the malicious behavior but thread the intel into these API’s positioning security teams for tomorrows’ threats.

My dream integrations include:

  • Upon observing FastFluxy behavior, sending OpenDNS an API call that resolves the offending domain to 127.0.0.1 or a warning page
  • Putting a mac address in an ACI “Penalty box” (quarantine endpoint group) when we see them accessing a system they are not supposed to
  • Sending an API call to the Cisco ASA API to create an ACL blocking a host that just nmapped your DMZ

As orchestration and automation continues to take shape within your own practices, please consider what kind of visibility available to them. How fast you can observe actionable intelligence will have a direct effect on how effective your orchestration and automation endeavors are. Wire Data analytics with ExtraHop has no peer when it comes to the ability to set conditions that make a transaction actionable and act on it. Orchestration and automation vendors will not find a better partner that will make their products better than ExtraHop.

Conclusion:
The threat landscape is drastically changing and the tools in the industry and rapidly trying to adapt. An orchestration tool is not effective without a good surveillance tool, a Wire Data analytics platform like ExtraHop is made better when coupled with an orchestration tool that can effectively receive REST based Intel. The solution to tomorrows’ threats will not involve a single vendor and the ability to integrate platforms using APIs will become key to implementing tomorrows’ solutions. The ExtraHop platform is the perfect visibility tool to add to your existing INFSEC portfolio. Whether you are looking to map out a Cisco ACI implementation or you want to thread wire data analytics into your Cisco Tetration investment, getting real-time analytics and visibility will make all of your security investments better. Wire Data Analytics will become a key part of any security team’s arsenal in the future and the days of closed platforms that cannot integrate with other platforms are coming to an end.

There is no security puzzle where ExtraHop’s Wire Data Analytics does not have a piece that fits.

If you’d like to see more, check out my YouTube channel:
https://www.youtube.com/playlist?list=PLPadIDS3iteYhQuFhWy2xZemdIFzMNtpr

Thanks for reading

John Smith

 

 

 

 

 

 

 

 

 

Advanced Persistent Surveillance: Threat Intelligence and Wire Data equals Real-time Wire Intelligence

Please watch the Video!!

As the new discipline of Threat Intelligence takes shape, Cyber Security teams will need to take a hard look at their existing tool sets if they want to take advantage of the dynamic, ever changing threat intelligence feeds providing them with information on which hosts are malicious and whether or not any of their corporate nodes have engaged in any sort of communications with any of the malicious hosts, DNS names or hashes that you are collecting from your CTI (Cyber Threat Intelligence) feeds. Currently the most common way that I see this accomplished is through the use of logs. Innovative products like Alienvault and Splunk have the ability to check the myriad of log files that they collect and cross reference them with CTI fees and check to see there have been any IP based correspondence with any known malicious actors called out by such feeds.

Today I want to talk about a different, and in my opinion, better way of integrating with Cyber Threat Intelligence using Wire Data and the ExtraHop Platform featuring the Discover and Explorer Appliances respectively.

How does it work? Well let’s first start with our ingredients.

  1. A threat analytics feed (open source, subscription, Bro or CIF created text file)
  2. A peer Unix-based system to execute a python script (that I will provide)
  3. An ExtraHop Discover Appliance
  4. An ExtraHop Explorer Appliance

Definitions:

  • ExtraHop Discover Appliance:
    An appliance that can passively (no agents) read data at speeds from 1GB to 40GB. It can also scale horizontally to handle large environments.
  • ExtraHop Explore Appliance:
    ExtraHop’s Elastic appliance that allows for grouping and string searching INTEL gathered off the wire.
  • Session Table: ExtraHop’s memcache that allows for instant lookup of known malicious hosts.

The solutions works by using the Unix peer to execute a python script that will collect the threat intelligence data. It then uploads the malicious hosts into the Discover Appliance’s Session Table (up to 32K records). The Discover appliance then waits to observe a session that connects with one of these known malicious sites. If it sees a session with a known site from the TI feed activities include, but are not limited to the following:

  • Updates a Threat Intelligence dashboard
  • Triggers an alert that warns the appropriate Incident Response team(s) about the connection to the malicious host
  • Writes a record to the ExtraHop Explorer Device
  • Triggers a Precision PCAP capturing the entire session/transaction to a PCAP file to be leveraged as digital evidence in the event that “Chet” the security guard needs to issue someone a cardboard box! (not sure if any of you are old enough to remember “Chet” from weird science)

Please Click Image:

ThreatIntel

Below you see the ExtraHop Threat Intelligence Monitoring Dashboard (last 30 minutes) showing the Client/Server and Protocol as well as the Alert and a running count of violations: (this is all 100% customizable)

Please Click Image:

On the Explorer Appliance, we see the custom data format for Malicious Host Access and we can note the regularity of the offense
Please Click Image:

And finally we have the Precision Packet Capture showing a PCAP file for forensics, digital evidence and if needed, punk busting.
Please Click Image:

Conclusion:
The entire process that I have outlined above took less than one minute to complete every single task (Dashboard, Alert, EXA, PCAP). According to Security Week, the average time to detect a breach has “Improved” to 146 days in their 2015 report. Cyber Threat Intelligence has a chance to drastically reduce the amount of time it takes to detect a breach but it needs a way to interact with existing data.  ExtraHop positions your Threat Intelligence investment to interact directly with the network, and in real time.  Many incumbent security tools are not built to accommodate solutions like CTI feeds via API or do not have an open architecture to leverage Threat Intelligence, much less use memcache to do quick lookups. The solution outlined above using ExtraHop with a Threat Intelligence feed positions INFOSEC teams to be able to perform Advanced Persistent Surveillance without the cost of expensive log indexing SIEM solutions. Since the data is analyzed in flight and in real time, you have a chance to greatly reduce your time to detection of a breach, maybe even start the Incident Response process within a few minutes!

What you have read here is not a unicorn, this exists today, you just need to open your mind to leveraging the network as a data source (in my opinion the richest) that can work in conjunction with your log consolidation strategy and maximize your investment in Cyber Threat Intelligence.

Incidentally, the “Malicious Host” you see in my logs is actually wiredata.net.  I did NOT want to browse any of the hosts on the blacklist so I manually added my host to the blacklist the accessed it.  Rest assured, WireData.net is not on any blacklists that I am aware of!

Thanks for reading!

John M. Smith

Covering The “Behind The Perimeter” Blind-Spot

Well, I cannot tell you what the next big breach will be, I CAN tell you that it will involve one critical system communicating to another system that it was/is not supposed to.  Whether that is ex filtration via secure ssh (SCP) to a server in Belarus or mounting a hidden drive share using a default service account that was never changed, behind the perimeter represents a rather large blind spot for many security endeavors.  In the video below, you are seeing a very quick and simple method for monitoring peer-to-peer communications using wire data with the ExtraHop Platform.  This is a somewhat painful process with logs due to the fact that logging build-up and tear-downs can impact the hardware being asked to do the logging and if you are licensed to pay by how much data you index, it can be expensive.  Grabbing Flow records directly off the wire positions your security practice to have this information readily available in real-time with no impact on the infrastructure as no system/switch is asked to run in debug mode.  Transactions are taken directly off the wire by the ExtraHop Discover Appliance and written directly to the Elastic ExtraHop Explorer Appliance to provide a large and holistic view of every transaction that occurs within your critical infrastructure.

This positions system-owners with the ability to (within minutes) to audit critical systems and account for transactions that are suspect.  In the video below, you will see how we can audit flow records by white listing expected communications and slowly leaving malicious communications no where to hide as you slowly reduce the surface area that you are trying to patrol.


I will follow this up with another post on how we can monitor Layer 7 metrics such as SQL Queries against critical databases.

Thanks for reading, please watch the video!

John Smith

My Evolution to Wire Data

A little over a week ago, Gartner Analysts Vivek Bhalla and Will Cappelli identified wire data as “the most critical source of availability and performance”, you can read the Gartner report here.  One of the reasons I started this blog in 2013 was because of the profound value I had discovered when using wire data as the source of truth in my environment. Today, I’d like to expand on a 2013 article “when uptime is not enough, the case for wire data” and discuss why I favor wire data as the best source of information that can even extend beyond availability and performance. I will discuss the value of evaluating transactions in-flight and some of the challenges I have found with other sources of Intelligence.

So why Wire Data?
Prior to 2013 and my first exposure to wire data I was a logging fiend. While at one of my previous employers, during the yearly holiday lunch we all had to put a headshot photo on a wall and our peers got to fill in the caption. My peers filled in “it’s not our fault, and I have the logs to prove it” under my picture. Not having the money for Splunk initially we had set up a Netscaler load balanced KIWI Syslog Farm that wrote to a SQL Database (we called it “SKUNK” for SQL-KIWI-Splunk-works). Later, at another employer I was able to purchase Splunk and used it extensively to collect logs and obtain operational awareness by leveraging the ad-hoc capabilities of the Splunk platform. Suffice it to say, I loved logs! While I am still an advocate of log data I started to find that anything that writes information to disk then evaluates the Intel AFTER it is written to disk becomes liable to the disk for performance as the database/PCAP/data set grows. We have seen this with some of the PCAP driven NPM vendors and logs can be liable to the same limitations. I have colleagues that have mid-high six figure investments in Indexing and Clustering technologies just to accommodate the size of their data store where they are keeping the logs. In the case of Splunk, they have done some innovating around hot, warm and cold disk storage which has assuaged some of the cost/performance burden but eventually, you end up with rack(s) of servers supporting your log infrastructure and if the data is not properly indexed or map-reduced, performance will suffer.

In Flight vs. On Disk
ExtraHop’s Wire Data analytics platform evaluates the transaction in transit. This provides some significant differences between Machine Data analytics (log data) and Wire Data Analytics. ExtraHop is NOT dependent on any other system for information. When given a span or tap, data is “taken” directly off the wire. Where a syslog solution depends on the end point to forward or syslog data off to them, ExtraHop observes the transaction in flight and evaluates it (uri, user-agent, process time, Database Query or Stored Procedure client ip, etc) at the time it happens. This ensures that transactions are quickly and accurately evaluated regardless of how much data is being written to disk. If a system is I/O bound, it could very easily impact its ability to send logs to a syslog server. However, since the ExtraHop appliance is basically providing a packet-level CCTV, transactions are observed and evaluated vs. waiting for them to be written to disk, then evaluated. If endpoints are I/O bound, the ExtraHop platform will not only continue to give you information/Intel about your degrading application performance, it will also tell you that your systems are I/O bound by reporting on zero windows observed. There are no agents or forwarders to install with ExtraHop as it is operating passively on the wire via a Tap or Span. As a security note, most hackers worth their weight in salt will shut off logging if they compromise a system. With ExtraHop they will not be able to hide as ExtraHop is not dependent on any syslog daemon or forwarder to get information. ExtraHop “TAKES” intelligence off the wire, whereas logging systems, while I still LOVE logs, depend on other systems to “GIVE” data.

Beyond Performance
Equally important to the Break-Fix/Performance value derived by Wire Data, there is also the Business Intelligence/Operational Analytics portion. Few people realize the level of value that can be derived from parsing payloads. In the retail space, I have seen Wire Data solutions that deliver real-time business analytics that had previously been a very long and costly OLAP/Data Warehousing job, now replace by parsing a few fields within the XML and delivering the data in real-time to the analytics teams as well then having the, now parsed and normalized, data forwarded to a back end database. With the Kerberos support introduced in 5.x, you can leverage the ExtraHop Memcached Session table to map users to IP Addresses then perform a lookup against it for every single transaction allowing you map ever single users transaction so that if they call and complain about performance, you can see every transaction that they have run or, if you suspect a user of malfeasance, you can audit their every move on your network and lastly, if you want to ensure that your CEO/CIO/CFO have a good experience or ensure that their credentials have not been phished, you can audit their transactions by user ID or Client IP Address. I tend to think of the ExtraHop appliance as a “prospectors pan” that they use to find gold, ExtraHop allows you to sift through the gravel and sand that can create a lot of noise in today’s 10GB, 40GB and soon 100GB networks and pull out those operational gold nuggets of Intel right off the wire. There’s a gold mine on every wire, ExtraHop is there to help you find it.

Conclusion
You will find numerous posts on this site showing integration between ExtraHop and Splunk, please understand I am NOT down on Splunk or any other logging technology, I am simply explaining the differences between using Machine data and Wire Data and why I use wire data as my first and best source of intelligence. The truth is, the ExtraHop platform is itself, an exceptionally good forwarder of data (you can read about it in other articles on this site) and can be used in conjunction with your machine data as it has the ability to “take” the Intelligence directly off the wire and “give” it to your back-end solution, if it’s Splunk/ELK, KAFKA, MongoDB or their own Big-Data appliance (ExtraHop EXA) . For my money, there isn’t a better forwarder than an ExtraHop appliance who is going to send you data regardless of the state of your servers/systems while at the same time, providing real-time visibility into transactions at the time they occur. I am very happy to see Gartner’s analysts starting to acknowledge the value that can be derived from Wire Data and recognize the hard work by a number amazing pioneers in Seattle.

What’s on your wire?

Thanks for reading, again if you want to read the Gartner report click here to download a copy.

John M. Smith