The threat hunting discipline continues to evolve in the security industry as organizations seek to become more proactive at finding threats to their infrastructure and engaging in what his fast becoming a “hand-to-hand” combat situation in IT security. At the time of this article there where over 7100 job postings on linked in with the terms “Threat Hunting” or “Threat Intelligence”. The deluge of data that we have undertaken in the last ten years has resulted in significant noise and difficulty finding context within the myriad of disparate data sources that exist in an enterprise. In this post, I want to discuss how we can “cut out the middle-man” in our “quest for context” gaining much needed agility, speed and making our investments in mash-up technologies such as SIEMs, big data lakes and innovations like Sqrrl more efficient and less work.
The case for wire data: I tend to over use the title above but I think there is a significant case for leveraging wire data to gain context. In the case of this specific hunt, I am going to search specifically for non-standard user agents that I note on my network. UA hunting is relatively common and while most hackers worth their salt will change the user agent name to hide there is still a large number of solutions that do not, specifically IoT devices, of which millions will be purchased in the next few weeks that could be arbitrarily connected to your networks. The case for using wire data here is that, YES, you can log user agent in your web logs and parse them out, write them to SIEM or database then query them to find those user agents that you consider to be actionable. This is the “Middle Man” I am trying to eliminate. You may need to collect several terabytes of logs that need to be indexed (not always free) and stored (not always fast). Leveraging the ExtraHop wire data analytics platform, we are parsing non-standard User-Agents directly off the wire (no “middle-man”) and bring context to the surface within milliseconds. This data can now be send to your back end SIEM, data lake or Sqrrl instance delivering a much higher (term I like to use) “intelligence yield” as, in addition to populating dashboards, integrating with orchestration API’s and creating alerts and emails, you line your threat intelligence coffers with BETTER DATA! Better SIEM, better data-lake, better back end systems.
Below is a dashboard created specifically for the UA Hunt inspired by the threat hunting project that includes the following metrics:
Total number of unique user-agents per host (If your server has 7 unique user agents…it might have some malware or unauthorized software on it)
Unique IPs by non-standard User Agent.
Python and PowerShell user-agents (POSH is rapidly becoming the weapon of choice for a number of bad actors, if they use it to phone home….BUSTED!)
Geo-coded INGRESS and EGRESS (Example: if you don’t do business with Belarus, maybe you should look into the POSH user agent connected to it…)
Drilling Down: You can also drill down into the specific conversations that were observed to get an idea of the client/server involved in the conversation as well as the request/response bytes (how big a file was sent) and URI.
Clicking the bull’s eye on the left will take you to the packets that can then be downloaded and analyzed as well.
Conclusion: The “Data problem” is a big one, solving it requires re-thinking how/where we do analytics. Back end SIEMs and data lakes are in desperate need of better data while not everything will send Syslog messages or Netflow data the common thread in all of the devices we are trying to secure is wire data. If it has an IP address, we can monitor it and log it. Threading wire data analytics into your security strategy will add significant agility, shutter speed and a higher intelligence yield making your entire security practice more effective. This is just one example of threat hunting with Wire Data and ExtraHop. Future posts will include the following hunts:
Detecting lateral movement via Explicit Credentials
Command and Control Detection
Threat hunting with wire data brings an entirely new data source to threat hunters and practitioners. Leveraging ExtraHop’s open solution that includes integration with REST API’s will make solving the “data problem” much easier and save you the time, cost and effort of parsing through terabytes and even petabytes of data to gain context.
In this post, we will couple ExtraHop’s wire data analytics, Anomali STAXX, a leading threat intelligence solution and Slack, a cloud-based collaboration platform to demonstrate how we can use orchestration and automation in a manner that helps today’s under-(wo)manned security teams meet today’s threats with the level of agility needed!
I was fortunate enough to be selected to speak at RSAC 2017 and it was surely a career highlight for me. As several analysts pointed out post-show, automation and orchestration seemed to be the flavor of the year. Over the last 36 months it has become glaringly obvious that we simply cannot keep bad actors and malicious software off of our networks. I have been preaching the folly of perimeter (only) based security since 2010. The speed with which systems are now compromised and the emergence of the “human vector” through phishing has all but assured us that the horde is behind the wall and needs to be directly engaged. The reliance on logs, SIEM products will give you a forensic view of what is going on but will do little to be effective against today’s threats where a system could be compromised by the time the log is written.
While the idea of automation and orchestration is a great one, there are issues with it and will not be the first time “self-defending networks” have been brought to market. Bruce Schneier makes a very good point in his “Schneier on Security” blog post when he states the following:
“You can only automate what you’re certain about, and there is still an enormous amount of uncertainty in cyber security”. He also makes one of the greatest quotes in INFOSEC history when he states “Data does equal information and information does not equal understanding”.
Perhaps the battle here is to get to a place of certainty, I too was once an advocate of “log everything and sort it out later” but the process of sorting through the data become extremely tedious and the amount of work it took to get to “certainty” I believe, gave bad actors time to operate while I wrote SQL queries, batch processes and parsing scripts for my context-starved data sets. Couple this with the fact that teams are digitally bludgeoned to death with alerts and warnings that the “INFOSEC death sentence” starts to take root as people get desensitized to the alerts.
So where do we find certainty and how do we use it? While the industry is still developing, there have been great strides in Threat Intelligence. ISACs around the world are working together to build shared intelligence around specific threats and making the information readily available via TAXII, STIXX and CIF. There is even a confidence level associated with each record that we are able to use as a guide to determine if a specific action is needed. The challenge with good threat intelligence is how we make it usable. Currently most threat Intel is leveraged in conjunction with a SIEM or logging product. While I certainly advocate for logs, there are some limitations with them.
Not everything logs properly (IoT Systems normally have NO logging at all)
You have a data gravity issue (you have to move the data into the cloud to be evaluated or you have to store petabytes of data to evaluate)
In some cases, only a small portion of the log is usable (but you pay to index the entire log with most platforms)
Their use is largely forensic with many of today’s threats
The case for Wire Data Analytics: The key difference that I want to point out here is that using Wire Data Analytics with ExtraHop you can perform quite a bit of analysis in flight. ExtraHop “takes” data off the wire and is not dependent on another system to “give” the data to it. The only prerequisite for ExtraHop is an IP address. Examples of how I have made a SIEM more effective using wire data include:
Reducing Logging by 5000% by looking at logins by IP and calculating the total THEN sending a syslog message to the SIEM for those IPs with more than 100 logins vs. sending tens of thousands of logs per minute to the SIEM and checking on the back end
Checking an EGRESS transaction to against threat intelligence THEN sending the syslog if there is a match
In an enterprise with tens of thousands of employees, rather than logging EVERY failed login, aggregate records into five minute increments then send those with more than 5 login failures to the SIEM.
The point here is that you can deliver some context when you leverage wire data analytics with your SIEM workflows. Using SIEM-only, you must achieve context by aggregating the logs and looking at them after they are written. Using ExtraHop with your SIEM, you are able to achieve context (and more importantly, get closer to Mr. Schneier’s certainty) BEFORE sending the data to the SIEM. You can keep all the workflows that are tied to the incumbent SIEM system, you are just getting better, and fewer, logs. Should I disable an account that has 50 login failures in the last five minutes (Locked out or not)…..HELL YES! While I don’t think that automation and orchestration are a panacea, I think there are SOME cases where the certainty level is high enough to orchestrate a response. Also, I believe that automation and orchestration is not just for responding but can be used to make your SOC more effective.
Now that I have, hopefully, established the merits of using Wire Data Analytics, let’s keep in mind orchestration does NOT have to be a specific action or response. Orchestration can also be used to make your team more agile and hopefully, more effective. Most security teams I come across have at least one, two and in some cases, three open positions. The fact is, at a time when threats are becoming more complex, finding people with the needed skills to confront them is harder than ever. The situation has gotten so bad that the other day I typed “Human Capital Crisis” in Google and it auto-filled “in cybersecurity”. The job is getting tougher and there are fewer of us doing it, what I am going to show you in this post will never replace a human being but it might ease some of the heavy lifting that goes into achieving situational awareness.
PHISHING: “PHUCK YOU, YOU PHISHING PHUCKS!!!” Anyone who has ever been phished or worked in an organization that is experiencing a phishing/spear phishing campaign has felt exactly as the section title says. Lets have a look at how we can help our security teams get better data by leveraging the API’s of three unique platforms to warn them when a known phishing site has been accessed.
For those of us who are working too hard to bring context to the deluge of data, my suggestion…get some REST!!! Below I am going to walk you through how I can monitor activity to known phishing sites by doing a mash up of three technologies using the RESTFUL API of all three platforms.
ExtraHop Discovery/Explorer appliance
ExtraHop provides wire data analytics and surveillance by working from a mirror of the traffic. Think of it as a CCTV for packets/transactions.
Anomali STAXX Virtual Machine
Anomali STAXX provides me lists of current threat intelligence. Think of this as equipping the CCTV operator with a list of suspicious characters to look for.
Slack Collaboration Community
Slack provides me a community at packetjockey.slack.com where my #virtualsoc team operations from anywhere in the world.
A python peer (Windows or Linux) This is the peer system that accesses the threat intelligence and pulls it off of the STAXX system and uploads the threat intelligence to the ExtraHop appliance.
How it works: As you can see in the drawing below, the Linux peer uses the REST API to get a list of known phishing sites then executes a Python script to upload the data into the memcache on the ExtraHop appliance equipping it with the threat intelligence it needs. The ExtraHop appliance uses an application inspection trigger that checks every outgoing URI to see if it is a known phishing site. If there is a match, an alert is sent to Slack, Email/SMS in addition to being logged on their own internal dashboards and search appliance.
What the final product looks like: From my Linux box, (I don’t dare go to these sites on my Windows or Mac laptop) I do a “wget” on one of the known phishing sites and within milliseconds (Yes milliseconds, watch the video if you don’t believe me). We get the client IP, Server IP and the site that they went to. From here we can find out who owns that client machine and get them to change their password immediately as well as issue an ACL for the server in case this is a spear phishing campaign and they are targeting specific uses. Also, before you ask, “Yes” we can import the list of known malicious email addresses and monitor key executive recipients in case one of them gets an email from a known malicious address. We can also check HTTP referrers against the phish_url threat intelligence.
In the screenshot below, you see my “wget” command and the result at 11:23:53 and you can see that the Slack warning came in at 11:26. If you watch the video you will see it takes milliseconds.
I believe that by using slack you can also color code certain messages and program in that awesome “WTF” emoji (if one exists) for specific messages ExtraHop sends. Also, if you are not comfortable with specific information being sent to slack, we can configure the appliance to send you a link to the LOCAL URI that ONLY you and your team can access.
Conclusion: While there is a lot of buzz around Orchestration and Automation I believe the pessimism around it is justified. Security teams have been promised a lot over the last few years and what we have found, especially lately, is that a lot of tried-and-true solutions either lack the shutter-speed or context to be effective. Here we are doing some orchestration and automation but we are doing so in order to give the HUMAN BEING better information. Our security director made a very good point to me the other day when he said the last thing a security team wants is more data. What we have hopefully shown in this post is that if you have open platforms like Anomali, SLACK and ExtraHop, you can craft an automation and orchestration solution that can actively help security teams in a manner that still leverages the nuance and rationalization that only exists in a human being. While there will be solutions that will effectively automatically block certain traffic, issue ACLs, Disable accounts, etc. We can also use automation to do some of the heavy lifting for today’s out(wo)manned security teams. To get where I think the Cyber Security space needs to be, it is going to take more than one product/tool/platform. If you have a solution that is closed and does not support any kind of RESTFUL API or open architecture, unless it fulfills a specific niche, get rid of it. If you are a vendor and you are selling a solution that is closed, do so at your own peril as I believe closed systems are destined to go the way of the dinosaur. By leveraging wire data with existing workflows, you can drastically reduce your TTWTF (time to WTF!??) and be better positioned to trade punches with tomorrow’s threats.
Thanks so much for reading, please watch the video.
Not exactly new news here but in case you have been in a coma the last two weeks. Google managed to engineer a successful SHA1 attack and intends to release the source code somewhere on or around the May 24th time frame. According to BusinessWire.com 21% of websites are still using SHA1 certificates. Basically over 1/5 of the sites on the internet are using a woefully weak cipher suite and if they are still doing so near the end of May, they will be doing so with the source code for how to exploit them. A colleague of mine once told me, as I was lamenting my frustration at apathy in the enterprise, “Sheep hate the sheep dog more than the wolf”. In this case, I see it more a matter of the sheep dog being so fed up that he or she is basically warning them that they will not only be left behind, but they are going to tell the wolves how to get at them. You may or may not agree with this methodology but regardless, those who do not heed the warning and fall in with the rest of the flock may find themselves being part of the “thinning the herd” as, most assuredly, the wolves will gather. One of the challenges in many enterprises is understanding what your exposure is. There are tools that will let you scan systems, etc. but the process is two-fold. You could spend hundreds of hours securing your servers only to be breached by a B2B Partner or an IoT device that has a weak cipher. While over 1/5 of the internet is still using SHA1, I am betting that internally it is much worse. If we have learned anything over the last 36 months it’s that the perimeter won’t keep folks out and while the wolves may gather in the DMZ, they will work just as easily in the dark when Fred in payroll opens that Email attachment or clicks that picture. As enterprise folks, we own the responsibility for thinning our own flock and keeping our own strays in line.
You may be wondering how you can do this, both internal, external and B2B? It may not go over well if you called your B2B partners and told them you were going to start scanning their systems. A solution that allows you to engage in careful surveillance of all SSL transactions and determine the cipher suites used will position you to be able to determine your entire exposure without scanning or crawling yours or any other’s network. Using ExtraHop’s wire data analytics you can observe your SSL Transactions and will be able to start the process of getting out of technical debt by fixing the issues one system, network and B2B partner at a time.
From the same cited article above, getting visibility into what your exposure is can be difficult:
“The results of our most recent analysis are not surprising” said Kevin Bocek, chief security strategist for Venafi. “Even though most organizations have worked hard to migrate away from SHA-1, they don’t have the visibility and automation necessary to complete the transition. We’ve seen this problem before when organizations had a difficult time making coordinated changes to keys and certificates in response to Heartbleed, and unfortunately I’m sure we are going to see it again.”
If you leverage our wire data analytics platform you can easily audit your exposure by using one of our canned reports on Cipher auditing or set up a quick trigger to audit it yourself.
How it works: ExtraHop sits back passively and observes network traffic via a mirror (Tap, Span, etc). So within this Application Inspection Trigger I am doing the following:
Checking to see if the Certificate signature has “SHA1″ anywhere.
If it exists, I write the record to our EXA Elastic appliance where I can get a quick look at what my exposure is to SHA1 both from clients and servers. I can also see who issued the weak certificate (be ready to call you IoT vendors and shrink wrapped software)
Once we have started to write the information to the ExtraHop Explorer Appliance (EXA) we can get an idea of what our exposure is in two clicks (allowing sufficient time to build the data set)
Click 1: Select “SHA1 Audit” from the Record Type combo box.
Click 2: In the “Group By” combo box, select “Server IP Address”
Below you see a list of every server using SHA1. Some of them are internet servers and some of them are internal systems. If we want, we can separate internal systems within the trigger using the .isRFC1918 function and only look at our internal systems.
Conclusion: Over the next two months, it will be important that we are able to determine our exposure to SHA1, not only for the servers we have on the internet, but internally and within the cloud providers that we are using. Moore’s law has dictated the hardware and computing power to break cipher suites will continue to get better and cheaper. SHA1 had a run of over 20 years (although it has been considered week for the last few years). Cipher suites becoming obsolete is part of the digital cycle of life. It took me a few minutes to write the trigger you see here and we already have canned auditing tools in the ExtraHop bundle gallery.
Getting visibility this quickly and getting an idea of our risk as easily as this is why we “wire data”.
As the new discipline of Threat Intelligence takes shape, Cyber Security teams will need to take a hard look at their existing tool sets if they want to take advantage of the dynamic, ever changing threat intelligence feeds providing them with information on which hosts are malicious and whether or not any of their corporate nodes have engaged in any sort of communications with any of the malicious hosts, DNS names or hashes that you are collecting from your CTI (Cyber Threat Intelligence) feeds. Currently the most common way that I see this accomplished is through the use of logs. Innovative products like Alienvault and Splunk have the ability to check the myriad of log files that they collect and cross reference them with CTI fees and check to see there have been any IP based correspondence with any known malicious actors called out by such feeds.
Today I want to talk about a different, and in my opinion, better way of integrating with Cyber Threat Intelligence using Wire Data and the ExtraHop Platform featuring the Discover and Explorer Appliances respectively.
How does it work? Well let’s first start with our ingredients.
A threat analytics feed (open source, subscription, Bro or CIF created text file)
A peer Unix-based system to execute a python script (that I will provide)
An ExtraHop Discover Appliance
An ExtraHop Explorer Appliance
ExtraHop Discover Appliance:
An appliance that can passively (no agents) read data at speeds from 1GB to 40GB. It can also scale horizontally to handle large environments.
ExtraHop Explore Appliance:
ExtraHop’s Elastic appliance that allows for grouping and string searching INTEL gathered off the wire.
Session Table: ExtraHop’s memcache that allows for instant lookup of known malicious hosts.
The solutions works by using the Unix peer to execute a python script that will collect the threat intelligence data. It then uploads the malicious hosts into the Discover Appliance’s Session Table (up to 32K records). The Discover appliance then waits to observe a session that connects with one of these known malicious sites. If it sees a session with a known site from the TI feed activities include, but are not limited to the following:
Updates a Threat Intelligence dashboard
Triggers an alert that warns the appropriate Incident Response team(s) about the connection to the malicious host
Writes a record to the ExtraHop Explorer Device
Triggers a Precision PCAP capturing the entire session/transaction to a PCAP file to be leveraged as digital evidence in the event that “Chet” the security guard needs to issue someone a cardboard box! (not sure if any of you are old enough to remember “Chet” from weird science)
Please Click Image:
Below you see the ExtraHop Threat Intelligence Monitoring Dashboard (last 30 minutes) showing the Client/Server and Protocol as well as the Alert and a running count of violations: (this is all 100% customizable)
Please Click Image:
On the Explorer Appliance, we see the custom data format for Malicious Host Access and we can note the regularity of the offense Please Click Image:
And finally we have the Precision Packet Capture showing a PCAP file for forensics, digital evidence and if needed, punk busting. Please Click Image:
Conclusion: The entire process that I have outlined above took less than one minute to complete every single task (Dashboard, Alert, EXA, PCAP). According to Security Week, the average time to detect a breach has “Improved” to 146 days in their 2015 report. Cyber Threat Intelligence has a chance to drastically reduce the amount of time it takes to detect a breach but it needs a way to interact with existing data. ExtraHop positions your Threat Intelligence investment to interact directly with the network, and in real time. Many incumbent security tools are not built to accommodate solutions like CTI feeds via API or do not have an open architecture to leverage Threat Intelligence, much less use memcache to do quick lookups. The solution outlined above using ExtraHop with a Threat Intelligence feed positions INFOSEC teams to be able to perform Advanced Persistent Surveillance without the cost of expensive log indexing SIEM solutions. Since the data is analyzed in flight and in real time, you have a chance to greatly reduce your time to detection of a breach, maybe even start the Incident Response process within a few minutes!
What you have read here is not a unicorn, this exists today, you just need to open your mind to leveraging the network as a data source (in my opinion the richest) that can work in conjunction with your log consolidation strategy and maximize your investment in Cyber Threat Intelligence.
Incidentally, the “Malicious Host” you see in my logs is actually wiredata.net. I did NOT want to browse any of the hosts on the blacklist so I manually added my host to the blacklist the accessed it. Rest assured, WireData.net is not on any blacklists that I am aware of!
The Ransomware epidemic has spread on the internet like a plague in the last 18 months. In fact, Ransomware netted $200 million in Q1 of this year. One might think with numbers like that they would get acquired or start an IPO in the next year!! I predict “Ransomware” makes it into Webster’s dictionary by 2017! (You read it here first)As many (well…to the extent that I could call my readers “many”) of you have read in my earlier posts, I believe that the lack of surveillance, due to budget, staffing or just good ole fashion apathy, is the primary reason for most security breaches. However, in the case of Ransomware, I believe that wire data offers the only way to truly combat this. Ransomware plays out in the blind spot behind the perimeter, in a day and age when your credit report can keep you from renewing your clearance or even getting a job. This coupled with the COMPLETE AND UTTER LACK of advocacy for the consumer or accountability when the information is wrong , an email stating that you are delinquent on a bill is always taken VERY seriously and to think that people will just not open it isn’t necessarily practical. That said, the phishing attempts will continue to evolve and as we block one, they will program another. This is okay! When you use ExtraHop’s wire data analytics, you can pivot too. I know that threat intelligence is still developing but with the availability of restful API’s and an open platform, you have a great shot at keeping your malware/Ransomware exposure to a minimum.
In this post, I want to demonstrate one of those methods. Today I am going to walk through how I set up ExtraHop to integrate with one of our partners (Citrix) OctoBlu platform to just give you an example of how who open platforms can integrate with one another and provide unparalleled visibility as well as access to automatic workflows that can be used to corral infected systems and decrease exposure. I have been studying attack vectors for Cryptowall and in many cases, a user is redirected to a .SWF URI that contains the malicious software or directs them to download/install it. I want to demonstrate how you can leverage ExtraHop and OctoBlu to be able to audit access to these files and ensure that they are, in fact, not from malicious sources.
Definitians: There is some overlapping nomenclature with OctoBlu and ExtraHop so I want to take the time to define it.
OctoBlu Trigger: (I am still learning about OctoBlu but…) An OctoBlu trigger is the item within an OctoBlu flow that initiates the actual workflow that is being performed.
On the ExtraHop System: We set up a trigger that looks for externally accessed SWF files.
Online 14 we are looking for any uri that has .swf in it.
On line 15 we indicate that we are looking for non RFC1918 addresses (no 10,192 or 172 networks, just external) serving up the .swf file.
Then on lines 23 – 26 we access the OctoBlu trigger URI location to kick off the OctoBlu Flow.
Line 24 calls out my specific OctoBlu URI. (to avoid pranksters who will send me 10,000 emails)
And Finally, on lines 29 – 41 we are initiating what we call a “Precision Packet Capture” that will also create a PCAP file that I can download and evaluate as digital evidence.
On the OctoBlu system: The OctoBlu flow has been set up to leverage four tools and one thing. You will see them defined below as follows:
Trigger (Tool): The Trigger is what initiates the flow, it has a specific URI assigned to it (called out in Line 24 above) and begins the workflow.
It receives the JSON payload from ExtraHop (Line 26) and sends the Server IP delivering the SWF File to the HTTP GET tool which then queries VirusTotal.com’s API passing my API Key and the Server IP.
It also passes the Client and Server IP’s as well as a URL to the “Compose Tool”
HTTP Get (Tool): This is the actual query of the VirusTotal API that checks to see if the IP is malicious or not.
Compose (Tool): Labeled “Consolidate JSON messages” below, this takes the JSON objects from both the initial trigger and the HTTP Get tools and creates a single set of metrics to be passed to the Template Tool.
Template (Tool): Labeled “Prepare Message” below, this takes all of the JSON metrics created in the previous tool and sends them to the “Send Email” thing
Send Email (Thing): This is the actual act of sending me an email warning me that I have had a user access a .SWF file from a malicious source.
(Please Click the Image)
The Warning from OctoBlu: Below is a copy of the email I received when I replayed the pcap file from malware-traffic-analysis.net. As you can see, it includes the Server, the Client and a link to the Server IP’s VirusTotal dossier.
The Digital Evidence: As noted on lines 29 – 41 in the ExtraHop trigger, we have also kicked off a Precision Packet Capture. This makes the actual transaction readily available to download and look at in WireShark and use to determine if there is an actual issue as well as leverage the PCAP itself as digital evidence. As you see below, you have a PCAP named “External SWF File Accessed”.
Conclusion: So, the question was asked by @Brianmadden on twitter as he remarked that OctoBlu was now “True Blue Citrix”, “What can you do with it?”. With the right integration platform I believe that there is quite a bit that can be done with OctoBlu both with ExtraHop but also with their own portfolio of tools. What I love about the two platforms is their open-ness and their ability to increase the aperture of both Security, Dev and Network Operations teams allowing them to have the kind of agility needed to fight in today’s “hand-to-hand combat” world where breaches and vectors pivot, stick and move on a monthly, weekly and daily basis.
Using ExtraHop I have been able to deliver the following integration solutions: (with more to come from an ExtraHop “Blu Bundle”).
Get warning about users who are experiencing high latency
Get warnings about long logon times that fall outside an SLA
Now the post above, where I am warned when an end user accesses an known malicious external flash content.
Other scenarios could include using the Netscaler HTTP Callout feature to warn you when a user launches an ICAPROXY session from outside the US (a breach that actually happened), or when a known malicious actor accesses the company website hosted on a Netscaler VIP. You could also, potentially, use OctoBlu and MeshBlu to shut off Netbios on a system that we see encrypting file shares with Cryptowall.
My last few posts have been centered around how we can go about finding potential breaches in your environment using ExtraHop’s wire data analytics platform. Most of these have involved placing a logical boundary around a set of CIDR blocks and reporting on L4 transactions that fall outside defined boundaries. In the case of our Stream Analytics Critical Control Points post, we look at connections from PROD to networks other than PROD and taking a packet capture when we note a violation. (Please see SACCP post).
In this next post, I want to talk about how we can provide surveillance around L7 security. In today’s post we are going to look at monitoring database traffic at Layer 7.
A disgruntled employee is about to start querying a sensitive database to steal important information. Since the user has approved credentials and will be accessing the database over approved ports and using approved protocols, many standard security tools will not detect this insider’s behavior as they are operating in the “Behind the Perimeter Blind Spot” that exists in most organizations. In this hypothetical scenario, we have a table called “Employees” that we want to audit any/all ad hoc selections against it. The way the CRM application is set up, the only transactions we should observe against this table would involve stored procedures. Any “Select, Insert, Update or Drop” methods should be alerted on immediately.
Database: CRM Sensitive Table: Employees
We set up the audit trigger below telling us to start a PCAP capture in the event that we see any sort of database transaction that includes “from employees”.
We assign this trigger to the Database server housing our CRM Application and we should then be alerted any time someone runs an ad hoc query against the database.
Now, when an insider makes an ad hoc query, we can alert, send a syslog or, as in the case of the trigger above, initiate a packet capture. As you can see below, we see a number of SENSITIVE TABLE PCAP files pertaining to the client that ran the report as well as the server it was run against.
When we look at the PCAP we can see the Ad Hoc query that they ran as well as use the PCAP file as digital evidence to begin the process of dealing with the individual violating the policy. As you can see, the user ran the query “select LastName from Employees”. This is demo data but that could have been Select * from Customers or CCards.
As I recall, the Anthem breach was actually a system owner who notices someone running Ad Hoc queries against their customer database. By providing Layer 7 visibility into the actual transaction, insiders can be effectively “named and shamed” when auditing with ExtraHop’s Wire Data platform. In today’s world, credentials are a joke (trust me, I sit at the core and look at packets all day). When an insider is using approved credentials and coming in over approved ports and protocols, the aperture needs to be increased to provide visibility into the L7 transactions to ensure that they are appropriate and that an insider is not running unauthorized queries against your sensitive data.
Layer 7 auditing with ExtraHop positions you to give insiders a cardboard box, not access!
I left the enterprise approximately 30 months ago after being a cubicle drone for the last 18 years. I now work for ExtraHop Networks, a software company that makes a wire data analytics platform for providing operational intelligence to organizations around their applications, the data that traverses their wire and basically shines light on the somewhat opaque world of packet analysis.
In the last few years, I can honestly say that I find myself getting a bit frustrated with the number of breaches that have occurred due, in my opinion, in large part to the lack of involvement by system owners in their own security. For my household alone, in the last 24 months, we are on our 5th credit card (in fact, I look at my expiration dates on most of my credit cards and chuckle on the inside knowing I will never make it.) I am also a former Federal Employee with a clearance so I also have the added frustration of knowing several Chinese hackers likely had access to my SF86 information (basically my personal and financial life story). In the last 15 years, we have added a range of regulatory framework, Security Operations Centers (SOC), I have watched INFOSEC budgets bulge while needing to justify my $300 purchase of Kiwi Syslog server. I have concluded that maybe the time has come for the industry to try a new approach. The breaches seem to get bigger and no matter what we put in place, insiders or hackers just move around it. At times I wonder if a framework I learned in my career prior to Information Technology may be just what the industry needs?
My first job out of College was with Maricopa County Environmental Health (I was the health inspector) and I was introduced to a concept called HACCP (Hazard Analysis Critical Control Point) and I think some of what I learned from it can be very relevant in analyzing today’s distributed and often problematic environments.
HACCP, pronounced “hassup”, is a methodology of ensuring food safety by the development of a series of processes that ensure, in most cases, that no one gets sick from eating your food. It involves evaluating the ingredients of each dish and determining which food is potentially hazardous and what steps need to be taken to ensure that quality is ensured/maintained from food prep to serving.
While working as the health inspector, I was required to visit every permit holder twice a year and perform a typical inspection that involved taking temperatures, making sure they had hot water, employees washed hands and stayed home when they were sick, etc. But in most if not all of the restaurants I inspected, the process of checking temperatures, ensuring there is soap at the hand wash station and making sure there is hot water did not JUST happen during an inspection, I knew that in most cases it went on even when I was not on the premises. Sadly, in today’s enterprise, generally systems are only checked and/or monitored when an application team is being audited. An incumbent INFOSEC team cannot be responsible for the day to day security of a shared services or hosting team’s applications any more than I could be in every single restaurant every single day. The operator has to take responsibility; I am proposing the same framework for today’s enterprise. Share services and hosting teams need to take responsibility for their own security and use INFOSEC as an auditing and escalation solution. I will attempt to parallel how ExtraHop’s Stream analytics solution can provide an easy way to accomplish this even in today’s skeleton crew enterprise environments.
Let’s start with some parallels.
An example of a HACCP based SOP would be:
The cooling of all pre-cooked foods will ensure that foods are cooled from 135 degrees to 70 degrees within two hours
The entire cooling process from 135 degrees to 41 degrees will not take more than 6 hours.
So, I am taking away the “H” and putting in an “S” for SACCP I am proposing that we do the same for our applications and systems that we support at the packet level. Just as ingredients may have chicken, cheese and other potentially hazardous ingredients applications may have SSO logins, access tokens, PII being transferred between DB and Middle or Front End tiers. We need to understand each part of an infrastructure that represents risk to an application and what an approved baseline is, what mitigation steps to take and who is responsible for maintaining it. Let’s take a look at the 7 HACCP/SACCP principles.
Principle 1 – Conduct a Hazard Stream Analysis
The application of this principle involves listing the steps in the process and identifying where there may be significant risk. Stream analytics will focus on hazards that can be prevented, eliminated or controlled by the SACCP plan. A justification for including or excluding the hazard is reported and possible control measures are identified.
Principle 2 – Identify the Critical Control Points
A critical control point (CCP) is a point, transaction or process at which control (monitoring) can be applied to ensure compliance and, if needed, a timely response to a breach.
Principle 3 – Establish Critical Limits
A full understanding of acceptable thresholds, ports and protocols of specific transactions will help with identifying when CCP is outside an acceptable use.
Principle 4 – Monitor Critical Control Point
Monitor compliance with CCPs using ExtraHop’s Stream analytics Discover and Explorer appliances to ensure that communications are within the expected and approved ports and protocols established in each CCP.
Principle 5 – Establish Corrective Action
Part of this is not only understanding what to do when a specific critical control point is behaving outside the approved limits but to also establish who owns the systems involved in each CCP. For example, if a Critical Control Point for a server in the middle-tier of an application is suddenly SCP-ing files out to a server in Russia, establish who is responsible for ensuring that this is reported and escalated as soon as possible as well as establish what will be done in the event a system appears to be compromised.
Principle 6 – Record Keeping
Using the ExtraHop Explorer appliance, custom queries can be set up and saved to ensure that there is proper compliance with established limits. Also integration with an external SIEM for communications outside the established limits can be enabled as well as HTTP push and Alerting.
Principle 7 – Establish Verification
Someone within the organization, either the INFOSEC team or team lead/manager must verify that the SACCP plan is being executed and that it is functioning as expected.
So what would a SACCP strategy look like?
Lets do a painfully simple exercise using both the ExtraHop Discover Appliance and ExtraHop Explorer Appliance to create a Stream Analytics Critical Control Point profile.
Scenario: We have a Network that we want to call “Prod”.
Principal 1: Analysis
Any system with an IP Address starting with “172.2” is a member of the Prod network and there should ONLY be INGRESS sourcing from the outside (The Internet) and Peer-to-Peer communications between Prod Hosts. No system on the Prod network should establish a connection OUTSIDE Prod.
Principal 2: Identify CCPs In this case, the only Critical Control Point (CCP) is the Prod network.
Principal 3: Limits As stated, the limits are that Prod hosts can accept connections from the outside BUT they should not establish any sessions outside the Prod network.
Principal 4: Monitoring
Using the ExtraHop Discover Appliance (EDA) we will create a trigger that identifies transactions based on the logical network names of their given address space and monitor both the ingress and egress of these networks.
In the figure below, we will outline how we are setting a logical boundary to monitor communications. In this manor we can lay the groundwork for monitoring the environment by first identifying which traffic belongs to which network.
You see on line 5 in the trigger below we are establishing which IP blocks belong to the source (egress) networks.
You then see on line 11 we are identifying the prod network as a destination (ingress).
*Important, you DO NOT have to learn to write triggers as we will write them for you but we are an open platform and we do provide an empty canvas to our customers should they want to paint their own masterpiece thus we are showing you how we do it.
Next we will leverage the ExtraHop Explorer Appliance (EXA) to demonstrate where the traffic is going. You will see on line 28 (although commented out) we are committing several metrics to the EXA such as source, destination, protocol, bytes, etc. This completes principal 4 and allows us to monitor the Prod network. In the figure below, you will see that we are grouping by “Sources”. You will note that Prod has successfully been classified and it has over one million transactions.
Principal 5: Establish Corrective Action Well, in our hypothetical prod network, we have noted that there are some anomalies. As you can see below, when we filter on Prod as the source and we group by the Destinations we see that 15 of our nearly 1.3 million transactions were External. In most situations, this would go largely unnoticed by several tools however using SACCP and the ExtraHop’s Stream Analytics platform, the hosting team or SOC are positioned to easily see that there is an issue and begin the process of escalating it or remedying the issue with further investigation.
*Note, we can easily create an alert that can warn teams of when a transaction occurs outside the expected set of transactions. We also have a RESTful API that can be interrogated by existing equipment to see anomalies.
Digging Deeper: As we dig a little deeper by pivoting to the Layer 7 communications (demonstrated in the video below) you will note that someone has uploaded a file to an external site at 220.127.116.11. Depending on what was in that file and existing policies, the mitigation could involve a cardboard box and a visit from the security guard.
Principal 6: Establish Record Keeping The ExtraHop Discover Appliance has the ability to send a syslog to an incumbent SIEM system as well as a RESTFUL push. There is also a full alerting suite that can alert via email or SNMP Trap. In most enterprises, there is already an incumbent record keeping system, the ExtraHop platform has a variety of ways to integrate with the incumbent solution.
Principal 7: Verification Someone should provide oversight of the SACCP plan and ensure that it is being executed and that it is having the desired results. This can either be the INFOSEC team management or hosting team management but someone should be responsible for ensuring that the shared services team(s) is (are) following the plan.
Conclusion: The time has come for a new strategy, in several other industries where there is a regulatory framework for safety, compliance and responsibility there exists a culture of the operators taking responsibility for ensuring that they are compliant. The Enterprise is over 30 years old and just as the Health Inspector cannot be in every restaurant every day or a policeman cannot be on every street corner, the time has come for the IT industry to ask that system owners take some of the responsibility for their own security.
Thanks for reading and please check out the video below.
ADDENDUM!!! (PUNKBUSTER OPTION!)
I wanted to take the time to show the next iteration of this, I call it precision punk busting…”err”..I mean Packet Capture.
As a result of the FTP Traffic out to the internet we notice that we have a PCAP waiting for us indicating that a system has violated the Prod policy.
We can also alert you that you have a PCAP waiting for you either via Syslog, SNMP or Email. This PCAP can be used as forensics, digital evidence against an insider or a way to verify just wha the “F” just happened.
Having this information readily available and alerting either a system owner or SOC team that a policy was violated is a much easier surveillance method than sorting through Terabytes of logs or sifting through a huge PCAP file to get what you want. Here we are ONLY writing PCAPs for those instances that violate the policy.
Well, I cannot tell you what the next big breach will be, I CAN tell you that it will involve one critical system communicating to another system that it was/is not supposed to. Whether that is ex filtration via secure ssh (SCP) to a server in Belarus or mounting a hidden drive share using a default service account that was never changed, behind the perimeter represents a rather large blind spot for many security endeavors. In the video below, you are seeing a very quick and simple method for monitoring peer-to-peer communications using wire data with the ExtraHop Platform. This is a somewhat painful process with logs due to the fact that logging build-up and tear-downs can impact the hardware being asked to do the logging and if you are licensed to pay by how much data you index, it can be expensive. Grabbing Flow records directly off the wire positions your security practice to have this information readily available in real-time with no impact on the infrastructure as no system/switch is asked to run in debug mode. Transactions are taken directly off the wire by the ExtraHop Discover Appliance and written directly to the Elastic ExtraHop Explorer Appliance to provide a large and holistic view of every transaction that occurs within your critical infrastructure.
This positions system-owners with the ability to (within minutes) to audit critical systems and account for transactions that are suspect. In the video below, you will see how we can audit flow records by white listing expected communications and slowly leaving malicious communications no where to hide as you slowly reduce the surface area that you are trying to patrol.
I will follow this up with another post on how we can monitor Layer 7 metrics such as SQL Queries against critical databases.