Ever feel like this guy? (Above), today breaches are becoming far more complicated than just a virus, they involve practices like social engineering and phishing as well as the old, tried and true injection based hacks like Cross Site Scripting and SQL Injection. As these breaches become more sophisticated the tools we use to combat them need to evolve. Additionally, we need to come to grips with the idea that we are going to stop every single breach. Malware, viruses, social engineering and inadvertent clicks on malicious links are becoming like the common cold, we need to approach breaches with the idea of when and not if. Because I have a local police/fire department does not mean that I don’t lock my doors, look through window to see who it is before I open the door or install smoke alarms. Sadly, while INFOSEC as a discipline has evolved over the last 15 years I fear it has made those of us who are responsible for hosting systems a bit lazy and dependent on the existing INFOSEC apparatus and we have stopped thinking about our own Security. To expect the INFOSEC group in your organization to take responsibility for your system’s security is not entirely different than me going to the mayor in my city and asking that a policeman and fireman be present in my house at all times, just not very practical. For that reason, I need to take some responsibility in securing my home and property and use police and fire for emergencies. In 2013, Schnucks had customer credit card numbers and expiration dates siphoned out of their network for a period of nearly four months according to Network World. Now, I don’t have any inside knowledge of the Schnucks breach and, as with most breaches, they are very tight lipped about how they were compromised but if it were a case of APT or Malware the data would have to get FROM their systems TO someone else’s system.
Using Extrahop you can grab data from the wire and report on Egress traffic and identify anomalies or things that just don’t belong. But to do this there needs to be a paradigm shift in how INFOSEC operates, instead of just focusing on preventing breaches, how about focusing on quickly determining if you are breached? It is estimated that between 30 and 70 percent of malware goes initially undetected by anti-virus software. Like the poor guy in the graphic above, Anti-virus companies do a lot of whack-a-mole and in the absence of a crystal ball; I am not sure what else they are supposed to do. If we are going to Depend solely on shrink-wrapped products to protect our digital intellectual property we might as well also call Miss Cleo’s Psychic hotline once a day to find out if we are breached. At the end of the day, we need resurgence in the human being, in most cases; the best candidate for this is..well..you. You should have a full understanding of which systems have PII, Financial or PHI on them and can more easily identify pattern behavior that fits and does not fit. Example, if you see packets from your SQL server housing patient data making connections to Belarus that MIGHT be something out of the ordinary. The simple act of having a report sent to you once or twice a day, or providing parsed real-time data to your operations staff and teaching them how to interpret it (the way insurance companies have been training fraud investigators for 20 years) can help folks spot things that don’t make sense or don’t look right.
Anyone who watched Sesame Street as a kid understands how to spot an anomaly.
Some of these packets is doin’ their own thing!
For the Gen-Xers who remember Sesame Street, once you have data and are able to collate it and logically represent it, finding problems, breaches, malware becomes much easier. If you look below, you see that three kids are playing baseball and one kid is playing football. If you look to the right, you can see that we have a number of packets going to Beijing. My lab is located in the US and there is no reason anyone should be visiting Beijing websites (unless you are writing a blog and need some data). I am not trying to insult anyone’s intelligence here. I am simply pointing out how easy this is to do with Extrahop and Splunk. I did not change a single configuration on any servers to increase the apache log level, I did not have to put my ASA in debug mode and I did not have to install any agents. Extrahop grabbed all of this right off the wire and handed it off to Splunk for parsing and geocoding.
In today’s post I want to talk about four Extrahop triggers than can help you take part in being your own blue team and taking responsibility for the security of your servers.
FLOW_TICK: Incoming Data
First let’s look at our incoming data, using the FLOW_TICK feature of Extrahop I can see in coming sessions by external client, internal server and port. Then integrating it with Splunk allows you to take it even further by geo coding the IP Information and performing reverse lookups allowing you to see where they are coming from and what their DNS names look like.
What I look for/observe from this data:
I have always felt that if the IP Address did not have a DNS record then there are a couple of possibilities. They are either up to no good OR their ISP does not properly set up their DNS which makes me wonder if they are paying attention to what their subscribers are doing. At any rate, no reverse lookup is always a red flag for me.
Next I look at connections from the Russian Federation and China as those likely stuck out to you as well. This is a home lab and those addresses are obviously performing recon as there is utterly no reason whatsoever anyone in China, Japan or Russia would have any desire to look at my home lab.
The last problematic entry is obviously the Shodan queries, InfoSec practitioners are probably chuckling at seeing it but the singlehop2.shodanhq.com (Shodan) is basically an intelligence gathering site for posting your open ports on a google-like website where hackers can go in and check for open ports. You do not want your systems on Shodan. You can email them and ask them to stop performing recon on your IPs. Won’t be the first time I have done it.
sourcetype=”Syslog” FLOW_TICK | rex field=_raw “ClientIP=(?<ip>.[^:]+)\sServerIP” | geoip ip | stats count(_time) as Instances by ServerIP ip ServerPort ip_city ip_region_name ip_country_name | lookup dnsLookup ip | Search ip_city!=””
A Quick Walk-thru of monitoring ClientIP with Extrahop:
FLOW_TICK: Outgoing Data:
Perhaps even more important in the case of Shnucks is the monitoring of EGRESS which Extrahop can do at a gigabit(S) per second rate. This provides me visibility into where traffic is going. There are numerous free csv files with blacklists of malware sites and the existing security Practitioners within your company likely know where to find this and can set up a lookup table for you in Splunk if one does not already exist. Basically, Extrahop, using the same FLOW_TICK trigger previously mentioned, can log the outgoing traffic giving me the ClientIP, Server IP, Port and Splunk provides geographic information (City, State/Region, Country) and using the same reverse lookup, provide me the DNS host name as well.
What I look/observe for from this data:
First, you want to make sure you understand EVERY SINGLE SYSTEM that has critical, private, financial or any type of digital intellectual property. From here, you want to note the Egress patterns of those systems and see if they are making any external connections. First thing I would ask myself is, outside of patches and updates, why the hell any system I have that has sensitive data would EVER make an outgoing connection. By outgoing I mean outside of the local intranet however that does not mean that you should not look for someone copying data internally THEN taking it out on their laptop. The second concern that you want to look for is if someone has fallen victim to a phishing scam and click a link that goes outside your organization to a country known for state sponsored Cybercrime or a site that just doesn’t look right.
Below you see a series of connections to Google, here is where you may see an abundance of data under the “Instances” column that does not make sense. If you see high number of packets to a host/IP that doesn’t have anything to do with your business, then it you need to run it down and find out what is being sent to, or downloaded from, them.
sourcetype=”Syslog” FLOW_TICK | rex field=_raw “ServerIP=(?<ip>.[^:]+)\sServerPort” | geoip ip | stats count(_time) as Instances by ClientIP ip ServerPort ip_city ip_region_name ip_country_name | lookup dnsLookup ip | Search ip_city!=””
A Quick Walk-thru of monitoring ServerIP with Splunk/Extrahop:
Using the Extrahop triggers you can get a real-time view of all SSL Connections made on within your network and outside of your network. Below you see a list of servers that are access from inside the LAN. You see the SSL Version (It is a bit Cryptic but 769 is TLS 1.0, 770 is TLS 1.1 and 771 is TLS 1.2. As you can see in the screenshot we have the server IP, the SSL Version and we can also see the cipher.
What to look/observe for in this data:
This will help in identifying sites with weak ciphers as well as help with enforcing cipher standards internally. You can query for KEYSIZE as well (It’s in the trigger) but I did not include that as I don’t think you can even request a 1024 bit Certificate anymore can you? Also visible below is the expiration date of the SSL Certs alerting you if someone is using an outdate certificate or if your own certificates are about to expire. As a lot of you are aware, most folks click right through the Cert warnings and go straight to the site.
I will be the first to admit I am not in the INFOSEC-Proper mold. I worked for a year doing event correlation in the early 2000’s but as far as noticing what an HTML Injection looks like I am really not that guy. BUT, what I can do is get an idea of what my HTTP traffic should look like and I can tell if someone is injecting new header values or issuing an unauthorized redirect.
What to look/observe for in the data:
The Extrahop HTTP Triggers can give you a broad level of visibility into your HTTP traffic alerting you to potentially compromised websites or even catching malware that is trying to sneak out over HTTP. I can also see, in real time, if a cookie is assigned to more than one IP Address and note if I have a session hijacking issue going on. I look at URI stems and within Splunk you could likely match them up with known malicious code. I look for a high amount of traffic on odd ports (trying to sneak out). I look for User-Agents like “Python” and other values that could indicate someone using Metasploit or some other hacking tool as most of your User-Agents should be of the “Mozilla” variety. As we have done previously, I might geocode the data to see where the users are actually going or I might also include the HOST header value. I highly recommend reading the Trigger API documentation for Extrahop because they trigger on a lot more than I can speak to with any level of expertise. If you are an INFOSEC practitioner and can comment on what to look for, please do.
Splunk Query: (the table was too big to include the GEO data)
HTTP_REQ | geoip ServerIP | table ClientIP ServerIP ServerPort Payload HTTP_uri HTTP_query User_Agent CookieID ServerIP_country_name | Search ServerIP_country_name!=””
Below is some of the traffic I generated to China (purposely)
HTTP_REQUEST : Cookie Watching
Another nice tool is to run a query that checks the total number of IP Addresses using a CookieID. If you EVER see more than one IP using the same Cookie ID you need to alert your INFOSEC team post haste! This is a very strong indication that someone is hijacking sessions.
Catching Hijacked PHP Sessions in seconds with Extrahop/Splunk:
What I look/observer for in the data:
As I stated, you want to check and make sure no cookies are in use by more than one IP Address. Remember, this is real time, you don’t have to back through Apache/IIS logs to look at it after the fact, this will be evident in real-time. If you have an Operations Center, this is one of the items I would alert on or have readily available to watch.
HTTP_REQUEST | stats distinct_count(ClientIP) by CookieID
As I stated, I did not have to make any configuration changes to any systems (outside of a port span on the switch) and I did not have to install any agents to get this data in real time. In light of some of the breaches I have looked at for this year one common theme has been that there didn’t seem to be anyone watching the door. No matter what malfeasance is going on, to steal information it is going to HAVE to come across the wire (outside of blatant hardware theft). If it happens on the wire, Extrahop positions you to see it within seconds of it happening and take immediate steps to mitigate within minutes instead of days, weeks and in some cases, months. For me, there are two types of systems, compromised, and “about-to-get” compromised. I am not saying that we should abandoned existing preventative measures and policies as we are bound by the regulatory framework for our existing verticals but I think the time has come for hosting operations to start to take some role in their own security. We are already seeing under-writers balk at paying for breaches due to what they perceive as inadequate steps to protect.
When I purchased my home owners insurance in Florida (any Floridians know we are the “whipping post” of the home owners insurance industry) they actually denied my policy because I did not have an arm-rail on my back steps. After complaining profusely I agreed to have an arm-rail installed (the great irony being, while moving in I fell ass-over-tea kettle due to the lack of the very arm-rail my agent was requiring). The point is, as insurance carriers and umbrella policy writers become more technically savvy, just like my house and just as the banks have had to do more fraud investigation with Credit Cards, expect them to start demanding more proactive approach. Algorithm-based information security is just a part of keeping your intellectual property secure and likely won’t be enough for regulators and umbrella policy writers in the future.
In the case of Schnucks, Liberty Mutual (Schnuck’s Umbrella Policy Writer) is already balking at paying for the breach stating that it is not “property damage”. Sony is in a similar predicament with Zurich over the PSP network breach. In the end corporations are going to want breaches covered under their umbrella policies or there will be supplemental policies for data breaches and it is going to take the insurance industry about 6 seconds to start requiring non-algorithm based security and more proactive approach. As the threat landscape changes the tools and methods you use need to change but expect the regulatory framework to change as well. Having logs that you consult days/weeks after a breach is not proactive enough to protect intellectual property, you need to be able to provide a precise narrative to person who can interpret it. You cannot get more proactive than grabbing the data right off the wire and Extrahop appliances can support up to 20 Gbps and can be clustered to support even more.
My next post will also be INFOSEC based covering DNS, Databases and CIFS.
Thanks for reading, I hope you enjoy the new site.
John M. Smith
To read about Extrahop’s Security position check out:
To download a free Discovery Edition of Extrahop follow the link below:
If you have ANY questions about how to set this up, don’t hesitate to reach out to me at firstname.lastname@example.org