My last few posts have been centered around how we can go about finding potential breaches in your environment using ExtraHop’s wire data analytics platform. Most of these have involved placing a logical boundary around a set of CIDR blocks and reporting on L4 transactions that fall outside defined boundaries. In the case of our Stream Analytics Critical Control Points post, we look at connections from PROD to networks other than PROD and taking a packet capture when we note a violation. (Please see SACCP post).
In this next post, I want to talk about how we can provide surveillance around L7 security. In today’s post we are going to look at monitoring database traffic at Layer 7.
A disgruntled employee is about to start querying a sensitive database to steal important information. Since the user has approved credentials and will be accessing the database over approved ports and using approved protocols, many standard security tools will not detect this insider’s behavior as they are operating in the “Behind the Perimeter Blind Spot” that exists in most organizations. In this hypothetical scenario, we have a table called “Employees” that we want to audit any/all ad hoc selections against it. The way the CRM application is set up, the only transactions we should observe against this table would involve stored procedures. Any “Select, Insert, Update or Drop” methods should be alerted on immediately.
Database: CRM Sensitive Table: Employees
We set up the audit trigger below telling us to start a PCAP capture in the event that we see any sort of database transaction that includes “from employees”.
We assign this trigger to the Database server housing our CRM Application and we should then be alerted any time someone runs an ad hoc query against the database.
Now, when an insider makes an ad hoc query, we can alert, send a syslog or, as in the case of the trigger above, initiate a packet capture. As you can see below, we see a number of SENSITIVE TABLE PCAP files pertaining to the client that ran the report as well as the server it was run against.
When we look at the PCAP we can see the Ad Hoc query that they ran as well as use the PCAP file as digital evidence to begin the process of dealing with the individual violating the policy. As you can see, the user ran the query “select LastName from Employees”. This is demo data but that could have been Select * from Customers or CCards.
As I recall, the Anthem breach was actually a system owner who notices someone running Ad Hoc queries against their customer database. By providing Layer 7 visibility into the actual transaction, insiders can be effectively “named and shamed” when auditing with ExtraHop’s Wire Data platform. In today’s world, credentials are a joke (trust me, I sit at the core and look at packets all day). When an insider is using approved credentials and coming in over approved ports and protocols, the aperture needs to be increased to provide visibility into the L7 transactions to ensure that they are appropriate and that an insider is not running unauthorized queries against your sensitive data.
Layer 7 auditing with ExtraHop positions you to give insiders a cardboard box, not access!
I left the enterprise approximately 30 months ago after being a cubicle drone for the last 18 years. I now work for ExtraHop Networks, a software company that makes a wire data analytics platform for providing operational intelligence to organizations around their applications, the data that traverses their wire and basically shines light on the somewhat opaque world of packet analysis.
In the last few years, I can honestly say that I find myself getting a bit frustrated with the number of breaches that have occurred due, in my opinion, in large part to the lack of involvement by system owners in their own security. For my household alone, in the last 24 months, we are on our 5th credit card (in fact, I look at my expiration dates on most of my credit cards and chuckle on the inside knowing I will never make it.) I am also a former Federal Employee with a clearance so I also have the added frustration of knowing several Chinese hackers likely had access to my SF86 information (basically my personal and financial life story). In the last 15 years, we have added a range of regulatory framework, Security Operations Centers (SOC), I have watched INFOSEC budgets bulge while needing to justify my $300 purchase of Kiwi Syslog server. I have concluded that maybe the time has come for the industry to try a new approach. The breaches seem to get bigger and no matter what we put in place, insiders or hackers just move around it. At times I wonder if a framework I learned in my career prior to Information Technology may be just what the industry needs?
My first job out of College was with Maricopa County Environmental Health (I was the health inspector) and I was introduced to a concept called HACCP (Hazard Analysis Critical Control Point) and I think some of what I learned from it can be very relevant in analyzing today’s distributed and often problematic environments.
HACCP, pronounced “hassup”, is a methodology of ensuring food safety by the development of a series of processes that ensure, in most cases, that no one gets sick from eating your food. It involves evaluating the ingredients of each dish and determining which food is potentially hazardous and what steps need to be taken to ensure that quality is ensured/maintained from food prep to serving.
While working as the health inspector, I was required to visit every permit holder twice a year and perform a typical inspection that involved taking temperatures, making sure they had hot water, employees washed hands and stayed home when they were sick, etc. But in most if not all of the restaurants I inspected, the process of checking temperatures, ensuring there is soap at the hand wash station and making sure there is hot water did not JUST happen during an inspection, I knew that in most cases it went on even when I was not on the premises. Sadly, in today’s enterprise, generally systems are only checked and/or monitored when an application team is being audited. An incumbent INFOSEC team cannot be responsible for the day to day security of a shared services or hosting team’s applications any more than I could be in every single restaurant every single day. The operator has to take responsibility; I am proposing the same framework for today’s enterprise. Share services and hosting teams need to take responsibility for their own security and use INFOSEC as an auditing and escalation solution. I will attempt to parallel how ExtraHop’s Stream analytics solution can provide an easy way to accomplish this even in today’s skeleton crew enterprise environments.
Let’s start with some parallels.
An example of a HACCP based SOP would be:
The cooling of all pre-cooked foods will ensure that foods are cooled from 135 degrees to 70 degrees within two hours
The entire cooling process from 135 degrees to 41 degrees will not take more than 6 hours.
So, I am taking away the “H” and putting in an “S” for SACCP I am proposing that we do the same for our applications and systems that we support at the packet level. Just as ingredients may have chicken, cheese and other potentially hazardous ingredients applications may have SSO logins, access tokens, PII being transferred between DB and Middle or Front End tiers. We need to understand each part of an infrastructure that represents risk to an application and what an approved baseline is, what mitigation steps to take and who is responsible for maintaining it. Let’s take a look at the 7 HACCP/SACCP principles.
Principle 1 – Conduct a Hazard Stream Analysis
The application of this principle involves listing the steps in the process and identifying where there may be significant risk. Stream analytics will focus on hazards that can be prevented, eliminated or controlled by the SACCP plan. A justification for including or excluding the hazard is reported and possible control measures are identified.
Principle 2 – Identify the Critical Control Points
A critical control point (CCP) is a point, transaction or process at which control (monitoring) can be applied to ensure compliance and, if needed, a timely response to a breach.
Principle 3 – Establish Critical Limits
A full understanding of acceptable thresholds, ports and protocols of specific transactions will help with identifying when CCP is outside an acceptable use.
Principle 4 – Monitor Critical Control Point
Monitor compliance with CCPs using ExtraHop’s Stream analytics Discover and Explorer appliances to ensure that communications are within the expected and approved ports and protocols established in each CCP.
Principle 5 – Establish Corrective Action
Part of this is not only understanding what to do when a specific critical control point is behaving outside the approved limits but to also establish who owns the systems involved in each CCP. For example, if a Critical Control Point for a server in the middle-tier of an application is suddenly SCP-ing files out to a server in Russia, establish who is responsible for ensuring that this is reported and escalated as soon as possible as well as establish what will be done in the event a system appears to be compromised.
Principle 6 – Record Keeping
Using the ExtraHop Explorer appliance, custom queries can be set up and saved to ensure that there is proper compliance with established limits. Also integration with an external SIEM for communications outside the established limits can be enabled as well as HTTP push and Alerting.
Principle 7 – Establish Verification
Someone within the organization, either the INFOSEC team or team lead/manager must verify that the SACCP plan is being executed and that it is functioning as expected.
So what would a SACCP strategy look like?
Lets do a painfully simple exercise using both the ExtraHop Discover Appliance and ExtraHop Explorer Appliance to create a Stream Analytics Critical Control Point profile.
Scenario: We have a Network that we want to call “Prod”.
Principal 1: Analysis
Any system with an IP Address starting with “172.2” is a member of the Prod network and there should ONLY be INGRESS sourcing from the outside (The Internet) and Peer-to-Peer communications between Prod Hosts. No system on the Prod network should establish a connection OUTSIDE Prod.
Principal 2: Identify CCPs In this case, the only Critical Control Point (CCP) is the Prod network.
Principal 3: Limits As stated, the limits are that Prod hosts can accept connections from the outside BUT they should not establish any sessions outside the Prod network.
Principal 4: Monitoring
Using the ExtraHop Discover Appliance (EDA) we will create a trigger that identifies transactions based on the logical network names of their given address space and monitor both the ingress and egress of these networks.
In the figure below, we will outline how we are setting a logical boundary to monitor communications. In this manor we can lay the groundwork for monitoring the environment by first identifying which traffic belongs to which network.
You see on line 5 in the trigger below we are establishing which IP blocks belong to the source (egress) networks.
You then see on line 11 we are identifying the prod network as a destination (ingress).
*Important, you DO NOT have to learn to write triggers as we will write them for you but we are an open platform and we do provide an empty canvas to our customers should they want to paint their own masterpiece thus we are showing you how we do it.
Next we will leverage the ExtraHop Explorer Appliance (EXA) to demonstrate where the traffic is going. You will see on line 28 (although commented out) we are committing several metrics to the EXA such as source, destination, protocol, bytes, etc. This completes principal 4 and allows us to monitor the Prod network. In the figure below, you will see that we are grouping by “Sources”. You will note that Prod has successfully been classified and it has over one million transactions.
Principal 5: Establish Corrective Action Well, in our hypothetical prod network, we have noted that there are some anomalies. As you can see below, when we filter on Prod as the source and we group by the Destinations we see that 15 of our nearly 1.3 million transactions were External. In most situations, this would go largely unnoticed by several tools however using SACCP and the ExtraHop’s Stream Analytics platform, the hosting team or SOC are positioned to easily see that there is an issue and begin the process of escalating it or remedying the issue with further investigation.
*Note, we can easily create an alert that can warn teams of when a transaction occurs outside the expected set of transactions. We also have a RESTful API that can be interrogated by existing equipment to see anomalies.
Digging Deeper: As we dig a little deeper by pivoting to the Layer 7 communications (demonstrated in the video below) you will note that someone has uploaded a file to an external site at 184.108.40.206. Depending on what was in that file and existing policies, the mitigation could involve a cardboard box and a visit from the security guard.
Principal 6: Establish Record Keeping The ExtraHop Discover Appliance has the ability to send a syslog to an incumbent SIEM system as well as a RESTFUL push. There is also a full alerting suite that can alert via email or SNMP Trap. In most enterprises, there is already an incumbent record keeping system, the ExtraHop platform has a variety of ways to integrate with the incumbent solution.
Principal 7: Verification Someone should provide oversight of the SACCP plan and ensure that it is being executed and that it is having the desired results. This can either be the INFOSEC team management or hosting team management but someone should be responsible for ensuring that the shared services team(s) is (are) following the plan.
Conclusion: The time has come for a new strategy, in several other industries where there is a regulatory framework for safety, compliance and responsibility there exists a culture of the operators taking responsibility for ensuring that they are compliant. The Enterprise is over 30 years old and just as the Health Inspector cannot be in every restaurant every day or a policeman cannot be on every street corner, the time has come for the IT industry to ask that system owners take some of the responsibility for their own security.
Thanks for reading and please check out the video below.
ADDENDUM!!! (PUNKBUSTER OPTION!)
I wanted to take the time to show the next iteration of this, I call it precision punk busting…”err”..I mean Packet Capture.
As a result of the FTP Traffic out to the internet we notice that we have a PCAP waiting for us indicating that a system has violated the Prod policy.
We can also alert you that you have a PCAP waiting for you either via Syslog, SNMP or Email. This PCAP can be used as forensics, digital evidence against an insider or a way to verify just wha the “F” just happened.
Having this information readily available and alerting either a system owner or SOC team that a policy was violated is a much easier surveillance method than sorting through Terabytes of logs or sifting through a huge PCAP file to get what you want. Here we are ONLY writing PCAPs for those instances that violate the policy.