A customer recently requested details on how Emerging Threats ensures coverage of rules which leverage TLS keywords for environments which have TLS decryption in place. This specific environment involves the TLS decryption process removing the TLS Protocol and presenting plain text HTTP to the IDS Engine.
A great question that I figured was worth sharing with everyone!
In general, Emerging Threats attempts to write signatures which provide coverage regardless if an environment has TLS decryption or not. In some cases, customers with TLS decryption capability will benefit from rules which are written specifically for HTTP traffic which has been decrypted.
In the future, Suricata Datasets will help reduce the number of rules and enable better IOC Based detection and other solutions are also being considered.
Context
In this context, focus will be placed on the scenario of inspecting outbound HTTP/TLS traffic from a user segment. As such example rules are used which look for âOutboundâ user traffic going to the general Internet.
A common deployment scenario for TLS Decryption is used to inspect network traffic originating from the Internet coming into an âinternet facingâ environment. This is common for inspecting incoming web requests for exploitation attempt of hosted services. Rules which are designed to inspect this traffic flow will not be discussed.
Itâs worth nothing that not all TLS decryption works the same. Some appliances/solutions will include TLS handshake and decrypted HTTP traffic, while others will only pass along the decrypted HTTP traffic. Understanding the traffic inspected by the IDS engine is key to understanding which rules apply and should be enabled within the IDS.
IOC Based Signatures
IOC based rules generally refers to rules which are written around a domain name. IOC based signatures are most common within the ruleset when information is provided by an external party, but there is not enough information to write a more specific signature.
Example 1: No Additional Traffic Available
As an example, consider the following tweet post from our good friend @Cyber0verload
This post contains a lists of domains that have been associated to the Gamaredon threat actor. At the time of the writing, a quick pivot on these domains did not reveal any publicly available samples which could be further examined.
In this scenario, we lack the information to write a rule based on the actual network traffic involving the domains. In an effort to still provide coverage for our customers and users, we opt to create detection of the domains themselves.
There are several different options for signatures that we can create on the domain itself, most options contains a dedicated buffer in Suricata:
- dns.query;
- tls.sni;
- http.host;
- tls.cert_subject (via the Common Name)
- subjectAltName (Feature #5234: SSL/TLS Sticky Buffer for subjectAltName - Suricata - Open Information Security Foundation)
It is the current standard practice of Emerging Threats to create rules for the dns.query and tls.sni.
Example 2: Decrypted Traffic Available
Next, lets consider a the post from @Suyog41 where a sample was available and we were able to create a signature for the decrypted network traffic.
In this case, using the provided sample, Emerging Threats was able to execute the malware within a sandbox environment and collect decrypted HTTP/2 communications to telegram.
As a side note, in this case, the TLS protocol is removed all together and is not observed by the IDS Engine. Only clear text HTTP/2 Traffic is presented.
As a result, the following rule was created, specific to this malware family which requires the use of TLS decryption. This dependency on TLS decryption is noted within the rule metadata via the Deployment metadata key which contains the value of SSLDecrypt.
Resulting Rule:
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE KWN Clipper Checkin via Telegram"; flow:established,to_server; http.method; content:"POST"; http.uri; content:"/bot"; startswith; content:"/sendMessage"; endswith; http.host; bsize:16; content:"api.telegram.org"; http.content_type; bsize:33; content:"application/x-www-form-urlencoded"; http.user_agent; content:"Go-http-client/"; startswith; http.request_body; content:"&text=FROM+UID+%3A+%0D%0A"; content:"%0D%0A+Victim+By+KWN++%0D%0ANEW+USER"; distance:0; fast_pattern; reference:url,twitter.com/suyog41/status/1645694318756220928; reference:md5,14485f6b7327d25d8a255b9feca41e7b; classtype:trojan-activity; sid:2044916; rev:1; metadata:attack_target Client_Endpoint, created_at 2023_04_11, deployment Perimeter, deployment SSLDecrypt, former_category MALWARE, performance_impact Low, signature_severity Major, updated_at 2023_04_11; target:src_ip;)
This is great for environments which have TLS decryption. However, users which lack this capability can still be alerted to the possible abuse of Telegram via the DNS and TLS rules. In this case, due to the use of a shared service (Telegram) an IOC based signature will not be as high fidelity as the MALWARE rule, but will have a long lifetime.
2033966 - ET HUNTING Telegram API Domain in DNS Lookup
2033967 - ET HUNTING Observed Telegram API Domain (api .telegram .org in TLS SNI)
Example 3: Unencrypted Traffic Available
This example is included just to round out the available âoutcomesâ as it relates to TLS. This sample shared by @fr0s7_ with supplemental information from @JAMESWT_MHT had clear text HTTP traffic.
In this case, a single rule was created and can be used by all customers regardless of having TLS decryption capabilities. As a result, there was no need for an IOC based signature, as we have rule that will detect this malware regardless of the domain the c2 server is hosted on. This creates a rule which has an increased longevity compared to an IOC based signature.
Resulting Rule:
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE LNK/Konni APT CnC Checkin (GET)"; flow:established,to_server; http.method; content:"GET"; http.uri; content:".php?vw="; fast_pattern; content:"&nv="; distance:3; within:4; pcre:"/(?:\x2ephp\x3fvw\x3d(?:[a-z0-9]{3})\x26(?:[a-z]{2})\x3d(?:[a-z0-9]{6}))$/"; http.user_agent; content:"WindowsPowerShell/"; reference:md5,6d6433c328f6cdce4a80efce3a29ea3e; reference:url,twitter.com/fr0s7_/status/1697506531724419277; classtype:trojan-activity; sid:2047883; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2023_09_01, deployment Perimeter, deployment SSLDecrypt, former_category MALWARE, confidence High, signature_severity Critical, tag Konni, tag c2, updated_at 2023_09_01, mitre_tactic_id TA0011, mitre_tactic_name Command_And_Control, mitre_technique_id T1071, mitre_technique_name Application_Layer_Protocol;)
Unique Considerations of IOC Based Signatures
Ruleset Lifecycle Considerations of IOC based Signatures
IOC based signatures have several unique considerations when writing them, the first and foremost is to consider the âlongevityâ of the indicator. This concept was touched on briefly as it related to the sample which used Telegram.
Some signatures, such as those for many online services which get abused by threat actors, such as Telegram, can have a fairly long life. Many of these rules are placed within the âINFOâ or âPOLICYâ category. Others signatures, such as malware c2 domains, are often short lived.
Rule writers have to balance the longevity of the rule when creating the rules with how large the ruleset itself is and how likely the threat actor is to be using/reuse the indicator.
Time To Review
To assist with rule lifecycle, Emerging Threats internally introduced a âTime To Reviewâ (TTR) process which applies to IOC based signatures. When creating an IOC based signature, the rule writer can prompt the rule to be reviewed in 30, 60, 90, etc days. When that TTR is expired, the rule is added to a list which rule writers then manually review and disable rules which have indicators that are no longer useful. Perhaps the indicator no longer resolves, the ownership of the domain has changed, the registar has placed the domain in a clientHold status, or the malware has moved to different c2, etc.
Along with this change, the âreviewed_atâ metadata field was created. This field allows users to see when the rules reviewed by Emerging Threats staff as part of the TTR process or for any other reason.
Ruleset âBloatâ
Another considerations specific to IOC based signatures is the impact of creating a large amount of rules has on the IDS engines, the management and lifecycle of the rules and the general âbloatâ of the ruleset. Every rule loaded into the engine creates overhead and impacts the performance of the engine.
If each IOC had every form of the rule created, there would be five different rules created for each IOC domain. When hundreds of domains are considered every week, the sheer amount of rules created would become unmanageable not only for ruleset users but also ruleset writers and the performance impact of loading so many rules into an IDS engine must be considered.
This is the major reason why only dns.query and tls.sni keyword based rules are created today. The inclusion of tls.sni rules was a recent change to the standard practice and was driven by customer request.
Limitations of DNS and TLS Signatures
It would be an injustice to discuss DNS and TLS signatures without discussing the serious limitations of them.
DNS: Sensor Placement
Depending on sensor âplacementâ, rules using the dns.query keyword can often alert on an recursive DNS resolvers making an DNS query on behalf of a user. This is very common in environments where sensors are placed at a network perimeter and internal hosts are not permitted to directly use external DNS servers.
This type of network environment and sensor placement results in increased analyst time correlating the request or not being able to identify the true requesting client at all. This was the driving factor behind including tls.sni based rules when creating dns.query rules. TLS traffic is less likely to be routed through a proxy server and thus has a higher likely of alerts containing the original requestor ip address. In cases where proxy servers are in use, special consideration must be taken to sensor placement.
Ideally sensor âplacementâ could be modified to see the original request directly from the requesting host or logging can be enabled on the recursive DNS server to assist in correlating an alert.
DNS: DoH/DoT
Recently there has been an increase in the adoption of RFC 8484 - DNS Queries over HTTPS (DoH)) and RFC 7858 - Specification for DNS over Transport Layer Security (TLS) (DoT). These protocols increase the privacy of internet users, but do impact the ability for IDS sensors to inspect DNS queries.
In the case of DNS over HTTPS, even when these streams are decrypted, the queries contained within are not parsed as standard DNS queries. There is currently a feature request to support this within Suricata
TLS: TLSv1.3 and eSNI
Similar to DoH/DoT there have been changes with the TLS protocols which increase the privacy of internet users but impact the ability for IDS sensors to inspect TLS traffic.
TLSv1.3 introduced changes which encrypt the server certificate details. This removes the effectiveness of rules using the tls.subject_cert and many other certificated based tls keywords. The other change is the adoption of a TLS Extension for Encrypted Server Name Identification. This extension enables the âServer Name Identificationâ field to be encrypted in the TLS handshake and removes the effectiveness of the tls.sni keyword.
TLS Protocol Inspection
There is one other type of rule that uses TLS information not yet mentioned. Rules which provide detection of vulnerabilities within the TLS protocol themselves. The most well known example of this might be âHeartBleedâ. In order to detect HeartBleed at the network level, the TLS exchange and other TLS protocol details must be observed by the engine. If a TLS decryption service/appliance is removing all the TLS protocol information, rules designed to inspect them will not be useful.
Long Term Solutions for IOC Based Signatures:
Datasets
Suricata 6.0 provides formal support for âDatasetsâ. There is a great presentation explaining the feature on Youtube: https://www.youtube.com/watch?v=dUUPwgHkuvo
As the Suricata 7.0 ruleset approaches release, it allows a unique oppurtunity to adopt move away from the typical 2+ rule format (DNS, TLS SNI, along with HTTP Host, TLS Cert, altDNSNames) that we provide today and instead start distributing our IOC based rules via datasets. This will allow one rule for each rule type per dataset to be written. As an example might have one dataset called âExternal IP Lookup Servicesâ, and 3 (or more) rules which use that dataset looking for the IOCs within the DNS, TLS SNI, and HTTP Host buffers. This not only provides a good way of managing several list, but reduces ruleset âbloatâ and enables our customers to select which datasets are of interest to their environment.
While Datasets are specific to Suricata, we will continue to support IOC based rules for Snort and supported versions of Suricata which do not support Dataset.
âDownstreamâ Inspection
Another possible solution is to remove the handling of IOCs from the IDS engine all together and instead use the same data in another system which processes the output of DNS, TLS, HTTP logging. Currently, this would rely upon the user to create this logic based on either Proofpointâs Domain Reputation lists or by extracting the IOC from IOC based rules.
Build Your Own Ruleset
In an ideal scenario, a solution could be designed that allows for users to select an output format for IOC based rules. If the user wants rules which leverage http.host only, then by adding a new and unique url to their ruleset manager, a rules file which contains all the IOC based rules formatted to inspect traffic using the http.host keyword.
This could be expanded to include tag filtering. As an example, if the user only wants MALWARE CnC domains, and domains listed in the âLiving Off Trusted Sites Projectâ and output in formats which utilize the dns.query, http.host and tls.sni keywords, thatâs what they get.
Even non-IDS engine formats could supported such as json, xml, datasets, etc.
This would represent a drastic change in how rules are produced within the Emerging Threats ruleset today, but is being considered.