Addressing HTTP/2 in Suri7

Background

HTTP/2 Traffic parsed by Suricata results in lowercase HTTP header names within many HTTP buffers. This is because the various RFCs for HTTP/2 - ([RFC9113 is the latest])(RFC 9113: HTTP/2) states the following (section 8.2 of RFC9113):

HTTP fields (Section 5 of [HTTP]) are conveyed by HTTP/2 in the HEADERS, CONTINUATION, and PUSH_PROMISE frames, compressed with HPACK [COMPRESSION].

Field names MUST be converted to lowercase when constructing an HTTP/2 message.

While the HTTP/1.1 RFC stated that http header names should be treated case insensitively, many browsers standardized on Pascal case aka “upper camel case”

While many browsers adopted Pascal case, it is very common to observe unique casing and even typos within malware families. These differences made for easy and efficient detection by IDS engines and are heavily used within the Emerging Threats Ruleset.

Example http.header_names

HTTP/1.1

REQ_HEADER_NAMES_DUMP, 30

00000000  0d 0a 48 6f 73 74 0d 0a 55 73 65 72 2d 41 67 65 |..Host..User-Age|
00000010  6e 74 0d 0a 41 63 63 65 70 74 0d 0a 0d 0a       |nt..Accept....  |

HTTP/2

00000000  0d 0a 3a 6d 65 74 68 6f 64 0d 0a 3a 70 61 74 68  |..:method..:path|
00000010  0d 0a 3a 73 63 68 65 6d 65 0d 0a 3a 61 75 74 68  |..:scheme..:auth|
00000020  6f 72 69 74 79 0d 0a 75 73 65 72 2d 61 67 65 6e  |ority..user-agen|
00000030  74 0d 0a 61 63 63 65 70 74 0d 0a 0d 0a           |t..accept....|

Example http.headers

HTTP/1.1

REQ_HEADERS_DUMP, 62

00000000  48 6f 73 74 3a 20 77 77 77 2e 67 6f 6f 67 6c 65 |Host: www.google|
00000010  2e 63 6f 6d 0d 0a 55 73 65 72 2d 41 67 65 6e 74 |.com..User-Agent|
00000020  3a 20 63 75 72 6c 2f 37 2e 38 31 2e 30 0d 0a 41 |: curl/7.81.0..A|
00000030  63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a       |ccept: */*....  |

HTTP/2

00000000  3a 6d 65 74 68 6f 64 3a 20 47 45 54 0d 0a 3a 70  |:method: GET..:p|
00000010  61 74 68 3a 20 2f 0d 0a 3a 73 63 68 65 6d 65 3a  |ath: /..:scheme:|
00000020  20 68 74 74 70 73 0d 0a 3a 61 75 74 68 6f 72 69  | https..:authori|
00000030  74 79 3a 20 77 77 77 2e 67 6f 6f 67 6c 65 2e 63  |ty: www.google.c|
00000040  6f 6d 0d 0a 75 73 65 72 2d 61 67 65 6e 74 3a 20  |om..user-agent: |
00000050  63 75 72 6c 2f 37 2e 38 31 2e 30 0d 0a 61 63 63  |curl/7.81.0..acc|
00000060  65 70 74 3a 20 2a 2f 2a 0d 0a                    |ept: */*..|

HTTP/2 pseudo-header fields

HTTP/2 uses special pseudo-header fields beginning with a ‘:’ character (ASCII 0x3a) to convey message control data (see Section 6.2 of [HTTP]).

There are four special “pseudo-header” fields defined within HTTP/2

  • :method
  • :scheme
  • :authority
  • :path

All HTTP/2 requests MUST include exactly one valid value for the ":method ", ":scheme ", and ":path " pseudo-header fields, unless they are CONNECT requests (Section 8.5).

:authority and Host

Based on the example HTTP/2 Header data above, the lack of a Host header can be observed. HTTP/1.x has no concept of an :authority header, and as such, many rule have been written that leverage the Host header. Consider the following rule:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE Atomic MacOS Stealer CnC Exfil (POST)"; flow:established,to_server; urilen:8; http.method; content:"POST"; http.uri; content:"/sendlog"; fast_pattern; http.header_names; content:"|0d 0a|Host|0d 0a|Content-Type|0d 0a|Content-Length|0d 0a 0d 0a|"; bsize:40; http.content_type; content:"application|2f|x|2d|www|2d|form|2d|urlencoded"; bsize:33; http.host; pcre:"/^(?:[0-9]{1,3}\.){3}[0-9]{1,3}$/"; reference:url,twitter.com/x3ph1/status/1703492680951509154; classtype:trojan-activity; sid:2048103; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2023_09_18, deployment Perimeter, former_category MALWARE, confidence Medium, signature_severity Major, updated_at 2023_09_18, reviewed_at 2023_09_18;)

This rule depends on the Host header appear, however, it is very common within HTTP/2 traffic, that the Host header is absent and “supplanted” by the :authority pseudo-header. This can create issues with many existing IDS rules.

Should this malware support HTTP/2, the above rule would likely not alert because there is not “Host” header.

The :authority pseudo-header and Host headers can represent the same data, and both can be found in HTTP/2 traffic, specifically when HTTP/1.x traffic has been captured by an “intermediary” host and used to generate a HTTP/2 request.

When an :authority pseudo-header is missing the Host header MUST be present. When both :authority pseudo-header and Host header are present, the entities must match.

Suricata’s handling of :authority and Host

When Suricata parses HTTP/2 traffic, the :authority pseudo header is used to populate the http.host buffer.
Reference: http2: http.host.raw keyword now works for HTTP2 · OISF/suricata@df03955 · GitHub
Reference: http2: http.host normalized keyword now works for HTTP2 · OISF/suricata@547e9f4 · GitHub

If you want to dig into the details of :authority, Host headers, and userinfo: expand the details, otherwise, accept the following:

If there is no :authority header, there MUST be a Host header, and it SHOULD be the first field.

With this information we can say that HTTP/2 traffic will contain either :authority or Host or both (Both are most likely when there is an “interception proxy” that has to deal with HTTP/2 and HTTP/1.1 traffic.

The Details

First, a review of what :authority means in reference to HTTP/2. RFC9113 defines it as

The ":authority " pseudo-header field conveys the authority portion (Section 3.2 of [RFC3986]) of the target URI (Section 7.1 of [HTTP]).

Off to RFC3986 - Section 3.2

The authority component is preceded by a double slash (“//”) and is terminated by the next slash (“/”), question mark (“?”), or number sign (“#”) character, or by the end of the URI.

  authority   = [ userinfo "@" ] host [ ":" port ]

So the :authority pseudo-header can include a bunch of details. A “Host” is a subset of the :authority

An intermediary that needs to generate a Host header field (which might be necessary to construct an HTTP/1.1 request) MUST use the value from the ":authority " pseudo-header field as the value of the Host field,

Clients that generate HTTP/2 requests directly MUST use the “:authority” pseudo-header field to convey authority information, unless there is no authority information to convey (in which case it MUST NOT generate “:authority”).

So… It MUST include :authority unless it doesn’t have any(?).

It continues

Clients MUST NOT generate a request with a Host header field that differs from the ":authority " pseudo-header field. A server SHOULD treat a request as malformed if it contains a Host header field that identifies an entity that differs from the entity in the ":authority " pseudo-header field.

An intermediary that needs to generate a Host header field (which might be necessary to construct an HTTP/1.1 request) MUST use the value from the ":authority " pseudo-header field as the value of the Host field

So HTTP/2 can, though isn’t required to have an :authority header and can optionally have a Host header. When both are present they should match, otherwise be treated as “malformed”, it’s most likely that a Host header field might be the result of a proxy translating between HTTP/1 and HTTP/2

RFC9110 - HTTP Semantics referenced adds a bit of clarity

7.2. Host and :authority

The “Host” header field in a request provides the host and port information from the target URI, enabling the origin server to distinguish among resources while servicing requests for multiple host names.

In HTTP/2 [HTTP/2] and HTTP/3 [HTTP/3], the Host header field is, in some cases, supplanted by the “:authority” pseudo-header field of a request’s control data.

Host = uri-host [ “:” port ] ; Section 4

The target URI’s authority information is critical for handling a request. A user agent MUST generate a Host header field in a request unless it sends that information as an “:authority” pseudo-header field. A user agent that sends Host SHOULD send it as the first field in the header section of a request.

For example, a GET request to the origin server for http://www.example.org/pub/WWW/ would begin with:

GET /pub/WWW/ HTTP/1.1
Host: www.example.org

Since the host and port information acts as an application-level routing mechanism, it is a frequent target for malware seeking to poison a shared cache or redirect a request to an unintended server. An interception proxy is particularly vulnerable if it relies on the host and port information for redirecting requests to internal servers, or for use as a cache key in a shared cache, without first verifying that the intercepted connection is targeting a valid IP address for that host.

This statement is critical

A user agent MUST generate a Host header field in a request unless it sends that information as an “:authority” pseudo-header field. A user agent that sends Host SHOULD send it as the first field in the header section of a request.

So if there is no :authority header, there MUST be a Host header, and it SHOULD be the first field.

With this information we can say that HTTP/2 traffic will contain either :authority or Host or both (Both are most likely when there is an “interception proxy” that has to deal with HTTP/2 and HTTP/1.1 traffic.

Userinfo Deep Dive

What happens with the :authority pseduo header contains userinfo?

The authority header can contain the following information
authority = [ userinfo "@" ] host [ ":" port ]

However, RFC 9113 also states

":authority " MUST NOT include the deprecated userinfo subcomponent for "http " or "https " schemed URIs.

which is a reference to RFC 9110: HTTP Semantics

The URI generic syntax for authority also includes a userinfo subcomponent ([URI], Section 3.2.1) for including user authentication information in the URI. In that subcomponent, the use of the format “user:password” is deprecated.

It would appear that in the normalized http.host populated from the :authority header, which includes userinfo, only the username value from the userinfo is parsed into the http.host buffer. After testing
requests #6426 and #6479 were created.

HPACK

HPACK is a new compression method used by HTTP/2 for headers. Cloudflare has a great explanation of how HPACK works. The key takeaway for IDS rule writers, is that we are fully dependent on the IDS engine to decompress and parse HTTP/2 headers.


Problem Statement #1:

FPs due to Mixed cased headers due to “intermediary host” changing HTTP/2 → HTTP/1

Example Traffic:

GET /catalog/123 HTTP/1.1
Host: foo.bar
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="101", "Google Chrome";v="101" 
sec-ch-ua-mobile: ?0
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36
sec-ch-ua-platform: "macOS" 
accept: */*
sec-fetch-site: cross-site
sec-fetch-mode: no-cors
sec-fetch-dest: script
referer: https://example.com/
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
Cache-Control: max-stale=0
Connection: Keep-Alive

Example Rule

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE Win32/Zemot URI Struct"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"/catalog/"; fast_pattern; pcre:"/\/catalog\/\d{3,}$/"; http.header; content:!"nap.edu|0d 0a|"; http.header_names; content:!"Accept-"; content:!"Referer|0d 0a|"; reference:md5,b8e0b97c8e9faa6e5daa8f0cac845516; classtype:trojan-activity; sid:2019458; rev:5; metadata:created_at 2014_10_17, updated_at 2020_10_07;)

Analysis:

SID 2019458 contains the logic:
content:!"Accept-"; content:!"Referer|0d 0a|";. however, because this traffic has been converted from HTTP/2, the original request includes a lowercase referer, accept-encoding, and accept-language headers.

Solution

Interm

  1. An additional content negation containing the lowercase version of all header name negations
  2. add a nocase option to all header name negations.

Long Term Proposed

A feature request for Suricata has been created to support case insensitive testing of HTTP header name existence - #6290

Problem Statement #2:

FPs due to Pascal Cased http header names in content negations on HTTP/2 traffic

This is a minor variation of Problem Statement #1, in this case however, the requested traffic is HTTP/2 without any “intermediary host”.

Example Traffic (wireshark output):

Stream: HEADERS, Stream ID: 15, Length 359, GET /uploads/images/2014/05/85c8a9dd9d86fac8dc277f0bdb5c6649.png
    Length: 359
    Type: HEADERS (1)
    Flags: 0x25, Priority, End Headers, End Stream
    0... .... .... .... .... .... .... .... = Reserved: 0x0
    .000 0000 0000 0000 0000 0000 0000 1111 = Stream Identifier: 15
    [Pad Length: 0]
    0... .... .... .... .... .... .... .... = Exclusive: False
    .000 0000 0000 0000 0000 0000 0000 1101 = Stream Dependency: 13
    Weight: 41
    [Weight real: 42]
    Header Block Fragment: 8205ab62dae838e44306a473150c0801698036c3cd91e1bf248fc8f3928c8f48413aeca0…
    [Header Length: 648]
    [Header Count: 16]
    Header: :method: GET
    Header: :path: /uploads/images/2014/05/85c8a9dd9d86fac8dc277f0bdb5c6649.png
    Header: :authority: s1.hostingkartinok.com
    Header: :scheme: https
    Header: user-agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0
    Header: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
    Header: accept-language: en-CA,en-US;q=0.7,en;q=0.3
    Header: accept-encoding: gzip, deflate, br
    Header: upgrade-insecure-requests: 1
    Header: sec-fetch-dest: document
    Header: sec-fetch-mode: navigate
    Header: sec-fetch-site: none
    Header: sec-fetch-user: ?1
    Header: pragma: no-cache
    Header: cache-control: no-cache
    Header: te: trailers

Example Rule:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE Steam Stealer"; flow:to_server,established; http.method; content:"GET"; http.uri; content:"/uploads/images/201"; fast_pattern; pcre:"/\.png$/"; http.header_names; content:!"Referer|0d 0a|"; content:!"Accept"; content:!"User-Agent|0d 0a|"; reference:md5,5f50e810668942e8d694faeabab08260; reference:url,blog.0x3a.com/post/107195908164/analysis-of-steam-stealers-and-the-steam-stealer; classtype:trojan-activity; sid:2020095; rev:5; metadata:created_at 2015_01_06, updated_at 2020_09_29;)

Analysis:

Sid 2020095 contains the following logic:
http.header_names; content:!"Referer|0d 0a|"; content:!"Accept"; content:!"User-Agent|0d 0a|";
However because the HTTP Request was made with “native” HTTP/2, all header names are lowercase. Because the content does not match, the rule alerts.

Solution

Interm

  1. An additional content negation containing the lowercase version of all header name negations
  2. add a nocase option to all header name negations.

Long Term Proposed

A feature request for Suricata has been created to support case insensitive testing of HTTP header name existence - #6290

Problem Statement #3:

FNs for malware that makes opportunistic use of HTTP/2 due to use of Pascal Cased HTTP header names in positive content matches

Example Traffic:

HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 92, GET /listfolder?path=/
        Length: 92
        Type: HEADERS (1)
        Flags: 0x05, End Headers, End Stream
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment: 82048e628321329e890b67f958d33c0c7f8741882f91d35d055c87a77a87c473cd41633a…
        [Header Length: 223]
        [Header Count: 7]
        Header: :method: GET
        Header: :path: /listfolder?path=/
        Header: :scheme: https
        Header: :authority: example.com
        Header: user-agent: Googlebot/
        Header: accept: */*
        Header: authorization: Bearer MznTtmeJEo6CdyMX1QZE/2wFslDv3cYXsXSHCsg5J8UxP1pqf8IG

Example Rule:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE Win32/RokRat CnC Activity (GET)"; flow:established,to_server; http.method; content:"GET"; http.uri; content:"|2f|listfolder|3f|path|3d 2f|"; bsize:18; fast_pattern; http.user_agent; content:"Googlebot"; http.header; content:"Authorization|3a 20|Bearer"; reference:url,research.checkpoint.com/2023/chain-reaction-rokrats-missing-link; classtype:trojan-activity; sid:2045277; rev:1; metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit, attack_target Client_Endpoint, created_at 2023_05_01, deployment Perimeter, deployment SSLDecrypt, former_category MALWARE, malware_family Rokrat, confidence High, signature_severity Major, updated_at 2023_05_01;)

Analysis:

SID 2045277 contains the following logic:
http.header; content:"Authorization|3a 20|Bearer";
However, because the traffic is HTTP/2, the Authorization header is lowercase. This results in a FN of the rule, which is inspecting for Pascal cased HTTP header names.

Solution

Interm

  1. add a simple nocase to any http header name regardless of the buffer (http.header_names, http.header, http.start, buffered, etc) used
  2. rewrite the logic so the header name has a nocase, but the value does not (as shown below).
    http.header; content:"Authorization"; nocase; content:"|3a 20|Bearer"; within:8

Long Term Proposed

Feature request has been created to support http.headers - dynamic sticky buffers - #5775. This would allow a header to be selected in a case insensitive manner, and allow the buffer to be populated with just the value of the header.

http.header:authorization; content:"Bearer"; startswith;

Problem Statement #4:

FNs for malware that uses the :authority pseudo-header and does not include a host header. This mostly impacts the use of the Host header name within http.header_names but could also include Host header within the http.header buffer

Example Traffic

None

Example Rule

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE DirtJumper Activity"; flow:established,to_server; threshold: type limit, track by_src, seconds 60, count 1; http.method; content:"POST"; http.header_names; content:!"Referer|0d 0a|"; content:"|0d 0a|Host|0d 0a|"; startswith; http.request_body; content:"&req="; pcre:"/^\d+?=\d+?(?:&ver=\d+?)?&req=\d+?(?:&r=)?$/"; reference:md5,5474129345d9756649c871f9c8b46287; reference:md5,ff5608e00d5e6e81af9c993461479e43; classtype:trojan-activity; sid:2018094; rev:3; metadata:created_at 2014_02_07, updated_at 2020_04_27;)

Analysis

Sid 2029237 uses the http.header_names buffer to include logic requiring that the Host header is the first header in the HTTP Request. However, should this malware (or the underlying HTTP library) support HTTP/2, this request would not contain a Host header and instead only contain the :authority pseudo-header.

Solutions / Considerations

There is no “good” solution here. Without a solution from the IDS Engine, this rule simply won’t work for HTTP/2 traffic. In order to avoid performance impact of being inspected against HTTP/2 traffic, this rule should be rewritten to use the http1 protocol.

If HTTP/2 support for this rule be desired, the detection logic of the Host header should be removed.

The removal of detection logic using the Host header will not be possible in all cases without significant impact to the rule’s fidelity and performance. An example of this is the following rule, where the Host header is used in combination with other header to not only enforce the ordering, but also create a better a stronger fast_pattern.

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET MALWARE Dridex/Bugat/Feodo POST Checkin"; flow:established,to_server; http.method; content:"POST"; http.header_names; content:!"User-Agent|0d 0a|"; content:!"Accept"; content:!"Connection|0d 0a|"; content:!"Referer|0d 0a|"; http.header; content:"Content-Length|3a 20|59|0d 0a|Host|3a 20|"; depth:26; http.host; pcre:"/^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}(?:\x3a\d{1,5})?$/"; http.uri; pcre:"/^\/[\/a-z0-9]+$/i"; reference:md5,2ddb6cb347eb7939545a1801c72f1f3f; classtype:command-and-control; sid:2018771; rev:6; metadata:created_at 2014_07_24, former_category MALWARE, updated_at 2020_05_01;)

Keyword nuances impacting HTTP/2 traffic

http.msg
- blank in http2
http.start
- does not exist for HTTP/2
http.request_line
<METHOD> <URI> HTTP/2\r\n

http.header
- includes the pseudo-headers and their values
http.header_names
- includes the pseudo-headers

HTTP/2 protocol

the http2 protocol is introduce for signatures that should only use HTTP/2.

New HTTP/1 and HTTP/2 Keywords

http.request_header;
http.response_header;

both of these keywords allow the inspection of a single header or pseudo-header (for HTTP/2). it includes the header name and value. These keywords differ from http.header in that http.header contains all http header names and values, where as http.reqeust_header contains each http header name and value by itself.

http.request_header; content:"X-CSRF-Token|3a 20|"; startswith; nocase; content:"en-us";

if you want to inspect multiple headers, you can declare the keyword multiple times.

http.request_header; content:":path|3a 20|"; startwith; content:".php"; endswith; http.request_header; content:"X-CSRF-Token|3a 20|"; startswith; nocase; content:"en-us";

OISF Tickets

Suricata feature requests to assist in this effort:

  1. Addressing Mixed Case in HTTP Headers Names and HTTP2 - #5774:
    Contains a general discussion of the issue as presented by an ET customer on rules that are suffering from FPs due to the lowercase http header names in HTTP/2.

  2. http.headers - dynamic sticky buffers - #5775
    Request for dynamic sticky buffers based on arbitrary HTTP headers names, which are case insensitive.
    http.header:x-custom-user-agent; content:"foobar"; startswith;

  3. warn when HTTP rules will only work for a specific version of HTTP - #5973
    Request to warn if a rule, due to keywords/buffers/protocols used, is only effective against HTTP/1. (http.start, http.stat_msg, etc)

  4. support case insensitive testing of HTTP header name existence - #6290
    Instead of having to http.header_names; content:"|0d 0a|User-Agent|0d 0a|"; nocase; use a natively case insensitive method.

    As taken from snort3: http_header_test:field user-agent,absent;

  5. HTTP/2 - http.host behavior when both :authority pseudo header and host header are present
    Currently it appears that only the :authority header makes it into http.host.

  6. HTTP/2 - new app-layer-event when :authority and host headers do not match
    the RFC states these two values should match, an app-layer-event is a handy way to detect this anomaly

  7. HTTP/2 - app-layer-event and normalization when userinfo is in the :authority pseudo header for the http.host header
    HTTP/2 :authority shouldn’t have userinfo in it. But if it is present, it currently makes it way into the http.host buffer. This request is to create an event and remove the user info when creating the http.host buffer.

  8. HTTP/2 - when userinfo is in the :authority pseudo header it breaks http.host
    This issue specifically requests that for HTTP/2 the userinfo be removed from the http.host buffer

  9. Lua support for HTTP/2
    Currently appears that the optional Suricata Lua support does not get any HTTP/2 traffic sent to it.

  10. transformation - strip_pseudo_headers
    A new transformation that will remove pseudo headers and their values from the applied buffer.

2 Likes