Skip to content

DRILL-8556: Various PCAP and PCAP-NG Improvements - #3087

Open
cgivre wants to merge 61 commits into
apache:masterfrom
cgivre:pcap-improvements
Open

cgivre wants to merge 61 commits into
apache:masterfrom
cgivre:pcap-improvements

Conversation

@cgivre

@cgivre cgivre commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

DRILL-8556: Various PCAP and PCAP-NG Improvements

Description

The PCAP-NG reader read the whole file into memory (twice: the raw bytes plus the third-party library's list of every parsed block), so large captures needed several times their size in heap. This PR streams PCAP-NG files block by block, removes the pcapngdecoder dependency, and fixes a set of decoding bugs in both the PCAP and PCAP-NG readers.

PCAP-NG reader

  • Streams blocks one at a time; memory no longer grows with file size.
  • Tracks interfaces per section: timestamps honor if_tsresol and if_tsoffset (nanosecond captures were off by 1000x), and packets are decoded for their interface's link type.
  • Decodes Ethernet (including 802.1Q/QinQ tags), raw IP, BSD loopback, PPP, Linux SLL/SLL2, and 802.11 / radiotap data frames. Unsupported link types return null IP fields instead of misread ones.
  • Skips unknown block types (previously it stopped silently at the first one) and tolerates a truncated final block.
  • Decodes each packet once per row instead of once per column.
  • Supports sessionizeTCPStreams.

Packet decoding (both readers)

  • The TCP header length was read from the wrong byte and then over-counted, and payloads included link-layer headers, Ethernet padding, and bytes past the captured length.
  • IPv6: extension headers were walked incorrectly and TCP fields ignored the 40-byte IPv6 header, so IPv6 TCP sequence numbers, flags, and ports were misread.
  • ARP_PROTOCOL was defined as IP protocol 0 (IPv6 hop-by-hop), so IPv6 multicast packets were labeled ARP and real ARP frames were not. ARP is now detected by EtherType; ICMPv6 is labeled ICMP.

Format plugin

  • The file format was a static field shared by all readers and was detected from the first file in the scan, so directories mixing .pcap and .pcapng sent files to the wrong reader.

TCP sessions (both readers)

  • One shared TcpSessionizer; only TCP packets are tracked (all non-TCP packets previously accumulated in a "session 0" that was never written).
  • Sessions still open at end of file are returned with a new session_closed column instead of dropped. Packets trailing a closed session no longer start a phantom session. connection_time is null when the handshake was not captured.

Protocol decoders

  • Pluggable decoders for application protocols, discovered with ServiceLoader: PacketProtocolDecoder for single packets and SessionProtocolDecoder for reassembled TCP sessions. A decoder only handles traffic that parses as its protocol. Design and decoder-writing guide: docs/dev/PcapProtocolDecoders.md.
  • New columns in both readers: parsed_protocol, parsed_data (one map per decoder) and decode_error. Decoding runs only when these columns are queried.
  • Packet decoders: DNS (including mDNS and LLMNR), HTTP/1.x, TLS ClientHello/ServerHello (SNI, ALPN, JA3, JA3S, JA4), DHCP, DHCPv6, ICMP and ICMPv6, ARP, NTP, Syslog, SSDP, SIP, TFTP, NetBIOS name service, RADIUS, SNMP (v1, v2c, v3), STUN, Kerberos, and QUIC v1 Initial (the ClientHello is decrypted with the initial keys derived from public values per RFC 9001: SNI, ALPN, JA4).
  • Session decoders: HTTP/1.x (keep-alive and chunked bodies handled), TLS handshake (negotiated version and cipher, certificate subject, issuer, validity, SANs), FTP control channel, SSH (identification and HASSH), DNS over TCP, SMTP, POP3 and IMAP (credentials, envelope and message headers), SMB2/3 (dialect, signing, and the authenticating identity from the NTLMSSP blob via a shared NTLMSSP parser), LDAP (bind DNs, result, search filters), Telnet, RDP (the mstshash cookie and requested security protocols) and MQTT.
  • Authentication protocols output identity metadata only. SMB/NTLM surfaces the account, domain and workstation but never the NTLM challenge, responses or session key, with or without exposeCredentials, since those are offline-crackable material rather than a password; Kerberos records only the encryption types, never the encrypted parts. Only JA4 is implemented, not the JA4S/JA4H/JA4SSH variants, whose license is not compatible with Apache.
  • Every decoder field can be queried without quotes; TestDecoderFieldNames checks this for every registered decoder.
  • Errors never stop a file from being read: malformed packets keep their readable fields and explain the problem in decode_error, and damaged or non-capture files return a row with only decode_error set instead of failing the query.
  • New format option exposeCredentials (default false): cleartext passwords found by decoders are only returned when it is set; usernames and a password_present flag are always returned.

Drill scan framework fixes (exec/java-exec), found while implementing the decoders:

  • OutputBatchBuilder rebuilt map arrays nested in maps as plain maps, dropping their offsets, so such batches failed validation.
  • ScanProjectionParser counted array dimensions from the root of the path, so projecting an element of an array inside a map (m.a[0]) failed as incompatible.
  • ResultSetLoaderImpl could fail a query while harvesting a full batch: filling the many unwritten decoder columns asked for more memory than the batch budget allowed. Those fills are now allowed while harvesting.

Read reliability (classic PCAP)

  • The reader could not read gzip-compressed files: a refill overwrote unread bytes, and packets kept for sessions pointed into the reused buffer.
  • A read failure part way through a file keeps the packets already read and adds an error row.

Documentation

Breaking change: the PCAP-NG timestamp column is renamed packet_timestamp, the name PCAP files already use. timestamp is a reserved word, so it had to be quoted in every query and an unquoted MIN(timestamp) was a parse error.

New PCAP-NG packet columns: captured_length, interface_id, interface_name, link_type, comment, direction, reception_type, fcs_length, drop_count, packet_hash. New session column: session_closed. In stat mode (stat: true), options a block does not have are now null (previously "" or -1), a comment column is added, and if_speed / if_tsoffset are BIGINT. The plugin README documents the columns and supported link types.

User-visible value changes from the bug fixes: classic PCAP data payloads, IPv6 TCP fields, type for ARP/ICMPv6, and is_corrupt (testv1.pcap reported 16 corrupt packets that were snaplen-truncated, not corrupt).

Testing

  • Replaced the corrupted todo/ fixtures (they had been run through a UTF-8 conversion) with valid Wireshark captures, and added generated fixtures covering link types, timestamp resolutions, packet options, multiple sections, encapsulations and sessionization. Fixtures were cross-checked with scapy.

  • Every classic PCAP payload in the test fixtures was compared against scapy (6,736 exact matches; 17 encrypted payloads differ only in how invalid UTF-8 renders). Test expectations that encoded the old bugs were corrected after verifying the new values against scapy.

  • Protocol decoders: unit tests for the registry, TCP stream reassembly, DNS and HTTP parsers (including compression-pointer loops, record and header caps, keep-alive pairing with chunked bodies), and query tests on generated fixtures for decoder columns and every error-handling case. Expected values were cross-checked with scapy.

  • Every decoder has unit tests and a query test on a generated fixture holding valid traffic, other traffic on the same port (left undecoded) and a malformed message (reported in decode_error). Fixtures are cross-checked with scapy where scapy supports the protocol (DNS, DHCP, NTP, SNMP, TLS, SSH KEXINIT and others); HASSH and JA3 values are computed independently.

  • The format-pcapng module: 332 tests pass, plus checkstyle and the license check.

  • Scan framework fixes: regression tests TestOutputBatchBuilder.testMapArrayInMap, TestProjectionParser.testArrayInMap and TestResultSetLoaderLimits.testHarvestFillsUnwrittenColumnsOverBudget; 501 scan/result-set-loader/projection tests and 246 JSON reader tests pass.

cgivre added 2 commits October 2, 2026 14:18
PCAP-NG reader:
- Stream blocks one at a time instead of reading the whole file into
  memory, and parse them directly, removing the pcapngdecoder dependency
- Track interfaces per section: timestamps honor if_tsresol and
  if_tsoffset, and each packet is decoded for its interface's link type
- Decode Ethernet (with 802.1Q/QinQ tags), raw IP, BSD loopback, PPP,
  Linux SLL/SLL2, and 802.11 / radiotap data frames
- Add interface_id, interface_name, link_type, captured_length,
  comment, direction, reception_type, fcs_length, drop_count and
  packet_hash columns
- Skip unknown block types instead of stopping, and tolerate a
  truncated final block
- Decode each packet once per row instead of once per column
- Stat queries: return null for missing options, add comment, make
  if_speed and if_tsoffset BIGINT, and use the interface's resolution
  for statistics timestamps
- Support sessionizeTCPStreams

Packet decoding (both readers):
- Fix the TCP header length and payload offsets, bound payloads by the
  IP length and captured length, and walk IPv6 extension headers
  correctly; IPv6 TCP sequence numbers, flags and ports were misread
- Detect ARP by EtherType and label ICMPv6 as ICMP; IPv6 packets with a
  hop-by-hop header were labeled ARP

Format plugin:
- Detect PCAP vs PCAP-NG per file; the format was a static field and was
  detected from the first file in the scan

TCP sessions (both readers):
- Share one TcpSessionizer; only TCP packets are tracked
- Return sessions still open at end of file, with a new session_closed
  column, and ignore packets trailing a closed session
- connection_time is null when the handshake was not captured

Tests: replace the corrupted todo/ fixtures with valid captures, add
fixtures for metadata, link types and sessionization, and correct
expectations that encoded the old decoding bugs.
@cgivre cgivre self-assigned this Oct 2, 2026
@cgivre cgivre added enhancement PRs that add a new functionality to Drill code-cleanup refactoring PR related to code refactoring doc-impacting PRs that affect the documentation dependencies labels Oct 2, 2026
@cgivre
cgivre marked this pull request as ready for review October 2, 2026 19:15
cgivre added 9 commits October 2, 2026 15:19
A leftover probe test and the three ServiceLoader registration files had no
license header, which failed the Apache RAT check.