In 2024, it’s not a stretch to say that just about everyone is on the internet in some shape or fashion every single day. From critical business functions to social media to streaming video, the world has become increasingly interconnected and all of this activity is broken down into raw data traversing national telecommunications networks. It’s not a stretch of the imagination to think hackers and cyber criminals are constantly attempting to hack this data and disrupt valuable daily activity, even at a national scale.
In fact, cyber attacks are becoming more complex and constantly targeting ISPs and regional telco networks across the globe. These networks are often associated with vital operations to society, from government agencies to critical infrastructure. When it comes to cyber, government defense organizations tasked with protecting its citizens are forced to balance consumer privacy with national defense. On this front, Internet Connection Records (ICR) have become a vital tool to giving government security teams the network visibility solution required to mount a proper cyber defense.
The Investigatory Powers Act of 2016
The term Internet Connection Records gained notoriety out of the Investigatory Powers Act which passed in 2016 in the UK. The act provided a new framework to govern the use and oversight of investigatory powers by law enforcement and the security and intelligence agencies. Specifically, the act expanded on the surveillance powers of the UK intelligence agencies and police by allowing records (ICRs) to be kept detailing all domestic internet activity. The goal is to provide law enforcement the ability to use ICRs to help solve domestic crimes.
Meanwhile, there’s been a growing public concern about what exactly is being done behind closed doors with these Internet Connection Records. The who, what, when, where and why of the mass data retention has been unclear and the test trials to collect this data has moved along slowly and in secrecy.
What are Internet Connection Records?
The phrase Internet Connection Records (ICRs) coined in the 2016 act are the details of a data flow rather than a user’s full blown browsing history. It’s metadata; it gives deeper layers of detail on a flow of online communication. Here are a few examples how network traffic metadata can be collected and classified as an ICR.
- NetFlow – NetFlow is a network protocol system that was created by Cisco. It has been used as a network traffic analyzer to determine a flow’s point of origin, destination, volume and paths on the network. This data helps network monitoring, planning and forensics.
- IPFIX – Internet Protocol Flow Information eXport (IPFIX) is version 9 of NetFlow. The IPFIX protocol uses very similar procedures as NetFlow for exporting network traffic data, however, IPFIX was designed with additional extensibility and is considered the upgraded version of the protocol.
- Zeek – Formerly known as Bro, Zeek is a passive, open-source network traffic analyzer tool used by many operators. It analyzes network traffic packets and creates “Zeek logs” which can be used to detect malicious activity within a network.
These are all forms of metadata summarizing IP network traffic flows. An IP flow consists of a group of packets that make up a network conversation. As a packet is forwarded within a router or switch, it is examined for a set of attributes, including IP source address, IP destination address, source port, destination port, Layer-3 protocol type, class of service and router or switch interface. All of this information is included in a basic Internet Connection Record along with timestamp and flow-based statistics.
History of Internet Connection Records
While the term may be relatively fresh, the concept of Internet Connection Records certainly are not. To understand ICRs, its useful to understand NetFlow and its influence on ICRs.
NetFlow was developed in 1995 by Cisco. It quickly gained popularity in the cybersecurity community and its usefulness eventually transformed how security teams mitigate threats. Originally developed to be a packet switching technology for Cisco routers, NetFlow morphed into a tool for monitoring network performance and traffic.
Before NetFlow, Simple Network Management Protocol (SNMP) was the standard protocol for network monitoring. As the name suggests, the use of SNMP was and still is a simple yet effective monitoring technique but the creation of NetFlow gave way to more detailed and in-depth traffic analysis.
Cisco regularly updated NetFlow, with their NetFlow v5 being one of the most widely used and the newer IPFIX protocol (v9) being an Internet Engineering Task Force (IETF) standard which opened the door to content-enriched metadata. As NetFlow/IPFIX evolved, the use of metadata and specifically enriched metadata (inclusion of deeper information for each network flow) has become an increasingly vital piece to network security and cyber threat hunting applications. Internet connection records provide cyber defense teams the visibility required to identify indicators of compromise on their network as well as alert on anomalous network activity.
Cyber Threat Hunting With ICRs
ICRs are valuable to government defense agencies because they provide more insight to an already heavily encrypted cyberspace. We have seen bad actors utilize increasingly complicated methods to hide within communication flows to embed themselves in networks’ attack surfaces for extended periods of time. ICRs can be instrumental in tracking cyber threats across large-scale networks because they provide metadata that allows security teams to identify suspicious behavior patterns without needing to analyze the content of communications. Here’s how ICRs contribute to cyber threat hunting:
- Anomaly Detection: ICRs can reveal unusual connection patterns, such as frequent connections to known malicious domains, unexpected international connections, or access to high-risk services, which could indicate command-and-control (C2) communication or unauthorized data exfiltration.
- Tracing Malicious Activity: By tracking connection records, analysts can trace the history of compromised devices to map out infection points and lateral movement across the network. This enables security teams to understand how a threat may have spread within a network, helping to contain it more effectively.
- Attribution and Identification of Threat Actors: ICRs that include device identifiers and connection timestamps help link specific actions to devices or users, which can be crucial in attributing suspicious activities to potential threat actors, especially in cases of insider threats or compromised accounts.
- Real-Time Threat Intelligence: Aggregating ICRs enables the rapid cross-referencing of activity against threat intelligence feeds to detect if users are connecting to known malicious IPs or domains, allowing proactive blocking or response before threats escalate.
- Building Behavioral Profiles: By monitoring ICR data over time, security teams can develop baseline behaviors for users, devices, and applications. This is particularly helpful for identifying deviations indicative of cyber threats, such as unusual access times, bandwidth spikes, or repeated access attempts to restricted areas.
In large-scale networks, where full packet inspection can be resource-intensive, ICRs offer an efficient and privacy-aware method to monitor network traffic, enabling the detection of early signs of malicious activity without straining monitoring systems.
NetQuest Sensors Generate Internet Connection Records
NetQuest’s Streaming Network Sensors (SNS) generate Internet Connection Records at telco scale optimizing visibility for government defense and telco security applications. NetQuest sensors generate unsampled (1:1 sampling) ICR metadata (exported via IPFIX or Kafka) across many 100G and 10G network links providing 100% visibility into network flows. In addition to standard NetFlow records, the SNS provides content-enriched metadata that includes layer 7 application classification and protocol-specific metadata for DNS, TLS, SSL, SSH, QUIC, HTTP, BGP, and other protocols.
The SNS also provides a unique level of ICR visibility for matching live network flows with indicators of compromise (IoC) using Encrypted Traffic Analysis. This includes matching JA3 fingerprints, exposing TLS handshake information, and other information used in modern threat hunting tactics.
Generating Internet Connection Records at a national scale is a challenging task for government defense agencies and telco security teams. NetQuest’s SNS sensors generate ICRs at terabit scale in a compact 1RU footprint. Contact NetQuest today for a demo!