Traffic Traces (pcap)
Overview
Packets captured by the UCSD network telescope infrastructure are written to pcap trace files and archived to disk for later analysis.
Definition
In STARDUST, pcap data is the raw traffic data collected from the network telescope. Each pcap file contains 1 hour of data and is typically over 100 GB large.
Pcap data from the last 30 days is stored in the telescope-ucsdnt-pcap-live container in the Swift object store. Older trace files, however, are rotated out and moved into a separate archive where a different process is used to access the data.
Properties
Packets contained within the pcap traces are unanonymized and not truncated in any way. They should exactly match what was observed on the wire at the telescope capture point.
User Guide
Recent pcaps (last 30 days)
The most recently generated trace files for the UCSD network telescope typically cover the last 30 days of telescope traffic and can be found in the telescope-ucsdnt-pcap-live container in the Swift object store.
Example
user@vm001:~$ tracepktdump pcapfile:swift://telescope-ucsdnt-pcap-live/datasource=ucsd-nt/year=2020/month=10/day=31/hour=22/ucsd-nt.1604181600.pcap.gz | head
Sat Oct 31 22:00:00 2020
Capture: Packet Length: 60/64 Direction Value: -1
Ethernet: Dest: 3c:fd:fe:19:d8:00 Source: 00:de:fb:ba:06:c7 Ethertype: 0x0800
IP: Header Len 20 Ver 4 DSCP 00 ECN 0 Total Length 40
IP: Id 54321 Fragoff 0
IP: TTL 241 Proto 6 (tcp) Checksum 46313
IP: Source 45.153.203.175 Destination 44.28.27.80
TCP: Source 43922 Dest 82
TCP: Seq 2846108233
Archived pcaps (available upon request)
Files can be brought back from the archive and into a container upon request, but this will not be an instantaneous process and will require manual intervention from a STARDUST administrator. Additionally, space on the Swift object store is limited so the amount of archived data that can be made available at any one time will be restricted. Please bear these limitations in mind when making a request for archived pcap data.
Reading/Processing Trace Files
Libtrace is the recommended method for reading and processing the pcap trace files, as it natively supports reading compressed pcaps through the Swift API.
Alternatively, if you have your own non-libtrace code that you want to run against the traces (e.g. something written using libpcap), you can use the wandiocat tool to stream the pcaps directly from Swift into your program (assuming it accepts stdin as an input source). Wandiocat installation instructions are documented, however, it should be installed on any STARDUST VM or container by default.
Please do not download the pcap files to your VM / container because the files are large and your disk space allocation is relatively small. Use libtrace or wandiocat to stream the data to you from the Swift object store instead.
