🧪 10-04: Feature Engineering (NSL-KDD)

---

📌 What Is NSL-KDD?

NSL-KDD is one of the most widely used public benchmark datasets for training and evaluating network intrusion detection systems. It's an improved version of the older KDD Cup 99 dataset, which was itself built from network traffic captured during a DARPA intrusion detection evaluation program in the late 1990s.

Why "NSL"?

The original KDD Cup 99 dataset had well-known problems:

• Massive numbers of duplicate records (some studies found over 75% of records were duplicates), which biased models toward whichever attack type was most repeated
• This let simple classifiers achieve unrealistically high accuracy without actually learning anything meaningful

NSL-KDD (produced by the Canadian Institute for Cybersecurity) fixes this by removing duplicate records and rebalancing the dataset, making it a fairer, more realistic benchmark. It's still widely used today for teaching and research because it's well-labeled, well-documented, and small enough to work with on a laptop.

💡 Because NSL-KDD is older, it doesn't reflect every modern attack technique — but the workflow for using it (cleaning, encoding, scaling, training, evaluating) is exactly the same workflow used on modern, proprietary traffic datasets in industry.

---

📦 What's Inside the Dataset

Each row in NSL-KDD represents one network connection, described by around 40+ features, plus a label identifying it as normal or a specific attack type (e.g., neptune, smurf, satan) — which is usually simplified into a binary normal vs. attack label for basic classification.

The features fall into a few broad groups:

Category	Example Features	What They Describe
Basic connection features	duration, protocol_type, service, flag	Fundamental properties of the connection itself
Content features	num_failed_logins, logged_in, root_shell	What happened inside the connection's payload
Traffic features (time-based)	count, srv_count, same_srv_rate	Statistics about connections to the same host/service in a recent time window
Traffic features (host-based)	dst_host_count, dst_host_srv_count	Similar statistics, aggregated per destination host
Byte counts	src_bytes, dst_bytes	Bytes sent from source to destination and back

A few concrete example features:

• duration — length of the connection in seconds (numeric)
• protocol_type — tcp, udp, or icmp (categorical/text)
• service — the network service on the destination, e.g., http, ftp, telnet (categorical/text)
• flag — the status of the connection, e.g., SF (normal completion), S0 (connection attempt, no reply) (categorical/text)
• src_bytes / dst_bytes — number of data bytes transferred in each direction (numeric)

---

🔤 Why Categorical Features Need Encoding

Machine learning models like SVMs only understand numbers — they cannot directly do math on the text string "tcp" or "http". So before training, every categorical (text-based) feature must be converted into a numeric representation. There are two common techniques:

Label Encoding

Assigns each unique category an integer:

tcp  -> 0
udp  -> 1
icmp -> 2

💡 Gotcha: label encoding implies an order (2 > 1 > 0) that doesn't actually exist between tcp, udp, and icmp. This is fine for the label column we're trying to predict (normal = 0, attack = 1 — a true two-class distinction), but it can mislead a model if used carelessly on an input feature with no natural order.

One-Hot Encoding (Dummy Variables)

Instead of one column with an arbitrary integer, one-hot encoding creates a separate binary (0/1) column for each category:

protocol_type	protocol_tcp	protocol_udp	protocol_icmp
tcp	1	0	0
udp	0	1	0
icmp	0	0	1

This avoids implying a false order between categories, which is why it's the preferred approach for input features like protocol_type, service, and flag.

---

📏 Why Numeric Features Need Scaling

Look at the range of raw NSL-KDD features:

• duration might range from 0 to a few thousand seconds
• src_bytes might range from 0 to tens of millions of bytes
• num_failed_logins might only ever be 0, 1, 2, or 3

If you feed these raw values into an algorithm like an SVM (covered in 10-05: SVMs for Classification), the features with naturally larger numeric ranges (like src_bytes) will dominate the distance calculations the model relies on internally, simply because their numbers are bigger — not because they're actually more important.

💡 Fix: apply feature scaling so every numeric feature is on a comparable scale before training. The most common tool for this is StandardScaler from scikit-learn, which transforms each feature to have a mean of 0 and a standard deviation of 1 (this is called standardization/z-score normalization).

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)   # learn mean/std from training data, then scale it
X_test_scaled = scaler.transform(X_test)          # apply the SAME scaling to test data

💡 Important gotcha: always fit the scaler only on the training data, then use .transform() (not .fit_transform()) on the test data. This prevents information from the test set from "leaking" into training — the model should never see statistics computed from data it's supposed to be evaluated on.

---

🧹 A Practical Preprocessing Example

Here's a simplified, generic pandas snippet illustrating the typical cleanup steps applied to a dataset like NSL-KDD before training:

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Load the dataset
dataset = pd.read_csv("network_traffic.csv")

# Drop rows with missing values
dataset = dataset.dropna()

# Separate features (X) from the label we want to predict (y)
X = dataset.drop("label", axis=1)
y = dataset["label"]

# Drop columns that carry no useful signal (e.g., constant or irrelevant columns)
X = X.drop(["num_outbound_cmds", "is_host_login"], axis=1)

# Encode the label into two numeric classes: 0 = normal, 1 = attack
y = LabelEncoder().fit_transform(y)

# One-hot encode categorical input columns (protocol_type, service, flag, etc.)
X = pd.concat(
    [pd.get_dummies(X.select_dtypes(include="object")),
     X.select_dtypes(exclude="object")],
    axis=1
)

This leaves X as an all-numeric table of features and y as a simple 0/1 array — exactly the shape an SVM expects.

---

📌 Key Takeaways

• NSL-KDD is a widely used, cleaned-up benchmark dataset for network intrusion detection, built to fix duplication problems in the older KDD Cup 99 dataset.
• Each row is a connection described by dozens of features: basic connection info, content features, and time/host-based traffic statistics.
• Categorical features (protocol_type, service, flag) must be encoded into numbers before a model can use them — one-hot encoding is preferred for input features to avoid implying a false order.
• Numeric features must be scaled (e.g., with StandardScaler) so that features with naturally larger ranges don't dominate the model.
• Always fit the scaler on training data only, then apply it to test data — never fit on test data.
• This same clean → encode → scale pipeline generalizes to virtually any tabular intrusion detection dataset, not just NSL-KDD.

---