Kolmogorov-Arnold Networks (KANs) are a novel class of neural network architectures based on the Kolmogorov-Arnold representation theorem, demonstrated potential advantages in accuracy and interpretability over Multi-Layer Perceptron (MLP) models. This paper comprehensively evaluates the robustness of various KAN architectures—including KAN, KAN-Mixer, KANConv_KAN, and KANConv_MLP—against adversarial attacks, a critical aspect that has been underexplored in current research. We compare these models with MLP-based architectures such as MLP, MLP-Mixer, and ConvNet-MLP across three traffic sign classification datasets: GTSRB, BTSD, and CTSD. The models were subjected to a range of adversarial attacks (FGSM, PGD, C&W, and BIM) with varying perturbation levels epsilon=0.01,0.1,1 and were trained under different strategies, including standard training, adversarial training, and Randomized Smoothing. Our experimental results demonstrate that KAN-based models, particularly the KAN-Mixer, exhibit superior robustness to adversarial attacks compared to their MLP counterparts. Specifically, the KAN-Mixer consistently achieved lower Success Attack Rates (SAR) and Degrees of Change (DoC) across most attack types and datasets while maintaining high accuracy on clean data. For instance, under FGSM attacks with epsilon=0.01, the KAN-Mixer outperformed the MLP-Mixer by maintaining higher accuracy and lower SAR. Adversarial training and Randomized Smoothing further enhanced the robustness of KAN-based models, with t-SNE visualizations revealing more stable latent space representations under adversarial perturbations. These findings underscore the potential of KAN architectures to improve neural network security and reliability in adversarial settings. The detailed results and code for all models are available at https://sie-lab-kr.github.io/Is-KAN-Robustness/.
In contrast, the MLP model is a baseline comparison to the KAN architecture. The MLP consists of two
fully connected hidden layers, identical in structure to the KAN model, with 256 units in the first
layer and 128 units in the second layer. The activation function used in the MLP is ReLU, which is
commonly used for its simplicity and computational efficiency.
These KANLinear layers are key to the model's flexibility. They use grid-based kernels with grid
size = 5 and spline order = 3, scaled by 1.0. The grid is dynamically updated within a range of -1
to 1 using a grid eps of 0.02, enhancing adaptability to the data distribution. The activation
function used in the KAN model is SiLU, which allows for smoother gradient flow during
backpropagation. Regularization is enforced through adaptive B-splines, which helps control
overfitting. Following the configuration outlined in Efficient KAN.
Both models' output layers consist of units corresponding to the number of classes in the
datasets. For training, we used the AdamW optimizer with a learning rate of 0.001 and a weight decay
of 1e-4, training over 100 epochs with a batch size of 64. The cross-entropy loss function was
employed to evaluate classification performance.
BTSRB Dataset: The BTSD includes a total of 62 classes, categorized into three superclasses: mandatory, prohibitive, and danger classes. The dataset is divided into 4,591 training images and 2,534 test images, providing variety in sign appearances to assess model robustness, depicted in Figure.
CTSRB Dataset: China's CTSD comprises 6,164 images in 58 categories, with 4,170 training images and 1,994 test images, offering a well-annotated dataset frequently used in traffic sign recognition research, illustrated in Figure.
GTSRB Dataset: The GTSRB dataset, widely used to benchmark traffic sign classification models, contains 43 classes with 39,209 training images and 12,630 test images, offering a diverse representation of German traffic signs, as shown in Figure.
Model | Dataset | Training Type | Attack Type | Epsilon (ε) | Alpha (α) | Num Iter | Accuracy | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|---|
KAN | GTSRB | Standard | BIM | 0.01 | 0.01 | 7 | 78 | 1 | 0.274808437 | 200 Samples |
MLP | GTSRB | Standard | BIM | 0.01 | 0.01 | 7 | 78 | 1.5 | 0.274774492 | 200 Samples |
KAN | BTSRB | Standard | BIM | 0.01 | 0.01 | 7 | 70 | 4.5 | 0.275408447 | 200 Samples |
MLP | BTSRB | Standard | BIM | 0.01 | 0.01 | 7 | 72 | 2 | 0.275390506 | 200 Samples |
KAN | CTSRD | Standard | BIM | 0.01 | 0.01 | 7 | 97 | 0 | 0.2763412 | 200 Samples |
MLP | CTSRD | Standard | BIM | 0.01 | 0.01 | 7 | 96.5 | 1.5 | 0.276363462 | 200 Samples |
KAN | GTSRB | Standard | BIM | 0.01 | 0.01 | 7 | 77.28424386 | 2.098178939 | Full Test Set | |
MLP | GTSRB | Standard | BIM | 0.01 | 0.01 | 7 | 80.53840063 | 1.694378464 | Full Test Set | |
KAN | BTSRB | Standard | BIM | 0.1 | 0.01 | 7 | 60 | 19.5 | 2.54839468 | 200 Samples |
MLP | BTSRB | Standard | BIM | 0.1 | 0.01 | 7 | 62.5 | 17 | 2.546227455 | 200 Samples |
KAN | CTSRD | Standard | BIM | 0.1 | 0.01 | 7 | 50 | 24.5 | 2.531177282 | 200 Samples |
MLP | CTSRD | Standard | BIM | 0.1 | 0.01 | 7 | 60 | 14 | 2.530832291 | 200 Samples |
KAN | GTSRB | Standard | BIM | 1 | 0.01 | 7 | 73.5 | 23.5 | 2.403743029 | 200 Samples |
MLP | GTSRB | Standard | BIM | 1 | 0.01 | 7 | 78.5 | 19.5 | 2.403558016 | 200 Samples |
Model | Dataset | Training Type | Attack Type | Epsilon (ε) | Alpha (α) | Num Iter | Accuracy | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|---|
KAN | GTSRB | Standard | PGD | 0.01 | 0.01 | 7 | 25.5 | 54 | 14.81653881 | 200 Samples |
MLP | GTSRB | Standard | PGD | 0.01 | 0.01 | 7 | 38 | 44.5 | 14.8165369 | 200 Samples |
KAN | BTSRB | Standard | PGD | 0.01 | 0.01 | 7 | 26.5 | 49.5 | 13.48466778 | 200 Samples |
MLP | BTSRB | Standard | PGD | 0.01 | 0.01 | 7 | 36 | 38 | 13.48466492 | 200 Samples |
KAN | CTSRD | Standard | PGD | 0.01 | 0.01 | 7 | 38 | 59 | 10.73444843 | 200 Samples |
MLP | CTSRD | Standard | PGD | 0.01 | 0.01 | 7 | 50 | 48 | 10.73444462 | 200 Samples |
KAN | GTSRB | Standard | PGD | 0.01 | 0.01 | 7 | 26.04908947 | 53.99049881 | Full Test Set | |
MLP | GTSRB | Standard | PGD | 0.01 | 0.01 | 7 | 34.11718131 | 49.90498812 | Full Test Set | |
KAN | BTSRB | Standard | PGD | 0.1 | 0.01 | 7 | 11.5 | 67 | 14.86472416 | 200 Samples |
MLP | BTSRB | Standard | PGD | 0.1 | 0.01 | 7 | 16.5 | 65 | 14.86374474 | 200 Samples |
KAN | CTSRD | Standard | PGD | 0.1 | 0.01 | 7 | 14 | 60.5 | 13.52961063 | 200 Samples |
MLP | CTSRD | Standard | PGD | 0.1 | 0.01 | 7 | 25.5 | 48.5 | 13.52859592 | 200 Samples |
KAN | GTSRB | Standard | PGD | 1 | 0.01 | 7 | 3.5 | 77 | 17.87055397 | 200 Samples |
MLP | GTSRB | Standard | PGD | 1 | 0.01 | 7 | 5.5 | 76 | 17.84584808 | 200 Samples |
KAN | GTSRB | Standard | PGD | 1 | 0.01 | 7 | 2.16152019 | 76.9517023 | Full Test Set | |
MLP | GTSRB | Standard | PGD | 1 | 0.01 | 7 | 4.893111639 | 77.30007918 | Full Test Set |
Training Type | Attack Type | Epsilon | Alpha | Num Iter | Accuracy (Attack) | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|
Standard | PGD | 0.01 | 0.01 | 7 | 0.315 | 0.43 | 26.96932602 | 200 Samples |
Standard | PGD | 0.01 | 0.01 | 7 | 0.353571429 | 0.566666667 | Full Test Set | |
Standard | PGD | 0.1 | 0.01 | 7 | 0.005 | 0.735 | 27.05032539 | 200 Samples |
Standard | PGD | 0.1 | 0.01 | 7 | 0.03015873 | 0.890079365 | Full Test Set |
Model | Dataset | Training Type | Attack Type | Iterations | Learning Rate | Accuracy | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|
KAN | GTSRB | Standard | CW | 1000 | 0.0001 | 27.5 | 51 | 11.39384842 | 200 Samples |
MLP | GTSRB | Standard | CW | 1000 | 0.0001 | 38.5 | 41 | 9.846359253 | 200 Samples |
KAN | BTSRB | Standard | CW | 1000 | 0.0001 | 27.5 | 47 | 9.717828751 | 200 Samples |
MLP | BTSRB | Standard | CW | 1000 | 0.0001 | 38.5 | 35.5 | 8.221411705 | 200 Samples |
KAN | CTSRD | Standard | CW | 1000 | 0.0001 | 42.5 | 54.5 | 7.069438457 | 200 Samples |
MLP | CTSRD | Standard | CW | 1000 | 0.0001 | 52.5 | 45.5 | 5.822953224 | 200 Samples |
KAN | GTSRB | Standard | CW | 1000 | 0.0001 | 26.87252573 | 52.24069675 | Full Test Set | |
MLP | GTSRB | Standard | CW | 1000 | 0.0001 | 34.65558195 | 47.53760887 | Full Test Set | |
KAN | BTSRB | Standard | CW | 1000 | 0.0001 | 44.92063492 | 43.80952381 | Full Test Set | |
MLP | BTSRB | Standard | CW | 1000 | 0.0001 | 60.51587302 | 29.56349206 | Full Test Set | |
KAN | CTSRD | Standard | CW | 1000 | 0.0001 | 36.90186537 | 60.2595296 | Full Test Set | |
MLP | CTSRD | Standard | CW | 1000 | 0.0001 | 52.39253852 | 45.17437145 | Full Test Set |
KAN_Mixer and MLP_Mixer Description:
These KANLinear layers are key to the model's flexibility. They use grid-based kernels with
a grid size of 5 and spline order of 3, scaled by 1.0. The grid is dynamically updated
within a range of -1 to 1 using a grid epsilon of 0.02, enhancing adaptability to the data
distribution. The activation function used in the KAN model is SiLU, which allows for
smoother gradient flow during backpropagation. Regularization is enforced through adaptive
B-splines, which helps control overfitting, following the configuration outlined in
Efficient KAN.
The MLP_Mixer model serves as a baseline comparison, consisting of multiple layers of
Multi-Layer Perceptrons, using ReLU activation functions for computational efficiency.
Both models' output layers consist of units corresponding to the number of classes in the
datasets. For training, we used the AdamW optimizer with a learning rate of 0.001 and a
weight decay of 1e-4, training over 100 epochs with a batch size of 64. The cross-entropy
loss function was employed to evaluate classification performance.
BTSRB Dataset: The BTSD includes a total of 62 classes, categorized into three superclasses: mandatory, prohibitive, and danger classes. The dataset is divided into 4,591 training images and 2,534 test images, providing variety in sign appearances to assess model robustness, depicted in the figure below.
CTSRB Dataset: China's CTSRB comprises 6,164 images in 58 categories, with 4,170 training images and 1,994 test images, offering a well-annotated dataset frequently used in traffic sign recognition research, illustrated in the figure below.
GTSRB Dataset: The GTSRB dataset, widely used to benchmark traffic sign classification models, contains 43 classes with 39,209 training images and 12,630 test images, offering a diverse representation of German traffic signs, as shown in the figure below.
Model | Dataset | Training Type | Attack Type | Epsilon (ε) | Alpha (α) | Num Iter | Accuracy (Attack) | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|---|
KAN_Mixer | BTSRD | Standard | FGSM | 0.01 | 0.41 | 0.34 | 26.97 | 200 Samples | ||
MLP_Mixer | BTSRD | Standard | FGSM | 0.01 | 0.43 | 0.49 | Full Test Set | |||
KAN_Mixer | CTSRD | Standard | FGSM | 0.1 | 0.04 | 0.71 | 27.13 | 200 Samples | ||
MLP_Mixer | CTSRD | Standard | FGSM | 0.1 | 0.05 | 0.87 | Full Test Set | |||
KAN_Mixer | GTSRD | Standard | FGSM | 1 | 0.00 | 0.74 | 42.56 | 200 Samples | ||
MLP_Mixer | GTSRD | Standard | FGSM | 1 | 0.00 | 0.92 | Full Test Set | |||
KAN_Mixer | BTSRD | Adversarial | FGSM | 0.01 | 0.56 | 0.04 | 26.97 | 200 Samples | ||
MLP_Mixer | BTSRD | Adversarial | FGSM | 0.01 | 0.74 | -0.28 | Full Test Set | |||
KAN_Mixer | CTSRD | Adversarial | FGSM | 0.1 | 0.53 | 0.06 | 27.13 | 200 Samples | ||
MLP_Mixer | CTSRD | Adversarial | FGSM | 0.1 | 0.70 | -0.24 | Full Test Set | |||
KAN_Mixer | GTSRD | Adversarial | FGSM | 1 | 0.02 | 0.30 | 42.33 | 200 Samples | ||
MLP_Mixer | GTSRD | Adversarial | FGSM | 1 | 0.10 | 0.36 | Full Test Set |
Model | Dataset | Training Type | Attack Type | Epsilon (ε) | Alpha (α) | Num Iter | Accuracy (Attack) | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|---|
KAN_Mixer | BTSRD | Standard | PGD | 0.01 | 0.01 | 7 | 0.32 | 0.43 | 26.97 | 200 Samples |
MLP_Mixer | BTSRD | Standard | PGD | 0.01 | 0.01 | 7 | 0.35 | 0.57 | 26.97 | 200 Samples |
KAN_Mixer | CTSRD | Standard | PGD | 0.1 | 0.01 | 7 | 0.01 | 0.74 | 27.05 | 200 Samples |
MLP_Mixer | CTSRD | Standard | PGD | 0.1 | 0.01 | 7 | 0.03 | 0.89 | 29.63 | 200 Samples |
KAN_Mixer | GTSRD | Standard | PGD | 1 | 0.01 | 7 | 0.00 | 0.74 | 33.32 | 200 Samples |
MLP_Mixer | GTSRD | Standard | PGD | 1 | 0.01 | 7 | 0.00 | 0.92 | 35.19 | 200 Samples |
KAN_Mixer | BTSRD | Adversarial | PGD | 0.01 | 0.01 | 7 | 0.46 | 0.05 | 26.97 | Full Test Set |
MLP_Mixer | BTSRD | Adversarial | PGD | 0.01 | 0.01 | 7 | 0.64 | -0.18 | Full Test Set | Full Test Set |
KAN_Mixer | CTSRD | Adversarial | PGD | 0.1 | 0.01 | 7 | 0.09 | 0.26 | 27.05 | Full Test Set |
MLP_Mixer | CTSRD | Adversarial | PGD | 0.1 | 0.01 | 7 | 0.20 | 0.25 | 29.63 | Full Test Set |
KAN_Mixer | GTSRD | Adversarial | PGD | 1 | 0.01 | 7 | 0.00 | 0.81 | 35.31 | Full Test Set |
MLP_Mixer | GTSRD | Adversarial | PGD | 1 | 0.01 | 7 | 0.00 | 0.93 | 35.19 | Full Test Set |
Model | Dataset | Training Type | Attack Type | Epsilon (ε) | Alpha (α) | Num Iter | Accuracy (Attack) | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|---|
KAN_Mixer | BTSRD | Standard | BIM | 0.01 | 0.01 | 7 | 0.68 | 0.06 | 0.55 | 200 Samples |
MLP_Mixer | BTSRD | Standard | BIM | 0.01 | 0.01 | 7 | 0.88 | 0.04 | Full Test Set | |
KAN_Mixer | CTSRD | Standard | BIM | 0.1 | 0.01 | 7 | 0.30 | 0.45 | 5.02 | 200 Samples |
MLP_Mixer | CTSRD | Standard | BIM | 0.1 | 0.01 | 7 | 0.44 | 0.48 | Full Test Set | |
KAN_Mixer | GTSRD | Standard | BIM | 1 | 0.01 | 7 | 0.00 | 0.74 | 28.01 | 200 Samples |
MLP_Mixer | GTSRD | Standard | BIM | 1 | 0.01 | 7 | 0.00 | 0.92 | Full Test Set | |
KAN_Mixer | BTSRD | Adversarial | BIM | 0.01 | 0.01 | 7 | 0.28 | 0.03 | 0.55 | 200 Samples |
MLP_Mixer | BTSRD | Adversarial | BIM | 0.01 | 0.01 | 7 | 0.42 | 0.03 | Full Test Set | |
KAN_Mixer | CTSRD | Adversarial | BIM | 0.1 | 0.01 | 7 | 0.03 | 0.29 | 5.04 | 200 Samples |
MLP_Mixer | CTSRD | Adversarial | BIM | 0.1 | 0.01 | 7 | 0.19 | 0.27 | Full Test Set | |
KAN_Mixer | GTSRD | Adversarial | BIM | 1 | 0.01 | 7 | 0.00 | 0.31 | 27.89 | 200 Samples |
MLP_Mixer | GTSRD | Adversarial | BIM | 1 | 0.01 | 7 | 0.03 | 0.43 | Full Test Set |
Model | Dataset | Training Type | Attack Type | Epsilon (ε) | Alpha (α) | Num Iter | Accuracy (Attack) | Success Rate | Degree of Change | Evaluation Samples |
---|---|---|---|---|---|---|---|---|---|---|
KAN_Mixer | BTSRD | Standard | CW | 0.01 | 1000 | 0.40 | 0.35 | 16.87 | 200 Samples | |
MLP_Mixer | BTSRD | Standard | CW | 0.01 | 1000 | 0.45 | 0.47 | Full Test Set | ||
KAN_Mixer | CTSRD | Standard | CW | 0.01 | 1000 | 0.28 | 0.44 | 19.46 | 200 Samples | |
MLP_Mixer | CTSRD | Standard | CW | 0.01 | 1000 | 0.36 | 0.54 | Full Test Set | ||
KAN_Mixer | GTSRD | Standard | CW | 0.01 | 1000 | 0.27 | 0.70 | 16.95 | 200 Samples | |
MLP_Mixer | GTSRD | Standard | CW | 0.01 | 1000 | 0.24 | 0.73 | Full Test Set | ||
KAN_Mixer | BTSRD | Adversarial | CW | 0.01 | 1000 | 0.37 | 0.44 | 17.79 | 200 Samples | |
MLP_Mixer | BTSRD | Adversarial | CW | 0.01 | 1000 | 0.51 | 0.42 | Full Test Set | ||
KAN_Mixer | CTSRD | Adversarial | CW | 0.01 | 1000 | 0.45 | 0.28 | 14.82 | 200 Samples | |
MLP_Mixer | CTSRD | Adversarial | CW | 0.01 | 1000 | 0.51 | 0.35 | Full Test Set | ||
KAN_Mixer | GTSRD | Adversarial | CW | 0.01 | 1000 | 0.40 | 0.56 | 14.43 | 200 Samples | |
MLP_Mixer | GTSRD | Adversarial | CW | 0.01 | 1000 | 0.31 | 0.66 | Full Test Set |
In this section, we investigate the effectiveness of KANs and CNN for traffic sign classification. We are implementing and comparing four different architectures: ConvNet-MLP, ConvNet-KAN, KANConv-MLP, and KANConv-KAN, based on the research by bodner and azam convolutionalkan. All models are trained using a batch size of 512 over 100 epochs, leveraging the Adam optimizer with a learning rate of 0.001 and employing Cross-Entropy Loss as the primary loss function. The ConvNet-MLP model is the baseline architecture, combining standard convolutional layers with a fully connected MLP. It consists of four convolutional layers: two with 32 filters and a kernel size 5x5, followed by two with 64 filters and a kernel size 3x3. These layers are interspersed with ReLU activations and 2x2 max-pooling layers, leading to fully connected layers with 256 neurons for class output. The ConvNet-KAN model builds on this by replacing the fully connected layers with a KAN linear layer, using a 10x10 grid and a spline order of 3 to model complex nonlinear relationships. This architecture retains the initial convolutional layers from the ConvNet-MLP model but introduces the KAN layer after flattening the feature maps to capture more sophisticated patterns in the data. The KANConv-MLP model further explores the potential of KAN by incorporating it directly into the convolutional layers. It employs two KAN-based convolutional layers, each consisting of five independent 3x3 convolutions, followed by max-pooling. These KAN layers aim to enhance the model's feature extraction capabilities by leveraging the adaptive spline functions of KAN during convolution. The output of these KAN-based convolutions is then passed through fully connected layers, similar to those in the ConvNet-MLP model. Finally, the KANConv-KAN model fully integrates KAN into the convolutional and fully connected layers, using KAN-based layers throughout the network. It applies two KAN convolutional layers, similar to those in KANConv-MLP, and follows them with a KAN linear layer for classification. This model is designed to fully leverage KAN's non-linear, adaptive properties to potentially achieve better performance in recognizing traffic signs, especially under adversarial conditions.
BTSRB Dataset: The BTSD includes a total of 62 classes, categorized into three superclasses: mandatory, prohibitive, and danger classes. The dataset is divided into 4,591 training images and 2,534 test images, providing variety in sign appearances to assess model robustness, depicted in Figure.
CTSRB Dataset: China's CTSD comprises 6,164 images in 58 categories, with 4,170 training images and 1,994 test images, offering a well-annotated dataset frequently used in traffic sign recognition research, illustrated in Figure.
GTSRB Dataset: The GTSRB dataset, widely used to benchmark traffic sign classification models, contains 43 classes with 39,209 training images and 12,630 test images, offering a diverse representation of German traffic signs, as shown in Figure.