Advances in Brain-Computer Interfaces and Neural Integration
ChaCha: A Dynamic Slope Activation Function for Enhanced Neural Network Performance
Abstract
Bidyut B. Chaudhuri and Archisman Chakraborti
Activation functions are pivotal in introducing non-linearity to neural networks, significantly impacting their training dynamics and overall performance. Despite the plethora of activation functions developed over the years, only a handful—such as ReLU (Rectified Linear Unit), Leaky ReLU, Tanh (Hyperbolic Tangent), Sigmoid, Swish, GELU, and Mish—are predominantly utilized in most applications. In this paper, we present a novel activation function we named ChaCha, characterized by its dynamically changing slope, continuity, and differentiability across its entire domain. Our extensive experiments demonstrate that incorporating ChaCha into various deep learning architectures (here shown on ResNet, transformer-based models, and Vanilla GAN) yield substantial performance enhancements. Specifically, on datasets like CIFAR-100 and Mini-ImageNet employing ChaCha achieved a 10–15% improvement in accuracy and F1 scores compared to their counterparts using traditional and state-of-the-art activation functions such as ReLU, ELU, Leaky ReLU, GELU, Sigmoid, SeLU, SiLU and Mish. This is true for metrics like Frechet Inception Distance (FID) ́ and Inception Score (IS) for the Vanilla GAN as well. The dynamic adaptability and straightforward implementation of ChaCha is expected to make it a compelling choice for researchers and practitioners aiming to boost neural network performance across a wide range of tasks.