FlowSegModel: Weather-Resilient Semantic Segmentation

This research developed a comprehensive methodology for semantic segmentation in adverse weather conditions, employing a FlowSegModel architecture that integrates RGB images with optical flow inputs within a modified DeepLabV3 framework. The system was evaluated on the VKITTI2 dataset, demonstrating robust performance under dynamic weather conditions including fog, rain, and overcast skies.

The FlowSegModel achieves mean Intersection over Union (IoU) scores of 0.8406 under normal conditions and 0.8034 in adverse scenarios, demonstrating only a 4.4% degradation. This resilience is attributed to optical flow’s complementary motion cues, which enhance visibility in degraded environments where traditional RGB-only methods struggle.

Key Components and Methodology

1. Dataset Preparation
   The VKITTI2 synthetic benchmark dataset was utilized, providing RGB image pairs, optical flow ground truth, and segmentation labels across multiple weather conditions (clone/normal, fog, rain, overcast) and camera viewpoints (15°/30° left/right). Flow denormalization was performed using f = 2/(2^16 - 1) × bgr_RG - 1, scaled to pixel coordinates. Images were concatenated into 5-channel tensors (3 RGB + 2 optical flow u/v components) and resized to 384×1248 pixels.

2. Optical Flow Estimation
   Four optical flow estimation models were evaluated: fine-tuned RAFT and SEA-RAFT (deep learning-based), alongside classical Farneback and Horn-Schunck methods. Deep models were pretrained on standard benchmarks and fine-tuned on VKITTI2. Evaluation metrics included End-Point Error (EPE = sqrt((u - u_gt)^2 + (v - v_gt)^2)), F1-all outlier rate (EPE > 3 pixels and relative error > 0.05), and precision thresholds (px<1, px<3, px<5). SEA-RAFT achieved superior performance with EPE of 0.7758 after fine-tuning, compared to RAFT’s 1.7861 and classical methods exceeding 512.

3. Segmentation Architecture (FlowSegModel)
   A modified DeepLabV3+ architecture with ResNet-101 backbone was implemented. The initial convolutional layer was reconfigured from 3 to 5 input channels to accommodate the RGB-flow fusion. The Atrous Spatial Pyramid Pooling (ASPP) module was optimized with adjusted dilation rates to capture multi-scale context under degraded visibility. The classifier head was adapted for 7-class pixel-wise predictions. Training utilized AdamW optimizer with learning rate 1e-4, cross-entropy loss, batch size 16, and early stopping based on validation mIoU over 20 epochs.

4. Cyclostationary Analysis
   Spectral Correlation Density (SCD) was computed from denoised signals via Spectral Correlation Function: S_x^α(f) = lim(T→∞) (1/T) × <X_T(f + α/2) · X_T*(f - α/2)>. This analysis generated 2D SCD images that exploit the periodic nature of motor emissions, providing robust features for classification.

5. Evaluation and Ablation Studies
   Performance was assessed using mean IoU (mIoU = (1/C) × Σ TP_c/(TP_c + FP_c + FN_c)), along with macro-averaged precision, recall, and F1-score. Comparative analyses revealed that fine-tuned deep learning models outperformed classical methods by over 96% in EPE reduction. An ablation study using an optical-flow-only variant demonstrated a 46.1% mean IoU improvement when fusing RGB and flow, confirming that flow-alone struggles with static scene discrimination absent color and texture data.

Technical Contributions

• Pioneered a 5-channel fusion architecture adapting DeepLabV3+ for RGB-optical flow integration, achieving weather-resilient segmentation.
• Demonstrated SEA-RAFT’s superiority over RAFT and classical methods, with 56.6% better EPE performance through weather-aware design and Mixture-of-Laplace loss optimization.
• Conducted comprehensive ablation studies quantifying RGB-flow synergy, showing flow-only variants achieve only 0.1431 IoU for vehicles versus 0.8375 for the full model.
• Validated end-to-end pipeline on VKITTI2 with PyTorch implementation on NVIDIA A100 GPUs, achieving minimal 4.4% mIoU degradation under adverse conditions.
• Advanced autonomous driving perception with practical implications for weather-invariant navigation and reduced real-world failure rates.

Visual Results

Performance Metrics

Optical Flow Comparison Across Methods

Segmentation Performance by Class

Ablation Study: Flow-Only vs Full RGB-Flow Model

Acknowledgments

This work was conducted at the Multidisciplinary AI Research Centre (MARC), University of Peradeniya, Sri Lanka, under the guidance of Prof. Roshan Godaliyadda and Prof. Parakrama Ekanayake from the Department of Electrical and Electronic Engineering. Collaborative contributions were made by Thiwanka Alahakoon, Oshada Rathnayake, and Sanjula Senadeera.