ICACT20250226 Slide.18
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (17).jpg) |
And this concludes my presentation. Thank you for your time and your attention.
|
|
ICACT20250226 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (16).jpg) |
To conclude, we introduced the Adaptive Cross-Attention Gated Network (ACAGN) for enhanced radar-camera fusion method in BEV space. Our proposed method integrates deformable cross-attention and a gated network mechanism to dynamically align and prioritize multimodal features. Our proposed method integrates deformable cross-attention and adaptive gated network. This addressed the challenges of spatial misalignment and feature integration. We also demonstrated robustness in complex and adverse weather driving scernarios such as day, night, and rain conditions in weather analysis. We achieved improvement over the baseline, with +0.2 increase in NDS (NuScenes Detection Score) and 0.3 increase in mAP (mean Average Precision) on the nuScenes dataset. For future work, fusion based occupancy mapping or trajectory planning using BEV will be researched.
|
|
ICACT20250226 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (15).jpg) |
Next, we report the visual qualitative results of detection on the nuScenes validation dataset. This figure compares the performance of Ground Truth, RCBEVDet, and adaptive cross attention gated network from left to right. Orange boxes are cars. Red boxes are construction vehicle. Blue boxes are pedestrians. Our method and the baseline are mostly identical but our method aligns closer to ground truth as shown in the red dotted box on the right.
|
|
ICACT20250226 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (14).jpg) |
We report the results of our ablation studies and weather analysis. We conducted ablation studies by progressively adding the CSF module, deformable cross-attention, and the gated network, shown in table 2. We set a baseline with CSF only becasuse it simply concatenates and feeds into CBR blocks. Consequently, adding deformable cross-attention and gate network sequentially improves model performance. Specifically, adopting the gated network further refines the fusion process and results in improvements in most of all metric except for mASE and mAOE.
We also evaluated the performance in various weather conditions under day, night, and rain weather scenarios. The table 3 compares our proposed model with baseline. ACAGN demonstrates consistent improvement, especially in rain condition with huge improvements than others.
|
|
ICACT20250226 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (13).jpg) |
We present the quantitative results of our evaluation on the nuScenes validation dataset. We compared our proposed method a Adaptive Cross-Attention Gated Network against other state-of-the-art methods including camera-only and camera-radar. We used R50 for the backbone network with image resolution of 256 x 704. We report two primary metric NDS and mAP and five true positive metrics as well. Our proposed method achieves the best NuScenes detection score of 56.9. This improves NDS with 0.2 improvement and mAP with 0.3 improvement over the baseline, RCBEVDet from CVPR 2024.
|
|
ICACT20250226 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (12).jpg) |
For our experiment, we used nuScenes dataset. It is a large scale outdoor multimodal benchmark providing full 360 degree coverage across all vision. The figure above illustrates the sensor setup on the car that includes 6 cameras, 5 radars for front and back, 1 lidar on top with imu. This dataset provides official splits with 700 scenes for training and 150 scenes for validation. For evaluation, we used two primary metrics. Mean average precision and NuScenes detection scores. NDS is a composite metric that combines and balances mAP with five true positive metrics to provide a evaluation of detection quality. The formula for NDS as shown below.
|
|
ICACT20250226 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (11).jpg) |
Finally, we added channel and spatial fusion module for further fusement of gated radar and camera features. This CSF module consists of several CBR blocks which sequentially composed of 3x3 convolution, batch normalization, and ReLU activation function. As shown in this figure, this concatenated gated features are passed through a CBR block with a residual connection followed by three additional CBR blocks to produce the final fused feature.
|
|
ICACT20250226 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (10).jpg) |
Next we added a gated network to selectively filter and prioritize the most informative features from each sensor dynamically. Specifically, the gated fusion network produces attention maps by applying 3 × 3 convolution followed by a sigmoid activation to the concatenated inputs. The sigmoid function produces values between 0 and 1. They are used to emphasize the relative importance of each modality. For example, in scenarios like rain or fog, the network dynamically assigns higher weights to radar features. And for sunny or clear condition, more weight will be assigned to camera feature, respectively. By applying these values as weights into the original input feature maps, the network can adjust the strength of each feature dynamically.
|
|
ICACT20250226 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (9).jpg) |
Deformable cross attention dynamically aligns and selectively focus on spatial regions. Instead of attending to all positions in the feature maps like standard cross-attention, deformable cross-attention uses learnable offsets to identify only the most relevant regions. This reduces computational complexity and mitigates spatial misalignment. With learnable positional encodings are added, radar BEV feature is transformed into queries and reference points, and Camera BEV features are used as keys and values in this cross-attention mechanism. And the same process is applied for the camera BEV feature as well.
|
|
ICACT20250226 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (8).jpg) |
Now, Adaptive Cross-Attention Gated Network follows in three steps: deformable cross-attention, gated network, and CSF module which is channel and spatial fusion
|
|
ICACT20250226 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (7).jpg) |
Before dive into our proposed fusion method, the first step is to convert multi-view images and radar point cloud data into a shared Bird’s Eye View (BEV) space from the baseline framework. Multi-view RGB images are first passed through an image backbone network to extract features. These features are then processed through a view transform module, which integrates information from multiple camera views and projects them onto a top-down BEV map. Finally, the camera BEV features are extracted from Image BEV encoder. Simultaneously, radar point cloud data is processed using a dual-stream radar backbone, which includes a point-based encoder for capturing local features and a transformer-based encoder for extracting global contextual information. The radar features are then refined using the RCS-aware scattering to handle radar’s sparsity and noise. Finally, the radar BEV features are obtained from radar BEV encoder.
|
|
ICACT20250226 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (6).jpg) |
This the overview architecture of our proposed Adaptive Cross-Attention Gated Network, or ACAGN. This enhances sensor fusion in Bird’s Eye View (BEV) space by integrating millimeter wave radar and multi-view RGB camera inputs. This framework incorporates cross-attention and adaptive gated network mechanisms with the CSF module which consists of several CBR blocks.
|
|
ICACT20250226 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Despite the heavy reliance on backbone networks for sensor fusion, handling spatial alignment effectively in BEV space for camera and radar still remains a challenge. To alleviate, we propose an advanced multimodal fusion method, Adaptive Cross-Attention Gated Network (ACAGN) which built upon RCBEVDet from CVPR 2024. This method integrates two key components. Deformable cross-attention improves spatial alignment with higher precision and adaptive gated network filters and prioritizes sensor features.
|
|
ICACT20250226 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To address this, there are several existing fusion methods introduced for radar-camera fusion. CRKD uses adaptive gated network. CRN [9] and BEV-Radar [33] use deformable cross-attention. UniBEVFusion [31] utilized Unified Feature Fusion (UFF) module. RCBEVDet integrates deformable cross-attention with CBR blocks. RCFusion [34] applies CBAM [24] with CBR blocks.
|
|
ICACT20250226 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
BEVFusion, in 2022, introduced an effective method to project both LiDAR and camera features into a shared Bird’s Eye View (BEV) space. This approach allows the conversion of 3D sensor data type into 2D representation while preserving geographic information. However, once LIDAR and camera features are being projected into BEV space as shown in figure, they are simply concatenated and passed to the BEV encoder to produce fused feature. Since simple concatenation does not provide a way to prioritize or weigh important features, this can lead to poor handling in spatial alignment and feature discrepancies for downstream tasks.
|
|
ICACT20250226 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In autonomous driving, the most commonly used multimodal sensors are cameras, radar, and LiDAR. While LiDAR is known for providing detailed geometric information, it is expensive and requires regular maintenance for consumer vehicle. On the other hand, radar is more cost-effective. When combined with cameras, radar-camera fusion integrates radar’s depth and velocity estimation with the semantic details provided by cameras. However, current radar feature extractors often use encoders such as PointPillars which were originally designed for LiDAR. Due to radar's sparsity and noise, this can lead to spatial misalignment and difficulties in feature integration.
|
|
ICACT20250226 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
.pptx (1).jpg) |
The contents are divided into five main sections for today's presentation. Introduction, related works, proposed method, results, and conclusion.
|
|
ICACT20250226 Slide.01
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
.pptx.jpg) |
Hi everyone, my name is Jiyong Lee and, today, I would like to present our work on advancing multimodal sensor fusion for autonomous driving system. The title of this paper is called ‘Adaptive cross attention gated network for radar-camera fusion in BEV space’.
|
