HyMTAS-GBM: Integrating Semantic Convolutions and Multi-Head Attention with Gradient Boosting for Robust Object Detection

Rahul Mandlik; Avinash Sharma

doi:10.5753/jbcs.2026.8072

Authors

Rahul Mandlik Department of Computer Science & Engineering, Oriental University, Indore-453555, Madhya Pradesh, India. https://orcid.org/0009-0001-0394-9057
Avinash Sharma Department of Computer Science & Engineering, Oriental University, Indore-453555, Madhya Pradesh, India. https://orcid.org/0009-0003-1192-8381

DOI:

https://doi.org/10.5753/jbcs.2026.8072

Keywords:

Object Detection, Machine Learning, Artificial Neural Networks, Transformer Models, Hybrid Multi-Head Attention, Gradient Boosting Machine

Abstract

Object detection in complex scenes requires models that can maintain precise spatial attention and reliable query representations. However, existing CNN and DETR-based detectors often exhibit diffuse cross-attention, weak query initialization, and limited robustness in cluttered or dynamic environments. This paper proposes HyMTAS-GBM, a hybrid multi-head transformer architecture that integrates attention-consensus learning, semantic-guided query selection, and gradient-boosting classification to improve detection reliability across image and video domains. The framework introduces Consensus Multi-Head Attention (CoMHA) to stabilize attention correlations, Semantic-Regional Query Selection (SRQS) for object-aware query initialization, and Regional Reinforcement Cross-Attention (RRCA) with learnable key-point agents for spatially constrained reasoning. A sparsity-guided feature amplification pipeline further enhances encoder representations for small and occluded objects. Extensive experiments on ScanNetV2, SUN RGB-D, ImageNet VID, and UA-DETRAC demonstrate consistent improvements, achieving 90.6 mAP25 and 85.3 mAP50 on ScanNetV2 while maintaining competitive efficiency.

Downloads

Download data is not yet available.

References

Athar, A., Mahadevan, S., Osep, A., Leal-Taixé, L., and Leibe, B. (2020). Stem-seg: spatio-temporal embeddings for instance segmentation in videos. In Lecture Notes in Computer Science, pages 158-177. Springer. DOI: 10.1007/978-3-030-58621-8_10.

Cao, B., Yao, H., Zhu, P., and Hu, Q. (2024). Visible and clear: Finding tiny objects in difference map. In Proceedings of the European Conference on Computer Vision (ECCV). DOI: 10.48550/arXiv.2405.11276.

Cao, G., Huang, W., Lan, X., Zhang, J., Jiang, D., and Wang, Y. (2025). Cross-dino: Cross the deep mlp and transformer for small object detection. IEEE Transactions on Multimedia, 27:7369-7379. DOI: 10.1109/tmm.2025.3599074.

Chen, X., Ma, H., Wan, J., Li, B., and Xia, T. (2017). Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1907-1915. DOI: 10.1109/CVPR.2017.691.

Chen, Y., Cao, Y., Hu, H., and Wang, L. (2020). Memory enhanced global-local aggregation for video object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10334-10343. DOI: 10.1109/CVPR42600.2020.01035.

Chen, Y., Han, B., Li, C., Ko, T., Hebert, M., and Shi, J. (2021). Hais: hierarchical aggregation for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15480-15489. DOI: 10.1109/ICCV48922.2021.01518.

Cheng, G., Yuan, X., Yao, X., Yan, K., Zeng, Q., Xie, X., and Han, J. (2023). Towards large-scale small object detection: survey and benchmarks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):13467-13488. DOI: 10.1109/TPAMI.2023.3290594.

Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., and Nießner, M. (2017). Scannet: richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2432-2443. DOI: 10.1109/CVPR.2017.261.

Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., and Zhang, L. (2021). Dynamic head: unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7373-7382. DOI: 10.1109/CVPR46437.2021.00729.

Du, Z., Hu, Z., Zhao, G., Jin, Y., and Ma, H. (2025). Cross-layer feature pyramid transformer for small object detection in aerial images. IEEE Transactions on Geoscience and Remote Sensing, 63:5625714. DOI: 10.1109/TGRS.2025.3572706.

F. Engelmann, M. Bokeloh, A. Fathi, B. Leibe and M. Nießner (2020). 3d-mpa: multi-proposal aggregation for 3d semantic instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8650-8659. DOI: 10.1109/CVPR42600.2020.00905.

Fan, Z., Wang, C., Yu, Q., Min, L., Wang, S., and Wang, B. (2026). So-detr: Small object detection transformer via state space duality encoder and multigroup rank supervision. IEEE Transactions on Geoscience and Remote Sensing, 64:5630013. DOI: 10.1109/tgrs.2026.3703859.

Ge, Z., Liu, S., Wang, F., Li, Z., and Sun, J. (2021). Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430. DOI: 10.48550/arXiv.2107.08430.

Gu, Y., Hu, Z., Zhao, Y., Liao, J., and Zhang, W. (2024). Mfgtn: a multi-modal fast gated transformer for identifying single trawl marine fishing vessel. Ocean Engineering, 303:117711. DOI: 10.1016/j.oceaneng.2024.117711.

Guo, Y., Yang, Z., Kou, F., Hu, Y., Zhang, R., and Yao, S. (2026). Adaptive spectrum-aware feature disentangled network for small object detection. arXiv preprint arXiv:2606.29029. DOI: 10.48550/arXiv.2606.29029.

Han, K. et al. (2023). A survey on vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:87-110. DOI: 10.1109/TPAMI.2022.3152247.

Han, M., Wang, Y., Chang, X., and Qiao, Y. (2020). Mining inter-video proposal relations for video object detection. In Lecture Notes in Computer Science, volume 12366. Springer. DOI: 10.1007/978-3-030-58589-1_26.

He, Y., Zhu, C., Wang, J., Savvides, M., and Zhang, X. (2019). Bounding box regression with uncertainty for accurate object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2883-2892. DOI: 10.1109/CVPR.2019.00300.

Huang, Z., Zhang, C., Jin, M., Wu, F., Liu, C., and Jin, X. (2024). Better sampling, towards better end-to-end small object detection. arXiv preprint arXiv:2407.06127. DOI: 10.48550/arXiv.2407.06127.

Jiang, J., Zhao, D., Shi, J., and Gao, Y. (2020). Pointgroup: dual-set point grouping for 3d instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4867-4876. DOI: 10.1109/CVPR42600.2020.00492.

Jocher, G. (2020). Ultralytics yolov5. [link].

Jocher, G., Chaurasia, A., and Qiu, J. (2023). Ultralytics yolov8. [link].

Jocher, G. and Qiu, J. (2024). Ultralytics yolo11. [link].

Law, H. and Deng, J. (2018). Cornernet: detecting objects as paired keypoints. In Proceedings of the European Conference on Computer Vision (ECCV), pages 734-750. DOI: 10.1007/978-3-030-01264-9_45.

Li, F., Zhang, H., Liu, S., Guo, J., Ni, L. M., and Zhang, L. (2024). Dn-detr: accelerate detr training by introducing query denoising. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:2239-2251. DOI: 10.1109/TPAMI.2023.3335410.

Li, H. et al. (2023). Multi-object tracking via deep feature fusion and association analysis. Engineering Applications of Artificial Intelligence, 124:106527. DOI: 10.1016/j.engappai.2023.106527.

Li, X., Wang, W., Wu, L., Chen, S., Hu, X., Yang, J., and Dai, J. (2020). Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. In Advances in Neural Information Processing Systems (NeurIPS). DOI: 10.48550/arXiv.2006.04388.

Li, Y. et al. (2022). Mvitv2: improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4794-4804. DOI: 10.1109/CVPR52688.2022.00476.

Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017a). Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2999-3007. DOI: 10.1109/ICCV.2017.324.

Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017b). Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2980-2988. DOI: 10.1109/ICCV.2017.324.

Liu, S. et al. (2022). Dab-detr: dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201.12329. DOI: 10.48550/arXiv.2201.12329.

Liu, Z., Zhang, Z., Cao, Y., Hu, H., and Tong, X. (2021). Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2929-2938. DOI: 10.1109/ICCV48922.2021.00294.

Lv, W., Xu, S., Zhao, Y., Wang, G., Wei, J., Cui, C., Du, Y., Dang, Q., and Liu, Y. (2023). Detrs beat yolos on real-time object detection. arXiv preprint arXiv:2304.08069. DOI: 10.48550/arXiv.2304.08069.

Mandlik, S., Labade, R., Chaudhari, S., and Agarkar, B. (2025a). Optimizing gait-based biometric recognition with deep dense networks. Brazilian Journal of Biometrics, 43:e-43857. DOI: 10.28951/bjb.v43i4.857.

Mandlik, S., Labade, R., Chaudhari, S., et al. (2025b). Pose-invariant gait recognition using an enhanced inception-resnet and vision transformer framework. Journal of Engineering and Applied Sciences, 72:234. DOI: 10.1186/s44147-025-00820-8.

Mandlik, S. et al. (2026). Dual attention embedded inception framework for robust biometric gait recognition. ELCVIA Electronic Letters on Computer Vision and Image Analysis, 25(2):79-105. DOI: 10.5565/rev/elcvia.2402.

Meng, D. et al. (2021). Conditional detr for fast training convergence. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3631-3640. DOI: 10.1109/ICCV48922.2021.00363.

Misra, I., Girdhar, R., and Joulin, A. (2021). An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2886-2897. DOI: 10.1109/ICCV48922.2021.00290.

Pagire, V., Chavali, M., and Kale, A. (2025). A comprehensive review of object detection with traditional and deep learning methods. Signal Processing, 237:110075. DOI: 10.1016/j.sigpro.2025.110075.

Pan, L., Luan, W., Zheng, Y., Li, J., Tao, L., and Xu, C. (2024). Graphfusion: integrating multi-level semantic information with graph computing for enhanced 3d instance segmentation. Neurocomputing, 602:128287. DOI: 10.1016/j.neucom.2024.128287.

Qi, C. R., Litany, O., He, K., and Guibas, L. J. (2019). Deep hough voting for 3d object detection in point clouds. arXiv preprint arXiv:1904.09664. DOI: 10.48550/arXiv.1904.09664.

Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016). You only look once: unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779-788. DOI: 10.1109/CVPR.2016.91.

Ren, S., He, K., Girshick, R., and Sun, J. (2017). Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137-1149. DOI: 10.1109/TPAMI.2016.2577031.

Rukhovich, D., Vorontsova, A., and Konushin, A. (2022). Imvoxelnet: image to voxels projection for monocular and multi-view 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2397-2406. DOI: 10.1109/WACV51458.2022.00133.

Russakovsky, O., Deng, J., Su, H., et al. (2015). Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252. DOI: 10.1007/s11263-015-0816-y.

Shehzadi, T., Hashmi, K. A., Liwicki, M., Stricker, D., and Afzal, M. Z. (2025). Object detection with transformers: a review. Sensors, 25:6025. DOI: 10.3390/s25196025.

Shen, Y., Geng, Z., Yuan, Y., Lin, Y., Liu, Z., Wang, C., Hu, H., Zheng, N., and Guo, B. (2023). V-detr: Detr with vertex relative position encoding for 3d object detection. arXiv preprint arXiv:2308.04409. DOI: 10.48550/arXiv.2308.04409.

Shi, S., Fang, Q., Xu, X., and Zhao, T. (2024). Similarity distance-based label assignment for tiny object detection. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13711-13718. DOI: 10.1109/IROS58592.2024.10801448.

Shi, W. and Rajkumar, R. (2020). Point-gnn: graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1711-1719. DOI: 10.1109/CVPR42600.2020.00178.

Shi, Z., Hu, J., Ren, J., Ye, H., Yuan, X., Ouyang, Y., He, J., Ji, B., and Guo, J. (2025). Hs-fpn: high frequency and spatial perception fpn for tiny object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 6896-6904. DOI: 10.1609/aaai.v39i7.32740.

Song, J., Zhou, M., Luo, J., Pu, H., Feng, Y., Wei, X., and Jia, W. (2025). Boundary-aware feature fusion with dual-stream attention for remote sensing small object detection. IEEE Transactions on Geoscience and Remote Sensing, 63:5600213. DOI: 10.1109/TGRS.2024.3514376.

Song, S., Lichtenberg, S. P., and Xiao, J. (2015). Sun rgb-d: a rgb-d scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 567-576. DOI: 10.1109/CVPR.2015.7298655.

Sun, G., Hua, Y., Hu, G., and Robertson, N. (2021a). Mamba: multi-level aggregation via memory bank for video object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2620-2627. DOI: 10.1609/aaai.v35i3.16365.

Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., Wang, C., and Luo, P. (2021b). Sparse r-cnn: end-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14454-14463. DOI: 10.1109/CVPR46437.2021.01422.

Tian, Z., Shen, C., Chen, H., and He, T. (2019a). Fcos: fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9626-9635. DOI: 10.1109/ICCV.2019.00972.

Tian, Z., Shen, C., Chen, H., and He, T. (2019b). Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9627-9636. DOI: 10.1109/ICCV.2019.00972.

Vu, T., Jang, H., Pham, T. X., and Yoo, C. D. (2019). Cascade rpn: delving into high-quality region proposal network with adaptive convolution. In Advances in Neural Information Processing Systems (NeurIPS). DOI: 10.48550/arXiv.1909.06720.

Vu, T., Kim, K., Luu, T. M., Nguyen, T., and Yoo, C. D. (2022). Softgroup for 3d instance segmentation on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2708-2717. DOI: 10.48550/arXiv.2203.01509.

Wang, A. et al. (2024). Yolov10: Real-time end-to-end object detection. In Advances in Neural Information Processing Systems (NeurIPS). DOI: 10.48550/arXiv.2405.14458.

Wang, H., Ding, L., Dong, S., Shi, S., Li, A., Li, J., Li, Z., and Wang, L. (2022a). Cagroup3d: class-aware grouping for 3d object detection on point clouds. In Advances in Neural Information Processing Systems (NeurIPS). DOI: 10.52202/068431-2173.

Wang, H. et al. (2022b). Rbgnet: ray-based grouping for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1110-1119. DOI: 10.1109/CVPR52688.2022.00118.

Wang, S., Lu, K., Xue, J., and Zhao, Y. (2025). Da-net: density-aware 3d object detection network for point clouds. IEEE Transactions on Multimedia, 27:665-678. DOI: 10.1109/TMM.2023.3245359.

Wei, X. and Konushin, A. (2026). Hff-big: hierarchical feature fusion with bi-directional gating for 3d indoor instance segmentation. Neurocomputing, 680:133233. DOI: 10.1016/j.neucom.2026.133233.

Wen, L., Du, D., Cai, Z., et al. (2020). Ua-detrac: a new benchmark and protocol for multi-object detection and tracking. Computer Vision and Image Understanding, 193:102907. DOI: 10.1016/j.cviu.2020.102907.

Wu, H., Chen, Y., Wang, N., and Zhang, Z. (2019). Sequence level semantics aggregation for video object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9217-9225. DOI: 10.1109/ICCV.2019.00931.

Xie, Q., Lai, Y.-K., Wu, J., Wang, Z., Zhang, Y., Xu, K., and Wang, J. (2020). Mlcvnet: multi-level context votenet for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10447-10456. DOI: 10.1109/CVPR42600.2020.01046.

Xu, C., Wang, J., Yang, W., Yu, H., Yu, L., and Xia, G.-S. (2022). Rfla: Gaussian receptive field based label assignment for tiny object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 526-543. DOI: 10.1007/978-3-031-20077-9_31.

Xu, Z., Hrustic, E., and Vivet, D. (2020). Centernet heatmap propagation for real-time video object detection. In Lecture Notes in Computer Science, volume 12370. Springer. DOI: 10.1007/978-3-030-58595-2_14.

Yang, Z., Liu, S., Hu, H., Wang, L., and Lin, S. (2019). Reppoints: Point set representation for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9657-9666. DOI: 10.1109/ICCV.2019.00975.

Yao, Y., Luo, Z., Li, S., Fang, T., and Zeng, A. (2023). Occformer: dual-path transformer for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15590-15600. DOI: 10.1109/ICCV51070.2023.00865.

Yin, X., Lian, X., Han, Z., Zhu, B., Li, H., and Nießner, M. (2023). Uni3d: exploring unified 3d representation at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 20226-20237. DOI: 10.48550/arXiv.2310.06773.

Yuan, X., Cheng, G., Cheng, J., Yao, R., and Han, J. (2026). Unc-sod: an uncertainty learning framework for small object detection. IEEE Transactions on Image Processing, 35:1127-1142. DOI: 10.1109/TIP.2026.3654892.

Yuan, X., Cheng, G., Yan, K., Zeng, Q., and Han, J. (2023). Small object detection via coarse-to-fine proposal generation and imitation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6317-6327. DOI: 10.1109/ICCV51070.2023.00581.

Zhang, H., Liu, H., and Kim, C. (2024). Semantic and instance segmentation in coastal urban spatial perception. Sustainability, 16:833. DOI: 10.3390/su16020833.

Zhang, H. et al. (2022). Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605. DOI: 10.48550/arXiv.2203.03605.

Zhang, S., Chi, C., Yao, Y., Lei, Z., and Li, S. Z. (2020). Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9759-9768. DOI: 10.1109/CVPR42600.2020.00978.

Zhao, L., Guo, J., Xu, D., and Sheng, L. (2021). Transformer3d-det: improving 3d object detection by vote refinement. IEEE Transactions on Circuits and Systems for Video Technology, 31:4735-4746. DOI: 10.1109/tcsvt.2021.3102025.

Zhou, X., Wang, D., and Krähenbühl, P. (2019). Objects as points. arXiv preprint arXiv:1904.07850. DOI: 10.48550/arXiv.1904.07850.

Zhou, Z., Zhao, X., Zhao, X., Zhu, Y., Zhao, Z., and Li, L. (2026). Gendist: Generative-driven knowledge distillation for small object detection in remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 64:1-16. DOI: 10.1109/tgrs.2026.3676015.

Zhou, Z. and Zhu, Y. (2024). Kldet: Detecting tiny objects in remote sensing images via kullback-leibler divergence. IEEE Transactions on Geoscience and Remote Sensing, 62:4703316. DOI: 10.1109/TGRS.2024.3382099.

Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2021). Deformable detr: deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR). DOI: 10.48550/arXiv.2010.04159.

Zhu, X., Wang, Y., Dai, J., Yuan, L., and Wei, Y. (2017). Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 408-417. DOI: 10.1109/ICCV.2017.52.

Zong, Z., Song, G., and Liu, Y. (2023). Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). DOI: 10.1109/ICCV51070.2023.00621.