A Transformer-Based Multimodal Cross-Attention Framework with Relational Graph Attention for Pill–Prescription Matching
Linh Nguyen Thi My1,2,*, Lap Thai Viet2, Hoai Truong Hieu2, Tham Vo1, Vinh Truong Hoang3,*
1 Faculty of Information Technology, Nguyen Tat Thanh University, 300A Nguyen Tat Thanh Street, Xom Chieu Ward, Ho Chi Minh City, Vietnam
2 Faculty of Information Technology, Van Lang School of Technology, Van Lang University, 69/68 Dang Thuy Tram Street, Binh Loi Trung Ward, Ho Chi Minh City, Vietnam
3 Faculty of Information Technology, Ho Chi Minh City Open University, 35–37 Ho Hao Hon Street, Cau Ong Lanh Ward, Ho Chi Minh City, Vietnam
* Corresponding Author: Linh Nguyen Thi My. Email: 
; Vinh Truong Hoang. Email: 
(This article belongs to the Special Issue: Advances in Object Detection and Recognition)
Computers, Materials & Continua https://doi.org/10.32604/cmc.2026.086877
Received 06 June 2026; Accepted 19 August 2026; Published online 07 September 2026
Abstract
Matching physical pills with drug names on a prescription is a safety-critical task for preventing medication errors. The pioneering PIMA (PIll-prescription MAtching) framework addressed this problem by aligning convolutional features (ResNet50) with textual embeddings (BERT) using a direction-agnostic GraphSAGE operator and a margin-based contrastive loss. While effective on controlled datasets, PIMA’s real-world deployment is hindered by its reliance on manual ground-truth annotations, sensitivity to background clutter, and a simplistic cosine similarity fusion that lacks selective attention. In this paper, we propose a redesigned, fully multimodal framework featuring five coordinated improvements: (i) two interchangeable visual encoders for the pill-recognition branch—a Vision Transformer (ViT-B/16) applied to background-removed crops, and an end-to-end Faster R-CNN/RoI-Align detector that learns features directly from raw images; (ii) a lightweight all-MiniLM-L6-v2 sentence encoder for efficient pill-name representation; (iii) an integrated PP-OCRv3 module that automates text and bounding box extraction, eliminating manual dependency; (iv) a Relational Graph Attention Network (R-GAT) with a pseudo-classifier gate to model directional spatial logic (e.g., left-to-right drug/quantity relations) and filter irrelevant non-drug text; and (v) a multimodal cross-attention head optimized via an InfoNCE objective for more stable and dense gradient convergence. Experiments on the real-world VAIPE dataset show that both variants substantially outperform a reproduced graph baseline: the ViT-B/16 variant raises the Top-1 matching accuracy from 49.89% to 82.46% and the end-to-end Faster R-CNN variant to 83.47%—absolute gains of 32.57 and 33.58 percentage points (pp), respectively, while placing the correct drug name among the top-five candidates for more than 99% of pills. The two variants trade off accuracy against preprocessing: the Faster R-CNN variant attains the best accuracy while operating directly on raw images, whereas the ViT-B/16 variant requires a separate background-removal step. In terms of computational footprint, the visual backbones contain approximately 86.6M (ViT-B/16) and 41.8M (Faster R-CNN/ResNet50-FPN) parameters, the 22.7M-parameter MiniLM encoder replaces the far heavier BERT of the baseline, and although training was distributed over four GPUs, inference requires only a single commodity GPU. Our system also remains robust to optical-character-recognition (OCR) noise, showing minimal accuracy degradation when switching from ground-truth to automated text extraction. These results confirm that a Transformer-based cross-attention architecture with directional spatial awareness provides an effective and deployable solution for automated pill–prescription verification.
Keywords
Pill–prescription matching; multimodal learning; vision transformer; relational graph attention; cross-attention; InfoNCE; medication safety
Open Access
Login
Register
Submit a Paper
Propose a Special lssue
Download PDF
Downloads
Citation Tools
