TY  - EJOU
AU  - My, Linh Nguyen Thi 
AU  - Viet, Lap Thai 
AU  - Hieu, Hoai Truong 
AU  - Vo, Tham 
AU  - Hoang, Vinh Truong 

TI  - A Transformer-Based Multimodal Cross-Attention Framework with Relational Graph Attention for Pill–Prescription Matching
T2  - Computers, Materials \& Continua

PY  - 
VL  - 
IS  - 
SN  - 1546-2226

AB  - Matching physical pills with drug names on a prescription is a safety-critical task for preventing medication errors. The pioneering PIMA (PIll-prescription MAtching) framework addressed this problem by aligning convolutional features (ResNet50) with textual embeddings (BERT) using a direction-agnostic GraphSAGE operator and a margin-based contrastive loss. While effective on controlled datasets, PIMA’s real-world deployment is hindered by its reliance on manual ground-truth annotations, sensitivity to background clutter, and a simplistic cosine similarity fusion that lacks selective attention. In this paper, we propose a redesigned, fully multimodal framework featuring five coordinated improvements: (i) two interchangeable visual encoders for the pill-recognition branch—a Vision Transformer (ViT-B/16) applied to background-removed crops, and an end-to-end Faster R-CNN/RoI-Align detector that learns features directly from raw images; (ii) a lightweight all-MiniLM-L6-v2 sentence encoder for efficient pill-name representation; (iii) an integrated PP-OCRv3 module that automates text and bounding box extraction, eliminating manual dependency; (iv) a Relational Graph Attention Network (R-GAT) with a pseudo-classifier gate to model directional spatial logic (e.g., left-to-right drug/quantity relations) and filter irrelevant non-drug text; and (v) a multimodal cross-attention head optimized via an InfoNCE objective for more stable and dense gradient convergence. Experiments on the real-world VAIPE dataset show that both variants substantially outperform a reproduced graph baseline: the ViT-B/16 variant raises the Top-1 matching accuracy from 49.89% to 82.46% and the end-to-end Faster R-CNN variant to 83.47%—absolute gains of 32.57 and 33.58 percentage points (pp), respectively, while placing the correct drug name among the top-five candidates for more than 99% of pills. The two variants trade off accuracy against preprocessing: the Faster R-CNN variant attains the best accuracy while operating directly on raw images, whereas the ViT-B/16 variant requires a separate background-removal step. In terms of computational footprint, the visual backbones contain approximately 86.6M (ViT-B/16) and 41.8M (Faster R-CNN/ResNet50-FPN) parameters, the 22.7M-parameter MiniLM encoder replaces the far heavier BERT of the baseline, and although training was distributed over four GPUs, inference requires only a single commodity GPU. Our system also remains robust to optical-character-recognition (OCR) noise, showing minimal accuracy degradation when switching from ground-truth to automated text extraction. These results confirm that a Transformer-based cross-attention architecture with directional spatial awareness provides an effective and deployable solution for automated pill–prescription verification.
KW  - Pill–prescription matching; multimodal learning; vision transformer; relational graph attention; cross-attention; InfoNCE; medication safety

DO  - 10.32604/cmc.2026.086877