LaRP-CLIP: Layer-Aware Refinement with Prototype Guidance for Zero-Shot Anomaly Detection

1  Introduction

Anomaly detection aims to identify samples that deviate from normal patterns and plays an important role in industrial quality inspection, medical image analysis, and intelligent manufacturing [1–5]. Traditional supervised anomaly detection methods usually require a large number of normal and anomalous samples for each new product category during training [6–8]. In real industrial scenarios, however, data collection is costly, privacy constraints are common, and anomalous samples are often extremely scarce [9,10]. Therefore, this paper focuses on zero-shot anomaly detection, where the goal is to detect anomalies in unseen target domains without requiring target-domain training samples [11–15].

To reduce dependence on target-domain annotation, zero-shot anomaly detection (ZSAD) has attracted increasing attention in recent years [16,17]. This paradigm relies on pre-trained vision-language models, such as Contrastive Language-Image Pre-training (CLIP) [18], and performs anomaly detection without requiring anomalous samples from the target domain. Among existing ZSAD approaches, CLIP-based methods have drawn particular interest because of their strong cross-modal representation capability [19–24]. As shown in Fig. 1a, these methods usually extract multi-layer features from a ViT backbone, compute similarity scores with a shared set of text prompts to obtain layer-wise anomaly maps, fuse these maps for pixel-level prediction, and use global image features for image-level anomaly scoring.

Figure 1: Typical pipeline and three major limitations of existing CLIP-based zero-shot anomaly detection methods. (a) Overall workflow. (b) Pixel-level limitations: all layers share the same text prompts and use fixed fusion. (c) Image-level limitation: reliance on global image-text similarity.

Despite recent progress, existing CLIP-based ZSAD methods still exhibit three major limitations. At the pixel level, all feature layers typically share the same set of text prompts, as illustrated in Fig. 1b. This design is suboptimal because shallow layers mainly capture texture details, whereas deep layers encode higher-level structure and semantics. Using the same prompts across all layers therefore introduces a mismatch between prompt semantics and feature characteristics, which degrades the quality of the resulting anomaly maps. Moreover, layer-wise anomaly maps are usually combined by a fixed fusion strategy that cannot adapt to the content of each input image. Responses from different layers may also conflict on the same defect, resulting in blurred boundaries and increased noise in the fused maps.

At the image level, existing methods usually rely only on global image–text similarity, as shown in Fig. 1c. When the input image contains a complex background or normal samples exhibit slight appearance variations, the global similarity score can be easily affected by irrelevant regions or benign changes. This weakens the robustness of image-level anomaly scoring.

To address these limitations, LaRP-CLIP is proposed. The main contributions of this paper are summarized as follows:

•   Layer-Aware Prompt Decoupling: The learnable prompts are divided into three independent groups, namely Global Prompts, Shallow Prompts, and Deep Prompts. This design allows different feature layers to interact with prompts that better match their semantic characteristics, thereby reducing the mismatch between shallow-layer texture information and deep-layer semantic representations.

•   Adaptive Weighted Fusion and Local Refinement: Learnable layer-wise weights are introduced to adaptively fuse the anomaly maps from multiple layers. In addition, an Error-Prior Guided Local Relation Refiner is designed to perform local correction by modeling inter-layer response discrepancies, producing anomaly maps with clearer boundaries and less noise.

•   Prototype Branch: During inference, high-confidence normal patch features are selected from the current test image to construct an image-specific normal prototype. This prototype is then combined with the global similarity score, which helps reduce the influence of complex backgrounds and normal appearance variations and improves the robustness of image-level anomaly scoring.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the proposed method. Section 4 reports the experimental results and analysis. Section 5 concludes the paper.

2  Related Work

2.1 Traditional Anomaly Detection Methods

Traditional anomaly detection methods are generally divided into supervised and unsupervised paradigms. Supervised methods rely on a sufficient number of normal and anomalous samples for each category during training [3,25]. In industrial practice, however, anomalous samples are often scarce, annotation is expensive, and data sharing may be restricted by privacy or production constraints. These factors greatly hinder the rapid deployment of anomaly detection models on new product lines.

Unsupervised methods are usually trained with normal samples only and identify anomalies through reconstruction error [26], autoencoding mechanisms [27], or one-class modeling [28]. Although such methods avoid the need for anomalous training data, they are still tied to the distribution of the training domain. Once the product type, surface texture, or imaging condition changes, their performance often degrades noticeably. This weak cross-domain generalization makes them less suitable for settings that require rapid deployment on unseen categories.

2.2 Zero-Shot Anomaly Detection

To reduce dependence on target-domain annotation, zero-shot anomaly detection (ZSAD) has received increasing attention in recent years [29,30]. The goal of ZSAD is to detect anomalies in a target domain without using anomalous samples from that domain during training [16]. Earlier studies mainly relied on reconstruction-based schemes or distribution modeling. In complex industrial scenes, however, reconstruction quality is easily affected by normal appearance variation, which weakens the distinction between normal samples and true anomalies [17,18].

With the development of large-scale foundation models, recent ZSAD research has shifted toward transfer-based solutions. Instead of learning anomaly-specific representations from target-domain data, these methods attempt to transfer general visual or cross-modal knowledge acquired during large-scale pre-training to downstream anomaly detection tasks. This shift has made vision-language models an important direction for zero-shot anomaly detection.

ZSAD is related to zero-shot learning (ZSL), where semantic knowledge is transferred to recognize unseen classes. Recent ZSL studies, such as SVIP [31], further improve visual-semantic alignment by modeling semantically contextualized visual patches. However, unlike ZSL that focuses on unseen category recognition, ZSAD aims to detect category-agnostic abnormal patterns in unseen domains. LaRP-CLIP adapts this semantic transfer principle to anomaly detection through normal/abnormal cross-modal alignment, layer-aware prompt decoupling, and image-specific prototype construction.

2.3 CLIP-Based Zero-Shot Anomaly Detection Methods

The emergence of CLIP [18] has accelerated the development of CLIP-based ZSAD methods because of its strong image–text alignment capability. WinCLIP [19] is an early representative method that performs zero-shot anomaly classification and segmentation through a window-based inference scheme and prompt-template ensemble. AnomalyCLIP [20] further introduces object-agnostic learnable prompts and optimizes them with both image-level and pixel-level supervision, showing that CLIP can be adapted more effectively to zero-shot anomaly detection.

Subsequent studies have mainly focused on improving prompt design, feature utilization, and anomaly discrimination. AA-CLIP [24] strengthens anomaly awareness in both the textual and visual spaces, thereby improving the discrimination between normal and abnormal patterns. GenCLIP [21] further explores prompt generalization and multi-layer feature usage to improve transferability across datasets. Despite this progress, most existing methods still rely on a single prompt form across different layers and use fixed fusion strategies, which limits their ability to handle layer-wise semantic differences and image-specific normal variations.

Table 1 summarizes representative CLIP-based ZSAD methods and highlights the differences from LaRP-CLIP.

Decoupled prompts and local refinement have been explored in prompt learning and segmentation literature but are not claimed as isolated contributions. The novelty of LaRP-CLIP lies in adapting these ideas for CLIP-based ZSAD: prompt decoupling is driven by the semantic hierarchy of ViT layers, local refinement is guided by inter-layer anomaly-response disagreement, and image-level scoring is enhanced by an inference-time normal prototype. This approach directly addresses cross-layer semantic mismatch, fixed map fusion, and unreliable global-only scoring in existing CLIP-based ZSAD methods.

3  Method

3.1 Overall Architecture

LaRP-CLIP is designed for both pixel-level anomaly localization and image-level anomaly classification in the zero-shot setting. As shown in Fig. 2, given an input image x∈R3×H×W, the visual encoder extracts a global image feature together with multi-layer local patch features. Meanwhile, the prompt learner generates text prompts for different semantic levels, which are then projected into a unified semantic space by the text encoder. The method contains two complementary branches: a pixel-level branch for anomaly localization and an image-level branch for robust anomaly scoring.

Figure 2: LaRP-CLIP consists of three main components: layer-aware prompt decoupling (left), adaptive weighted fusion with local refinement (middle), and the prototype branch for robust image-level scoring (right).

Let the global feature extracted by the visual encoder be denoted by fg∈RD, and let the selected multi-layer patch features be

F={F1,F2,…,FL},Fl∈RNl×D,(1)

where Nl is the number of patches at the l-th layer and D is the feature dimension.

The pixel-level branch is built upon three steps: layer-aware semantic alignment, adaptive cross-layer fusion, and local refinement. To better match the heterogeneous semantics of different ViT layers, the local prompts are decoupled into shallow prompts and deep prompts, allowing patch features from different layers to be aligned with anomaly semantics at suitable granularities. Based on the resulting layer-wise anomaly responses, learnable layer weights are introduced to perform adaptive fusion across layers. A lightweight Error-Prior Guided Local Relation Refiner is then applied to the fused anomaly map to suppress noisy responses and improve local boundary quality.

The image-level branch is introduced to improve the robustness of anomaly scoring. During inference, a Prototype Branch constructs an image-specific normal prototype from high-confidence normal patches selected from the current test image. This prototype serves as an intra-image normal reference and is combined with the global similarity score, so that image-level anomaly prediction becomes less sensitive to background clutter and benign appearance variation.

3.2 Layer-Aware Local Prompt Learning

Local anomaly representations exhibit clear hierarchical differences across ViT layers. Shallow features mainly encode edges, textures, and other high-frequency local patterns, and are therefore more suitable for describing fine-grained anomalies such as scratches, cracks, and stains. By contrast, deep features place greater emphasis on structure, shape, and semantic consistency, making them more suitable for structural anomalies such as missing parts, misalignment, and deformation. If all layers share a single set of local prompts, features from different semantic levels are forced into the same text space, which leads to a clear semantic granularity mismatch.

To alleviate this issue, LaRP-CLIP divides the learnable prompts into three groups, namely global prompts, shallow prompts, and deep prompts. After the text encoder, the corresponding text embeddings are written as

[(g^n,g^a),(s^n,s^a),(d^n,d^a)]∈R2×D,(2)

where the superscripts n and a denote the normal and abnormal classes, respectively.

The layers used for local anomaly modeling are divided into a shallow set S and a deep set D, satisfying

S∪D={1,2,…,L},S∩D=∅.(3)

For the patch features Fl at the l-th layer, each patch feature is first normalized by

f^il=fil‖fil‖2,(4)

where fil denotes the i-th patch feature at layer l. The text embeddings are normalized in the same way. If l∈S, the shallow prompt embeddings (s^n,s^a) are used; if l∈D, the deep prompt embeddings (d^n,d^a) are used.

The similarity between the i-th patch and the corresponding normal or abnormal text embedding is defined as

zi,cl=τ(f^il)⊤e^cl,c∈{n,a},(5)

where τ is the temperature coefficient and e^cl denotes the text embedding assigned to layer l. The anomaly response of the i-th patch is then obtained by binary softmax:

pi,al=exp⁡(zi,al)exp⁡(zi,nl)+exp⁡(zi,al).(6)

The anomaly map of the l-th layer is written as

Ml=Reshape⁡({pi,al}i=1Nl).(7)

For image-level scoring, the global feature interacts only with the text embeddings corresponding to the global prompts. The resulting text-driven anomaly score is defined as

stext=exp⁡(τgf^g⊤g^a)exp⁡(τgf^g⊤g^n)+exp⁡(τgf^g⊤g^a).(8)

With this design, shallow patch features are aligned with shallow anomaly semantics, deep patch features are aligned with deep anomaly semantics, and the global feature is used for image-level anomaly judgment. In this way, the cross-modal alignment process becomes layer-aware, which helps reduce the semantic mismatch introduced by using a shared prompt across all feature layers.

3.3 Layer-Adaptive Fusion and Local Relation Refinement

The layer-wise anomaly maps {Ml}l=1L differ in detail sensitivity, structural stability, and noise level. Fixed fusion or equal-weight averaging cannot fully exploit the complementary information carried by different layers. To address this issue, LaRP-CLIP introduces a set of learnable fusion parameters {αl}l=1L, which are normalized by softmax to obtain the layer weights:

ωl=exp⁡(αl)∑k=1Lexp⁡(αk),l=1,…,L.(9)

The fused anomaly map is then computed as

Mfused=∑l=1LωlMl.(10)

This design allows the contribution of each layer to be learned from data, rather than being fixed in advance.

Although adaptive fusion improves the integration of multi-layer responses, the fused map may still contain local noise or imprecise boundaries. This issue becomes more evident when different layers produce inconsistent responses in the same region. To capture such disagreement, an error prior is introduced to measure the deviation of each layer from the fused result. For the l-th layer, the deviation map is defined as

El=|Ml−Mfused|.(11)

The aggregated error prior is then written as

Eprior=∑l=1LωlEl.(12)

The error prior provides a local measure of cross-layer inconsistency. Regions with similar responses across layers tend to have small values, whereas regions with conflicting responses yield larger values. Based on this cue, LaRP-CLIP concatenates the fused anomaly map with the error prior and feeds them into a lightweight local relation refinement module. This module adopts a 3×3 convolutional structure to predict a residual correction:

ΔM=ψ([Mfused,Eprior]).(13)

The final pixel-level anomaly map is given by

Mfinal=Mfused+ΔM.(14)

The refinement module is used to correct local errors in the fused anomaly map rather than to regenerate the entire prediction. It mainly adjusts regions with blurred boundaries, discontinuous responses, or noticeable local noise, while preserving the main structure provided by the preceding layer-wise fusion stage.

3.4 Prototype Branch

Relying only on global text matching is often insufficient for stable image-level anomaly scoring, because it does not fully reflect the local distribution of anomaly evidence within an image. This limitation becomes more apparent when anomalous regions are small, the background is complex, or normal samples exhibit noticeable appearance variation. To improve the robustness of image-level scoring, a Prototype Branch is introduced during inference. This branch constructs a normal prototype from high-confidence normal regions in the current test image and uses it as an additional reference for anomaly judgment.

The layer-wise anomaly maps are first accumulated to obtain a text-driven anomaly map:

Mtext=∑l=1LMl.(15)

The patch features from the last layer, denoted by {pi}i=1N, are then used as local semantic representations. The map Mtext is downsampled to the patch grid to produce an anomaly score qi for each patch. Based on these scores, a set of high-confidence normal patches is selected, and its index set is denoted by 𝒩. The normal prototype is computed as

pnorm=∑i∈𝒩ωipi∑i∈𝒩ωi,(16)

where ωi is derived from qi, so that patches with lower anomaly scores receive larger weights.

The deviation of each patch from the normal prototype is then measured by

μi=1−cos⁡(pi,pnorm).(17)

These deviations are reshaped into a two-dimensional map to form the prototype-guided anomaly map, which is further combined with the text-driven anomaly map to obtain an auxiliary anomaly map for image-level scoring. Since image-level prediction is more influenced by the most salient anomalous regions than by the average response over the whole image, top-k pooling is applied to the fused map to obtain the auxiliary score smap. The final image-level anomaly score is defined as

sfinal=stext+λsmap1+λ,(18)

where λ is a balancing coefficient.

Unlike a fixed normal template shared across all images, the Prototype Branch derives a normal reference directly from the current test image. This makes the image-level score less sensitive to background clutter and benign appearance variation. The branch is used only for image-level scoring and does not affect the generation of pixel-level anomaly maps.

The Prototype Branch is excluded from training because it is designed as a non-parametric test-time adaptation module. It constructs an image-specific normal prototype from high-confidence normal patches in each test image, rather than learning a fixed prototype from auxiliary training data.

3.5 Training Objective

To jointly optimize image-level anomaly discrimination and pixel-level anomaly localization, LaRP-CLIP uses a combined training objective. The overall loss is defined as

ℒ=ℒimg+ℒpixel,(19)

where ℒimg is the image-level loss and ℒpixel is the pixel-level loss.

For image-level supervision, let y∈{0,1} denote the image label and let stext denote the global text-driven anomaly score. The image-level loss is defined by binary cross-entropy:

ℒimg=−ylog⁡stext−(1−y)log⁡(1−stext).(20)

For pixel-level supervision, the intermediate outputs are not treated as independent prediction targets. Instead, the loss is divided according to the two stages of the pixel-level branch, namely alignment and refinement:

ℒpixel=ℒalign+λrℒref,(21)

where ℒalign supervises the front-end layer-aware semantic alignment and adaptive fusion, ℒref supervises the refinement stage, and λr is the weight assigned to the refinement term.

The alignment loss is applied to the shallow-layer anomaly maps, the deep-layer anomaly maps, and the fused anomaly map:

ℒalign=1|S|∑l∈Sℒseg(Ml,Y)+1|D|∑l∈Dℒseg(Ml,Y)+λfℒseg(Mfused,Y),(22)

where Y is the pixel-level ground-truth mask, Ml is the anomaly map at layer l, Mfused is the adaptively fused anomaly map, and S and D denote the shallow-layer set and deep-layer set, respectively. The function ℒseg(⋅) denotes the segmentation loss, implemented as a combination of focal loss and dual-channel Dice loss for the normal and abnormal classes. The coefficient λf controls the relative contribution of the fused-map supervision.

This design serves two purposes. First, separate supervision on shallow and deep anomaly maps encourages the decoupled prompts to learn anomaly semantics at different granularities. Second, direct supervision on Mfused ensures that the fusion stage itself produces a discriminative anomaly map before refinement.

The refinement loss is applied only to the final anomaly map:

ℒref=ℒseg(Mfinal,Y)+λΔ‖Mfinal−Mfused‖1,(23)

where λΔ is the weight of the residual regularization term. This term limits the difference between the refined anomaly map and the fused anomaly map, thereby constraining the magnitude of the correction introduced by the refinement module.

The Prototype Branch is used only during inference and is therefore not included in the training objective.

3.6 Training and Inference Procedure

To improve the readability and reproducibility of the proposed method, the pseudo code of LaRP-CLIP is provided in Algorithm 1. The algorithm summarizes the complete training and inference procedure, including layer-aware prompt matching, adaptive fusion with local refinement, and the Prototype Branch for image-level scoring. During training, the loss in Eq. (19) is used to optimize the learnable parameters. During inference, all learnable parameters are fixed, and only the forward pass is performed to obtain the final anomaly scores.

4  Experiments

4.1 Experimental Settings

The experiments are evaluated on the MVTec AD [32], VisA [23], MPDD [33], and BTAD [34] industrial defect datasets, as well as the HeadCT [35], BrainMRI [35], Br35H [36], ISIC [37], CVC-ColonDB [38], CVC-ClinicDB [38], and TN3K [39] medical imaging datasets. For evaluation metrics, pixel-level performance is measured by pixel-level AUROC and PRO, while image-level performance is measured by image-level AUROC and AP.

All experiments were conducted on a single NVIDIA RTX 4090 GPU (24 GB VRAM) using Python 3.12.7 and the PyTorch 2.0.0 framework. The Adam optimizer was employed with an initial learning rate of 1×10−3, a batch size of 8, and a total of 15 training epochs. To ensure reproducibility, the random seed was fixed at 111. The visual backbone network adopts the pre-trained CLIP model (ViT-L/14@336px), with the input resolution uniformly set to 518×518. During training, only the parameters of the learnable prompt learner are optimized, while the CLIP visual encoder remains frozen.

The prompt learner uses a learnable prompt length of 12, a text embedding depth of 9, and feature layers selected as [6,12,18,24]. The temperature coefficient is τ=0.07, the balancing coefficient is λ=1.0, λf=1, λr=0.5, and λΔ=0.2. The Prototype Branch is enabled during the testing phase with top-k set to 8.

4.2 Main Results

Tables 2 and 3 summarize the quantitative comparison between LaRP-CLIP and state-of-the-art zero-shot anomaly detection methods on industrial defect datasets and medical imaging datasets, respectively. The highest value of each metric (AUROC or AP/PRO) is highlighted in red and the second-highest value is highlighted in blue.

Fig. 3 presents a qualitative comparison of anomaly maps produced by different prompt strategies. The three prompts show notably different response patterns across samples. The Shallow Prompt is more sensitive to local appearance variations and can better preserve fine-grained details, making it effective for subtle texture anomalies. The Deep Prompt generates more compact and semantically aligned activation on abnormal regions, which is beneficial for anomalies with relatively clear structural semantics. In contrast, the Global Prompt frequently produces diffuse or less discriminative responses, leading to incomplete localization or increased background interference. This comparison indicates that different ViT layers capture anomaly cues at different semantic granularities, while the proposed layer-aware prompt decoupling offers a more suitable mechanism for aligning prompt representations with layer-specific features.

Figure 3: Qualitative comparison of anomaly maps generated using different prompts. From left to right: input image, ground-truth mask, anomaly map using shallow prompt, deep prompt, and global prompt.

Fig. 4 presents qualitative anomaly localization results of LaRP-CLIP on representative samples from both industrial and medical domains. The third row shows the fused anomaly maps obtained after multi-layer aggregation, while the fourth row shows the refined anomaly maps after applying the Error-Prior Guided Local Relation Refiner. As observed, the fused maps can already roughly highlight the abnormal regions, but they still contain scattered responses and imprecise boundaries in several cases. After refinement, the anomaly regions become more compact and better aligned with the ground-truth masks, with reduced background interference and clearer structural outlines. This improvement is consistently visible in both industrial samples with thin or irregular defects and medical samples with relatively compact lesion regions, indicating that the refinement stage is beneficial for producing more precise pixel-level localization.

Figure 4: Qualitative anomaly localization results of LaRP-CLIP on representative samples from industrial and medical domains. From top to bottom: input image, ground-truth mask, fused anomaly map, and refined anomaly map. The refined maps exhibit clearer boundaries and reduced background interference compared with the fused maps.

Fig. 5 presents a qualitative comparison between the global similarity map and the prototype-guided similarity map. The selected high-confidence normal prototype regions in the second column are used to build an image-specific normal reference for the current test sample. Compared with the global similarity map, which tends to exhibit scattered or distracting responses on normal structures, the prototype-guided similarity map yields cleaner activations and better highlights the actual abnormal regions. The improvement is visible across samples from both industrial and medical domains, indicating that the Prototype Branch helps reduce the influence of normal appearance variation and complex background content during anomaly localization.

Figure 5: Qualitative results showing the effect of the prototype branch. From left to right: input image, selected high-confidence normal prototype regions (red dots), global similarity map, and prototype-guided similarity map.

4.3 Ablation Study

To assess the contribution of each component in LaRP-CLIP, an ablation study is conducted on MVTec AD. The results are reported in Table 4, where image-level performance is evaluated by AUROC and AP, and pixel-level performance is evaluated by AUROC and PRO.

As shown in Table 4, removing any component leads to a decline in performance, indicating that each module contributes to the overall capability of the framework. Among all variants, removing the Layer-Aware Prompt Decoupling causes the largest performance drop, with image-level results decreasing from (92.8, 96.5) to (88.4, 93.7) and pixel-level results decreasing from (91.5, 85.8) to (87.9, 80.2). This result suggests that aligning prompt representations with the semantic granularity of different ViT layers is critical for both anomaly classification and localization, which is also consistent with the qualitative comparison in Fig. 3.

When the Adaptive Weighted Fusion is removed, the performance decreases to (89.7, 94.9) at the image level and (89.1, 82.6) at the pixel level, suggesting that fixed fusion is less effective for aggregating anomaly cues from multiple layers. Removing the Local Relation Refiner also results in a noticeable decline, particularly in pixel-level localization, where the PRO score drops from 85.8 to 81.9. This observation is consistent with the qualitative comparison in Fig. 4, where the refinement stage produces cleaner anomaly regions and clearer boundaries.

By comparison, removing the Prototype Branch leads to a relatively smaller drop, from (92.8, 96.5) to (90.5, 95.4) at the image level and from (91.5, 85.8) to (91.0, 85.1) at the pixel level. Although its effect is less pronounced than that of the other modules, it still brings consistent gains, indicating that image-specific normal prototypes help stabilize anomaly scoring under appearance variation. This trend is also supported by the visual comparison in Fig. 5, where the prototype-guided similarity maps show more concentrated responses on abnormal regions and less interference from normal structures.

The results reveal clear performance degradation when any component is removed. Removing layer-aware prompt decoupling leads to the largest drop, decreasing image-level AUROC by 4.4% and pixel-level AUROC by 3.6%, confirming its critical role in resolving semantic granularity mismatch across layers. Removing the Error-Prior Guided Local Relation Refiner causes noticeable declines in pixel-level metrics (AUROC drops by 3.1% and PRO by 4.0%), highlighting its importance in sharpening boundaries and suppressing noise. Eliminating the Prototype Branch primarily affects image-level robustness, while equal-weight fusion and the absence of error prior both result in suboptimal fusion quality.

To further evaluate the influence of key hyperparameters, sensitivity analyses are conducted on MVTec AD. Three factors are considered: the balancing coefficient λ, the top-k value used in prototype-guided pooling, and the shallow/deep layer partition. Since λ and top-k directly affect the final image-level anomaly score, their influence is evaluated using image-level AUROC and AP. The shallow/deep layer partition affects layer-aware prompt matching and anomaly map generation, and is therefore evaluated using pixel-level AUROC and PRO.

As shown in Table 5, LaRP-CLIP remains stable with different λ and top-k values. When λ=0, the score relies only on global text similarity, leading to weaker detection. Increasing λ incorporates local anomaly evidence, improving performance, but too large a λ overemphasizes local responses, slightly degrading performance. The best result is achieved at λ=1.0, where global and local evidence complement each other.

For top-k, a very small value is more sensitive to noisy local activations, while a very large value may include normal background regions and dilute anomaly evidence. The setting top-k=8 achieves the best performance and is therefore adopted as the default configuration.

Table 6 shows that assigning only layer 6 to the shallow group lacks sufficient texture supervision, while assigning layer 18 weakens the distinction between shallow and deep prompts. The best pixel-level performance is achieved with the partition [6,12]/[18,24], aligning with the ViT feature hierarchy: shallow layers capture texture, while deep layers encode structure and semantics.

4.4 Time Complexity Analysis

Let N denote the number of image patches and L denote the number of selected feature layers. In LaRP-CLIP, the extra computation beyond the frozen CLIP backbone comes from layer-wise prompt matching, adaptive fusion, refinement, and prototype construction.

Layer-aware prompt matching and patch–text similarity computation across L layers require O(LN). Adaptive weighted fusion and error-prior estimation are also performed on layer-wise anomaly responses, with complexity O(LN). The Local Relation Refiner is implemented using lightweight 3×3 convolutions and has complexity O(N). The Prototype Branch, including high-confidence patch selection and prototype construction, also scales as O(N).

The total additional complexity is therefore O(LN), excluding the frozen backbone. This is on the same order as existing CLIP-based methods that rely on multi-layer inference. On a single NVIDIA RTX 4090 GPU, with an input resolution of 518×518, LaRP-CLIP takes about 280 ms to process one image during inference.

5  Conclusion

This paper presents LaRP-CLIP, a zero-shot anomaly detection framework for industrial and medical images. The framework introduces layer-aware prompt decoupling, adaptive weighted fusion with an Error-Prior Guided Local Relation Refiner, and a Prototype Branch for image-specific normal prototype construction. These designs are intended to alleviate three common issues in existing CLIP-based anomaly detection methods, namely semantic granularity mismatch across layers, inflexible fusion across feature levels, and interference from background or normal appearance variations.

Experimental results on multiple industrial defect and medical imaging datasets show that LaRP-CLIP achieves strong performance in both image-level detection and pixel-level localization. The qualitative results further show that the proposed framework produces cleaner anomaly responses and more precise boundaries in challenging cases. These results suggest that LaRP-CLIP is a practical zero-shot solution for anomaly detection in scenarios where anomalous samples are limited or unavailable. Future work will consider extending the framework to multi-class anomaly settings and to more complex real-world environments.

Acknowledgement: Not applicable.

Funding Statement: This research was funded by the Key Research and Development Program of Zhejiang Province No. 2023C01141, the Science and Technology Innovation Community Project of Yangtze River Delta No. 23002410100.

Author Contributions: Xing Fang: Responsible for software implementation, experimental validation, and preparation of the original manuscript draft. Yuanfang Chen: Led the conceptualization and methodological design, provided project administration and supervision, contributed to manuscript revision and editing, and secured funding. Qiang Lin: Conducted data collection. Kun Yang: Offered methodological guidance, technical support, and constructive feedback during manuscript review. Gyu Myoung Lee: Provided conceptual guidance and supervision. All authors reviewed and approved the final version of the manuscript.

Availability of Data and Materials: All datasets used in this study are publicly accessible, and links to the original data sources are included in the manuscript for reference.

Ethics Approval: Not applicable.

Conflicts of Interest: The authors declare no conflicts of interest.

References

1. Wu P, Pan C, Yan Y, Pang G, Yan Q, Wang P, et al. Deep learning for video anomaly detection: a review. IEEE Trans Neural Netw Learn Syst. 2026. doi:10.1109/TNNLS.2025.3647892. [Google Scholar] [PubMed] [CrossRef]

2. Huang H, Wang P, Pei J, Wang J, Alexanian S, Niyato D. Deep learning advancements in anomaly detection: a comprehensive survey. IEEE Internet Things J. 2025;12(21):44318–42. doi:10.1109/JIOT.2025.3585884. [Google Scholar] [CrossRef]

3. Li Z, Yan Y, Wang X, Ge Y, Meng L. A survey of deep learning for industrial visual anomaly detection. Artif Intell Rev. 2025;58(9):279. doi:10.1007/s10462-025-11287-7. [Google Scholar] [CrossRef]

4. Ammar MB, Mendoza A, Belkhir N, Manzanera A, Franchi G. Foundation models and Transformers for anomaly detection: a survey. Inf Fusion. 2025;126(7):103517. doi:10.1016/j.inffus.2025.103517. [Google Scholar] [CrossRef]

5. Laghari AA, Khan AA, Ksibi A, Hajjej F, Kryvinska N, Almadhor A, et al. A novel and secure artificial intelligence enabled zero trust intrusion detection in industrial internet of things architecture. Sci Rep. 2025;15(1):26843. doi:10.1038/s41598-025-11738-9. [Google Scholar] [PubMed] [CrossRef]

6. Xie S, Wu X, Wang MY. Semi-patchcore: a novel two-staged method for semi-supervised anomaly detection and localization. IEEE Trans Instrum Meas. 2025;74:3506012. doi:10.1109/TIM.2025.3527588. [Google Scholar] [CrossRef]

7. Li R, Ma H, Wang R, Song H, Zhou X, Wang L, et al. Application of unsupervised learning methods based on video data for real-time anomaly detection in wire arc additive manufacturing. J Manuf Process. 2025;143:37–55. doi:10.1016/j.jmapro.2025.03.113. [Google Scholar] [CrossRef]

8. Koren O, Koren M, Peretz O. A procedure for anomaly detection and analysis. Eng Appl Artif Intell. 2023;117(2):105503. doi:10.1016/j.engappai.2022.105503. [Google Scholar] [CrossRef]

9. Nizam H, Zafar S, Lv Z, Wang F, Hu X. Real-time deep anomaly detection framework for multivariate time-series data in industrial IoT. IEEE Sens J. 2022;22(23):22836–49. doi:10.1109/JSEN.2022.3211874. [Google Scholar] [CrossRef]

10. Liu L, Li J, Lv J, Wang J, Zhao S, Lu Q. Privacy-preserving and secure industrial big data analytics: a survey and the research framework. IEEE Internet Things J. 2024;11(11):18976–99. doi:10.1109/JIOT.2024.3353727. [Google Scholar] [CrossRef]

11. Aich A, Peng KC, Roy-Chowdhury AK. Cross-domain video anomaly detection without target domain adaptation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision; 2023 Jan 2–7; Waikoloa, HI, USA. p. 2579–91. doi:10.1109/WACV56688.2023.00261. [Google Scholar] [CrossRef]

12. Hashemi MJ, Keller E, Tizpaz-Niari S. Detecting unseen anomalies in network systems by leveraging neural networks. IEEE Trans Netw Serv Manag. 2022;20(3):2515–28. doi:10.1109/TNSM.2022.3220775. [Google Scholar] [CrossRef]

13. Yang Z, Soltani I, Darve E. Anomaly detection with domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2023 Jun 17–24; Vancouver, BC, Canada. p. 2958–67. doi:10.1109/CVPRW59228.2023.00297. [Google Scholar] [CrossRef]

14. Cho M, Kim T, Shim M, Wee D, Lee S. Towards multi-domain learning for generalizable video anomaly detection. Adv Neural Inf Process Syst. 2024;37:50256–84. doi:10.52202/079017-1591. [Google Scholar] [CrossRef]

15. Mao W, Wang G, Kou L, Liang X. Deep domain-adversarial anomaly detection with one-class transfer learning. IEEE/CAA J Autom Sin. 2023;10(2):524–46. doi:10.1109/JAS.2023.123228. [Google Scholar] [CrossRef]

16. Li A, Qiu C, Kloft M, Smyth P, Rudolph M, Mandt S. Zero-shot anomaly detection via batch normalization. Adv Neural Inf Process Syst. 2023;36:40963–93. doi:10.52202/075280-1784. [Google Scholar] [CrossRef]

17. Peng Y, Lin X, Ma N, Du J, Liu C, Liu C, et al. SAM-LAD: segment anything model meets zero-shot logic anomaly detection. Knowl-Based Syst. 2025;314(4):113176. doi:10.1016/j.knosys.2025.113176. [Google Scholar] [CrossRef]

18. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: Proceedings of the International Conference on Machine Learning; 2021 Jul 18–24; Virtually. p. 8748–63. doi:10.48550/arXiv.2103.00020. [Google Scholar] [CrossRef]

19. Jeong J, Zou Y, Kim T, Zhang D, Ravichandran A, Dabeer O. WinCLIP: zero-/few-shot anomaly classification and segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2023 Jun 17–24; Vancouver, BC, Canada. p. 19606–16. doi:10.1109/CVPR52729.2023.01878. [Google Scholar] [CrossRef]

20. Zhou Q, Pang G, Tian Y, He S, Chen J. AnomalyCLIP: object-agnostic prompt learning for zero-shot anomaly detection. In: Proceedings of the International Conference on Learning Representation; 2023 May 1–5; Kigali, Rwanda. doi:10.48550/arXiv.2310.18961. [Google Scholar] [CrossRef]

21. Kim D, Park C, Cho S, Lim H, Kang M, Lee J, et al. Generalizing CLIP prompts for zero-shot anomaly detection. Pattern Recognit. 2026;178(9):113406. doi:10.1016/j.patcog.2026.113406. [Google Scholar] [CrossRef]

22. Chen X, Han Y, Zhang J. A zero-/few-shot anomaly classification and segmentation method for CVPR 2023 (VAND) workshop challenge tracks 1&2: 1st Place on Zero-shot AD and 4th place on few-shot AD. arXiv:2305.17382. 2023. doi:10.48550/arXiv.2305.17382. [Google Scholar] [CrossRef]

23. Zou Y, Jeong J, Pemula L, Zhang D, Dabeer O. Spot-the-difference self-supervised pre-training for anomaly detection and segmentation. In: Proceedings of the European Conference on Computer Vision; 2022 Oct 23–27; Tel Aviv, Israel. Cham, Switzerland: Springer; 2022. p. 392–408. doi:10.1007/978-3-031-20056-4_23. [Google Scholar] [CrossRef]

24. Ma W, Zhang X, Yao Q, Tang F, Wu C, Li Y, et al. AA-CLIP: enhancing zero-shot anomaly detection via anomaly-aware CLIP. In: Proceedings of the Computer Vision and Pattern Recognition Conference; 2025 Jun 10–17; Nashville, TN, USA. p. 4744–54. doi:10.1109/CVPR52734.2025.00447. [Google Scholar] [CrossRef]

25. Ruff L, Kauffmann JR, Vandermeulen RA, Montavon G, Samek W, Kloft M, et al. A unifying review of deep and shallow anomaly detection. Proc IEEE. 2021;109(5):756–95. doi:10.1109/JPROC.2021.3052449. [Google Scholar] [CrossRef]

26. Gong D, Liu L, Le V, Saha B, Mansour MR, Venkatesh S, et al. Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019 Oct 27–Nov 2; Seoul, Republic of Korea. p. 1705–14. doi:10.1109/ICCV.2019.00179. [Google Scholar] [CrossRef]

27. Schlegl T, Seeböck P, Waldstein SM, Langs G, Schmidt-Erfurth U. f-AnoGAN: fast unsupervised anomaly detection with generative adversarial networks. Med Image Anal. 2019;54(3):30–44. doi:10.1016/j.media.2019.01.010. [Google Scholar] [PubMed] [CrossRef]

28. Ruff L, Vandermeulen R, Goernitz N, Deecke L, Siddiqui SA, Binder A, et al. Deep one-class classification. In: Proceedings of the International Conference on Machine Learning; 2018 Jul 10–15; Stockholm, Sweden. p. 4393–402. [Google Scholar]

29. Ren J, Tang T, Jia H, Xu Z, Fayek H, Li X, et al. Foundation models for anomaly detection: vision and challenges. AI Mag. 2025;46(4):e70045. doi:10.1002/aaai.70045. [Google Scholar] [CrossRef]

30. Li Y, Goodge A, Liu F, Foo CS. PromptAD: zero-shot anomaly detection using text prompts. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision; 2024 Jan 3–8; Waikoloa, HI, USA. p. 1093–102. doi:10.1109/WACV57701.2024.00113. [Google Scholar] [CrossRef]

31. Chen Z, Zhao Z, Guo J, Li J, Huang Z. SVIP: semantically contextualized visual patches for zero-shot learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2025 Oct 19–20; Honolulu, HI, USA. p. 3346–56. doi:10.1109/ICCV51701.2025.00320. [Google Scholar] [CrossRef]

32. Bergmann P, Fauser M, Sattlegger D, Steger C. MVTec AD—a comprehensive real-world dataset for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2019 Jun 16–20; Long Beach, CA, USA. p. 9592–600. doi:10.1109/CVPR.2019.00982. [Google Scholar] [CrossRef]

33. Jezek S, Jonak M, Burget R, Dvorak P, Skotak M. Deep learning-based defect detection of metal parts: evaluating current methods in complex conditions. In: Proceedings of the 2021 13th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT); 2021 Oct 25–27; Brno, Czech Republic. New York, NY, USA: IEEE. p. 66–71. doi:10.1109/ICUMT54235.2021.9631567. [Google Scholar] [CrossRef]

34. Mishra P, Verk R, Fornasier D, Piciarelli C, Foresti GL. VT-ADL: a vision transformer network for image anomaly detection and localization. arXiv:2104.10036. 2021. doi:10.48550/arxiv.2104.10036. [Google Scholar] [CrossRef]

35. Salehi M, Sadjadi N, Baselizadeh S, Rohban MH, Rabiee HR. Multiresolution knowledge distillation for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2021 Jun 20–25; Nashville, TN, USA. p. 14902–12. doi:10.1109/CVPR46437.2021.01466. [Google Scholar] [CrossRef]

36. Hamada A. Br35H:: brain tumor detection 2020. IEEE Dataport. 2025. doi:10.21227/t5k6-5r54. [Google Scholar] [CrossRef]

37. Codella NC, Gutman D, Celebi ME, Helba B, Marchetti MA, Dusza SW, et al. Skin lesion analysis toward melanoma detection: a challenge at the 2017 international symposium on biomedical imaging (ISBIhosted by the international skin imaging collaboration (ISIC). In: Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018); 2018 Apr 4–7; Washington, DC, USA. New York, NY, USA: IEEE; 2018. p. 168–72. doi:10.1109/ISBI.2018.8363547. [Google Scholar] [CrossRef]

38. Tajbakhsh N, Gurudu SR, Liang J. Automated polyp detection in colonoscopy videos using shape and context information. IEEE Trans Med Imaging. 2015;35(2):630–44. doi:10.1109/TMI.2015.2487997. [Google Scholar] [PubMed] [CrossRef]

39. Gong H, Chen G, Wang R, Xie X, Mao M, Yu Y, et al. Multi-task learning for thyroid nodule segmentation with thyroid region prior. In: Proceedings of the 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI); 2021 Apr 20–21; Nice, France. New York, NY, USA: IEEE; 2021. p. 257–61. doi:10.1109/ISBI48211.2021.9434087. [Google Scholar] [CrossRef]

40. Zhou K, Yang J, Loy CC, Liu Z. Learning to prompt for vision-language models. Int J Comput Vis. 2022;130(9):2337–48. doi:10.1007/s11263-022-01653-1. [Google Scholar] [CrossRef]