[EN]AI-Assisted Museum Exhibition Classification: Overcoming the Reflection Challenge

📖 Background

With advancements in image recognition and AI technologies, museums are eager to develop more intuitive visitor guides. Ideally, visitors could use their mobile devices to snap photos of exhibits, and the system would instantly recognize the artifacts and provide detailed guidance. However, achieving this vision presents multiple challenges:

1. Glass Reflections: Many artifacts are encased in glass display cases, leading to reflections that obscure the objects and impede accurate AI recognition.
2. Angles and Positions: Visitors may struggle to capture complete images due to positioning or angle constraints.
3. External Interferences: Poor lighting or cluttered backgrounds further complicate image recognition.

To address these challenges, our project developed an AI system that accurately identifies museum exhibits despite these difficulties, especially glass reflections.

---

✨ Our Solution

1. Reflection Removal Model

Background

Reflections from display cases are a significant obstacle in museum environments. Existing open-source reflection removal models were insufficient for handling the complex lighting and reflective scenarios found in museum exhibition halls. To overcome this:

• We developed a specialized reflection removal model that adapts to diverse, non-fixed environments.
• We framed the reflection removal task as a style transformation and employed the Pix2pix GAN model with Sobel Features to guide the model’s attention to reflective areas. This approach effectively minimized reflection interference while preserving key artifact details.

2. Artifact Identification Model

To identify multiple artifacts within a single photo, we used the YOLOv8n model. We manually labeled 2,000 artifact images and used data augmentation techniques (e.g., flipping, scaling, and brightness adjustments) to enhance model robustness.

• We trained two versions of the model: the original YOLOv8n and a reflection-enhanced version that uses images processed by the reflection removal model.

3. Combining Reflection Removal and YOLO Detection

We developed two artifact identification pathways:

1. Original Identification: Direct YOLOv8n detection on the original images.
2. Reflection-Removed Identification: YOLOv8n detection after removing reflections.

By comparing the outputs of both versions, we chose the optimal identification result, ensuring accurate artifact recognition across different environments.

---

🏆 Results & Achievements

Data Outcomes

• V1 Model (Original YOLO): Achieved a classification accuracy of 92.7%.
• V2 Model (With Reflection Removal): Combined accuracy improved to 94.5%, reducing misclassification and enhancing recognition quality.

Qualitative Analysis

• Focus on Reflections: The reflection removal model successfully identified reflective areas and minimized interference while preserving exhibit details.
• Detection Improvement: Combining both reflection removal and YOLO detection reduced misclassifications, especially in environments with high reflections.

---

📲 App Development

We deployed the reflection removal and YOLOv8n models in a dual-platform mobile app built using Android Studio and Xcode. This app allows users to easily capture images of artifacts and get instant, accurate information.

Core Features

1. Camera Functionality: Users can capture exhibit photos effortlessly.
2. Automated Processing: Photos undergo both reflection removal and artifact identification.
3. Interactive Information Display: Detailed exhibit information is displayed, providing an enhanced visitor experience.

App Screenshots

Initial app screen when opened	Camera ready for capturing images

Image after capturing	Results after image processing

📷 Demo

---

🚀 Benefits & Real-World Application

Achievements & Advantages

• Reflection Removal: Improved artifact recognition in museums with glass display cases.
• YOLO Multi-Object Detection: Accurate classification even for multiple artifacts in a single image.
• Dual-Platform App: Available on both iOS and Android, enhancing user accessibility.

Practical Application

1. Museum Experience Enhancement: Visitors can use the app to quickly get information about exhibits, improving interactivity.
2. Wider Cultural and Educational Use: The solution can be expanded to cultural venues such as art galleries and educational institutions.

---

🎯 Future Prospects

1. Model Optimization: Further optimize reflection removal and YOLO models for better accuracy and faster performance.
2. Multi-Language Support: Expand the app’s UI to support multiple languages for international visitors.
3. Broader Applications: Apply the solution beyond museums, such as product photography, academic research, and commercial exhibits.

---

🤝 Summary

This project has successfully enhanced automated museum exhibit identification, overcoming challenges like glass reflections. We look forward to future collaborations with museums and expanding the solution to other sectors to continue advancing AI in cultural heritage.

---

[中文]AI輔助博物館展品分類：克服玻璃反光挑戰

📖 背景

隨著圖像識別技術和AI技術的進步，博物館希望開發更直觀的參觀導覽方式：參觀者可以通過手持行動裝置拍攝展品，系統即時識別展品並提供詳細導覽資訊。然而，要實現這一目標存在多個挑戰：

1. 玻璃反光：許多展品放置在玻璃展示櫃中，反光干擾會影響AI對文物的準確識別。
2. 拍攝角度與位置：參觀者因位置或角度限制，無法拍攝完整展品影像，影響識別精度。
3. 外在干擾因素：光線不足或背景雜亂等因素，增加了拍攝困難度並影響AI辨識效果。

針對這些挑戰，我們開發了一種AI系統，即便面對上述各種干擾（尤其是玻璃反光），依然能精確識別博物館展品。

---

✨ 我們的解決方案

1. 反光消除模型

背景

展品展示櫃的反光是博物館中一個重大阻礙。現有的開源反光消除模型無法處理博物館展覽廳內的複雜照明和反射場景。為了解決這個問題：

• 我們開發了一個針對非固定環境的反光消除模型，能夠適應不同的照明條件和反射情境。
• 我們將反光消除任務作為風格轉換，採用了基於Pix2pix GAN的生成對抗網絡模型，並引入Sobel Feature指導模型注意反光區域，有效降低反光干擾，保護展品的關鍵細節。

2. 展品識別模型

為了在單張照片中識別出多個展品，我們採用了YOLOv8n模型，並對2000張展品圖像進行了手動標註，通過數據增強（翻轉、縮放、亮度調整等）來提高模型的穩健性。

• 我們訓練了兩個版本的模型：一個是原始YOLOv8n，另一個是利用反光處理後圖像的改進版。

3. 結合反光消除與YOLO檢測

我們開發了兩種展品識別方式：

1. 原始識別：直接在原始圖像上應用YOLOv8n檢測。
2. 反光消除後識別：經過反光消除模型處理後再進行YOLOv8n檢測。

通過比較兩種方式的結果，我們選擇了最優的識別結果，確保在不同環境下達到準確的文物識別效果。

---

🏆 成果與成就

數據結果

• V1模型（原始YOLO）：分類準確率達到**92.7%**。
• V2模型（反光處理後）：結合準確率提升至**94.5%**，有效減少誤判並提高識別質量。

質性分析

• 專注於反光區域：反光消除模型成功找出並去除反光干擾，保護展品細節。
• 檢測改進：反光消除與YOLO檢測的結合降低了誤判率，尤其是在反光嚴重的環境中。

---

📲 行動應用開發

我們將反光消除模型與YOLOv8n模型部署在雙平台的行動應用中，利用Android Studio和Xcode構建，讓用戶能輕鬆拍攝展品照片並獲取詳細資訊。

核心功能

1. 相機功能：用戶可輕鬆拍攝展品照片。
2. 自動化處理：照片同時經過反光去除和展品識別處理。
3. 互動資訊顯示：提供展品的詳細介紹，提升參觀體驗。

應用截圖

應用啟動的初始界面	相機準備拍攝

拍攝後的影像	圖像處理後的結果

📷 影片演示

---

🚀 實際應用價值

成就與優勢

• 反光消除：有效改善了博物館中玻璃展示櫃中的文物識別。
• YOLO多目標檢測：即便在單張影像中包含多個展品，也能準確分類。
• 雙平台應用：同時支援iOS和Android，增強了用戶可達性。

實際應用

1. 博物館體驗提升：參觀者可輕鬆使用應用拍攝並獲取展品資訊，提升互動性。
2. 文化和教育用途：此技術可擴展至其他文化場所，如藝術畫廊和教育機構。

---

🎯 未來展望

1. 模型優化：進一步優化反光消除和YOLO模型，以提高準確率和運行速度。
2. 多語言支援：擴展應用界面，支援多種語言，以適應來自世界各地的訪客。
3. 應用場景拓展：不僅限於博物館導覽，此方案還可應用於產品拍攝、學術研究和商業展覽等領域。

---

🤝 總結

這個項目成功克服了博物館展品的玻璃反光挑戰，實現了更精確的自動化識別。我們期待未來與更多博物館合作，並將該方案擴展至其他領域，持續推進AI在文化遺產中的應用。