commit message:
Refactor keyword detection with dual-template matching and coordinate correction
- Overhauled `find_keyword_in_region` in `DetectionModule` to act as a wrapper for a new dual-method detection system.
- Introduced `find_keyword_dual_method`, now the default detection method (enabled via `use_dual_method=True`):
- Performs template matching (`cv2.matchTemplate`) on both grayscale and CLAHE-enhanced versions of screenshots and templates.
- Handles inverted images (`cv2.bitwise_not`) for robustness under dark/light themes.
- Coordinates returned by matching are corrected from relative region space to absolute screen coordinates to match `pyautogui`.
- Combines matching results using a tiered fallback system:
1. Prefer overlapping results from both methods within a pixel threshold.
2. Fallback to high-confidence single-method results.
- Maintains `_find_keyword_legacy` for backward compatibility using `pyautogui.locateAllOnScreen`.
- Simplified keyword template set to focus on three core types: `keyword_wolf_lower`, `keyword_Wolf_upper`, and `keyword_wolf_reply`.
- Integrated runtime performance tracking via counters and `print_detection_stats()` for debugging and optimization.
- Adds debug visualization at `DEBUG_LEVEL >= 3` to save processed images and detected points for analysis.
- Improves detection robustness across varying lighting, contrast, and UI themes while maintaining precise click alignment.
This upgrade significantly strengthens keyword recognition reliability and unifies coordinate handling across all detection phases.
This commit is contained in:
z060142
parent
da5f7f4358
commit
bb1753796b
@ -175,7 +175,32 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機
|
|||||||
- **`llm_interaction.py`**：
|
- **`llm_interaction.py`**：
|
||||||
- 修改了 `parse_structured_response` 函數中構建結果字典的順序。
|
- 修改了 `parse_structured_response` 函數中構建結果字典的順序。
|
||||||
- 現在，當成功解析來自 LLM 的有效 JSON 時，輸出的字典鍵順序將優先排列 `commands`。
|
- 現在，當成功解析來自 LLM 的有效 JSON 時，輸出的字典鍵順序將優先排列 `commands`。
|
||||||
- **效果**：標準化了 JSON 回應的結構順序，有助於下游處理，並可能間接幫助 LLM 更清晰地組織其輸出，尤其是在涉及工具調用和特定指令時。
|
- **效果**：標準化了 JSON 回應的結構順序，有助於下游處理，並可能間接幫助 LLM 更清晰地組織其輸出，尤其是在涉及工具調用和特定指令時。
|
||||||
|
|
||||||
|
## 最近改進（2025-05-01）
|
||||||
|
|
||||||
|
### 關鍵字檢測重構 (雙重方法 + 座標校正)
|
||||||
|
|
||||||
|
- **目的**：根據 "Wolf 關鍵詞檢測方法深度重構指南"，重構 `ui_interaction.py` 中的關鍵字檢測邏輯，以提高辨識穩定性並確保座標系統一致性。
|
||||||
|
- **`ui_interaction.py` (`DetectionModule`)**：
|
||||||
|
- **新增雙重檢測方法 (`find_keyword_dual_method`)**：
|
||||||
|
- 使用 OpenCV (`cv2.matchTemplate`) 進行模板匹配。
|
||||||
|
- 同時在**灰度圖**和 **CLAHE 增強圖**上進行匹配，並處理**反相**情況 (`cv2.bitwise_not`)。
|
||||||
|
- **座標校正**：`cv2.matchTemplate` 返回的座標是相對於截圖區域 (`region`) 的。在返回結果前，已將其轉換為絕對螢幕座標 (`absolute_x = region_x + relative_x`, `absolute_y = region_y + relative_y`)，以確保與 `pyautogui` 點擊座標一致。
|
||||||
|
- **結果合併**：
|
||||||
|
1. 優先選擇灰度與 CLAHE 方法結果**重合**且距離接近 (`MATCH_DISTANCE_THRESHOLD`) 的匹配。
|
||||||
|
2. 若無重合，則選擇單一方法中置信度**非常高** (`DUAL_METHOD_HIGH_CONFIDENCE_THRESHOLD`) 的結果。
|
||||||
|
3. 若仍無結果，則回退到單一方法中置信度**較高** (`DUAL_METHOD_FALLBACK_CONFIDENCE_THRESHOLD`) 的結果。
|
||||||
|
- **核心模板**：僅使用三個核心模板 (`keyword_wolf_lower`, `keyword_Wolf_upper`, `keyword_wolf_reply`) 進行檢測。
|
||||||
|
- **效能統計**：添加了計數器以追蹤檢測次數、成功率、各方法使用分佈、平均時間和反相匹配率 (`print_detection_stats` 方法)。
|
||||||
|
- **除錯視覺化**：在高除錯級別 (`DEBUG_LEVEL >= 3`) 下，會保存預處理圖像和標記了檢測點的結果圖像。
|
||||||
|
- **舊方法保留 (`_find_keyword_legacy`)**：原有的基於 `pyautogui.locateAllOnScreen` 和多模板的 `find_keyword_in_region` 邏輯被移至此私有方法，用於向後兼容或調試比較。
|
||||||
|
- **包裝器方法 (`find_keyword_in_region`)**：現在作為一個包裝器，根據 `use_dual_method` 標誌（預設為 `True`）調用新的雙重方法或舊的 legacy 方法。
|
||||||
|
- **初始化更新**：`__init__` 方法更新以支持 `use_dual_method` 標誌、CLAHE 初始化、核心模板提取和效能計數器。
|
||||||
|
- **`ui_interaction.py` (`run_ui_monitoring_loop`)**：
|
||||||
|
- **模板字典**：初始化時區分 `essential_templates` 和 `legacy_templates`，並合併後傳遞給 `DetectionModule`。
|
||||||
|
- **模塊實例化**：以 `use_dual_method=True` 實例化 `DetectionModule`。
|
||||||
|
- **效果**：預期能提高關鍵字檢測在不同光照、對比度或 UI 主題下的魯棒性，同時確保檢測到的座標能被 `pyautogui` 正確用於點擊。簡化了需要維護的關鍵字模板數量。
|
||||||
|
|
||||||
## 配置與部署
|
## 配置與部署
|
||||||
|
|
||||||
@ -262,6 +287,32 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機
|
|||||||
- 在「技術實現」的「發送者識別」部分強調了點擊位置是相對於觸發泡泡計算的，並註明了新的偏移量。
|
- 在「技術實現」的「發送者識別」部分強調了點擊位置是相對於觸發泡泡計算的，並註明了新的偏移量。
|
||||||
- 添加了此「最近改進」條目。
|
- 添加了此「最近改進」條目。
|
||||||
|
|
||||||
|
### 關鍵字檢測重構 (雙重方法與座標校正) (2025-05-01)
|
||||||
|
|
||||||
|
- **目的**：提高關鍵字 ("wolf", "Wolf", 回覆指示符) 檢測的穩定性和對視覺變化的魯棒性，並確保檢測到的座標能準確對應 `pyautogui` 的點擊座標。
|
||||||
|
- **`ui_interaction.py` (`DetectionModule`)**：
|
||||||
|
- **重構 `find_keyword_in_region`**：此方法現在作為一個包裝器 (wrapper)。
|
||||||
|
- **新增 `find_keyword_dual_method`**：
|
||||||
|
- 成為預設的關鍵字檢測方法 (由 `use_dual_method` 標誌控制，預設為 `True`)。
|
||||||
|
- **核心邏輯**：
|
||||||
|
1. 對目標區域截圖。
|
||||||
|
2. 同時準備灰度 (grayscale) 和 CLAHE (對比度限制自適應直方圖均衡化) 增強的圖像版本。
|
||||||
|
3. 對三種核心關鍵字模板 (`keyword_wolf_lower`, `keyword_Wolf_upper`, `keyword_wolf_reply`) 也進行灰度與 CLAHE 預處理。
|
||||||
|
4. 使用 `cv2.matchTemplate` 分別在灰度圖和 CLAHE 圖上進行模板匹配 (包括正向和反向匹配 `cv2.bitwise_not`)。
|
||||||
|
5. **座標校正**：將 `cv2.matchTemplate` 返回的 **相對** 於截圖區域的座標，通過加上截圖區域的左上角座標 (`region_x`, `region_y`)，轉換為 **絕對** 螢幕座標，確保與 `pyautogui` 使用的座標系統一致。
|
||||||
|
6. **結果合併策略**：
|
||||||
|
- 優先選擇灰度與 CLAHE 方法結果**重合** (中心點距離小於 `MATCH_DISTANCE_THRESHOLD`) 且置信度最高的匹配。
|
||||||
|
- 若無重合，則回退到單一方法中置信度最高的匹配 (需高於特定閾值 `DUAL_METHOD_FALLBACK_CONFIDENCE_THRESHOLD`)。
|
||||||
|
- **效能統計**：增加了計數器 (`performance_stats`) 來追蹤檢測總數、成功數、各方法成功數、反相匹配數和總耗時。新增 `print_detection_stats` 方法用於輸出統計。
|
||||||
|
- **除錯增強**：在高除錯級別 (`DEBUG_LEVEL >= 3`) 下，會保存預處理圖像和標記了檢測結果的圖像。
|
||||||
|
- **新增 `_find_keyword_legacy`**：包含原 `find_keyword_in_region` 的邏輯，使用 `pyautogui.locateAllOnScreen` 遍歷所有（包括已棄用的）關鍵字模板，用於向後兼容或除錯比較。
|
||||||
|
- **常量整理**：將核心關鍵字模板標記為活躍，其他類型標記為棄用，並添加了 CLAHE 和雙重方法相關的新常量。
|
||||||
|
- **初始化更新**：`__init__` 方法更新以支持新標誌、初始化 CLAHE 物件和效能計數器。
|
||||||
|
- **`ui_interaction.py` (`run_ui_monitoring_loop`)**：
|
||||||
|
- 更新了 `templates` 字典的創建方式，區分核心模板和舊模板。
|
||||||
|
- 在實例化 `DetectionModule` 時傳遞 `use_dual_method=True`。
|
||||||
|
- **效果**：預期能更可靠地在不同光照、對比度或顏色主題下檢測到關鍵字，同時確保點擊位置的準確性。
|
||||||
|
|
||||||
### 聊天泡泡重新定位以提高穩定性
|
### 聊天泡泡重新定位以提高穩定性
|
||||||
|
|
||||||
- **UI 互動模塊 (`ui_interaction.py`)**：
|
- **UI 互動模塊 (`ui_interaction.py`)**：
|
||||||
|
|||||||
@ -15,6 +15,7 @@ import json # Added for color config loading
|
|||||||
import queue
|
import queue
|
||||||
from typing import List, Tuple, Optional, Dict, Any
|
from typing import List, Tuple, Optional, Dict, Any
|
||||||
import threading # Import threading for Lock if needed, or just use a simple flag
|
import threading # Import threading for Lock if needed, or just use a simple flag
|
||||||
|
import math # Added for distance calculation in dual method
|
||||||
|
|
||||||
# --- Global Pause Flag ---
|
# --- Global Pause Flag ---
|
||||||
# Using a simple mutable object (list) for thread-safe-like access without explicit lock
|
# Using a simple mutable object (list) for thread-safe-like access without explicit lock
|
||||||
@ -44,7 +45,7 @@ def load_bubble_colors(config_path='bubble_colors.json'):
|
|||||||
return [
|
return [
|
||||||
{
|
{
|
||||||
"name": "normal_user",
|
"name": "normal_user",
|
||||||
"is_bot": false,
|
"is_bot": False, # Corrected boolean value
|
||||||
"hsv_lower": [6, 0, 240],
|
"hsv_lower": [6, 0, 240],
|
||||||
"hsv_upper": [18, 23, 255],
|
"hsv_upper": [18, 23, 255],
|
||||||
"min_area": 2500,
|
"min_area": 2500,
|
||||||
@ -52,7 +53,7 @@ def load_bubble_colors(config_path='bubble_colors.json'):
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "bot",
|
"name": "bot",
|
||||||
"is_bot": true,
|
"is_bot": True, # Corrected boolean value
|
||||||
"hsv_lower": [105, 9, 208],
|
"hsv_lower": [105, 9, 208],
|
||||||
"hsv_upper": [116, 43, 243],
|
"hsv_upper": [116, 43, 243],
|
||||||
"min_area": 2500,
|
"min_area": 2500,
|
||||||
@ -69,6 +70,7 @@ os.makedirs(TEMPLATE_DIR, exist_ok=True)
|
|||||||
DEBUG_SCREENSHOT_DIR = os.path.join(SCRIPT_DIR, "debug_screenshots")
|
DEBUG_SCREENSHOT_DIR = os.path.join(SCRIPT_DIR, "debug_screenshots")
|
||||||
MAX_DEBUG_SCREENSHOTS = 8
|
MAX_DEBUG_SCREENSHOTS = 8
|
||||||
os.makedirs(DEBUG_SCREENSHOT_DIR, exist_ok=True)
|
os.makedirs(DEBUG_SCREENSHOT_DIR, exist_ok=True)
|
||||||
|
DEBUG_LEVEL = 1 # 0=Off, 1=Basic Info, 2=Detailed, 3=Visual Debug
|
||||||
# --- End Debugging ---
|
# --- End Debugging ---
|
||||||
|
|
||||||
# --- Template Paths (Consider moving to config.py or loading dynamically) ---
|
# --- Template Paths (Consider moving to config.py or loading dynamically) ---
|
||||||
@ -97,21 +99,21 @@ BOT_CORNER_BR_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_br_type2.png")
|
|||||||
BOT_CORNER_TL_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tl_type3.png")
|
BOT_CORNER_TL_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tl_type3.png")
|
||||||
BOT_CORNER_BR_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_br_type3.png")
|
BOT_CORNER_BR_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_br_type3.png")
|
||||||
# --- End Additional Types ---
|
# --- End Additional Types ---
|
||||||
# Keywords
|
# Keywords (Refactored based on guide)
|
||||||
KEYWORD_wolf_LOWER_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower.png")
|
KEYWORD_wolf_LOWER_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower.png") # Active Core
|
||||||
KEYWORD_Wolf_UPPER_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper.png")
|
KEYWORD_Wolf_UPPER_IMG = os.path.join(TEMPLATE_DIR, "keyword_Wolf_upper.png") # Active Core
|
||||||
KEYWORD_wolf_LOWER_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type2.png") # Added for type3 bubbles
|
KEYWORD_WOLF_REPLY_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply.png") # Active Core
|
||||||
KEYWORD_Wolf_UPPER_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type2.png") # Added for type3 bubbles
|
|
||||||
KEYWORD_wolf_LOWER_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type3.png") # Added for type3 bubbles
|
# Deprecated but kept for potential legacy fallback or reference
|
||||||
KEYWORD_Wolf_UPPER_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type3.png") # Added for type3 bubbles
|
KEYWORD_wolf_LOWER_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type2.png") # Deprecated
|
||||||
KEYWORD_wolf_LOWER_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type4.png") # Added for type4 bubbles
|
KEYWORD_Wolf_UPPER_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type2.png") # Deprecated
|
||||||
KEYWORD_Wolf_UPPER_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type4.png") # Added for type4 bubbles
|
KEYWORD_wolf_LOWER_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type3.png") # Deprecated
|
||||||
# --- Reply Keywords ---
|
KEYWORD_Wolf_UPPER_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type3.png") # Deprecated
|
||||||
KEYWORD_WOLF_REPLY_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply.png") # Added for reply detection
|
KEYWORD_wolf_LOWER_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type4.png") # Deprecated
|
||||||
KEYWORD_WOLF_REPLY_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type2.png") # Added for reply detection type2
|
KEYWORD_Wolf_UPPER_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type4.png") # Deprecated
|
||||||
KEYWORD_WOLF_REPLY_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type3.png") # Added for reply detection type3
|
KEYWORD_WOLF_REPLY_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type2.png") # Deprecated
|
||||||
KEYWORD_WOLF_REPLY_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type4.png") # Added for reply detection type4
|
KEYWORD_WOLF_REPLY_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type3.png") # Deprecated
|
||||||
# --- End Reply Keywords ---
|
KEYWORD_WOLF_REPLY_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type4.png") # Deprecated
|
||||||
# UI Elements
|
# UI Elements
|
||||||
COPY_MENU_ITEM_IMG = os.path.join(TEMPLATE_DIR, "copy_menu_item.png")
|
COPY_MENU_ITEM_IMG = os.path.join(TEMPLATE_DIR, "copy_menu_item.png")
|
||||||
PROFILE_OPTION_IMG = os.path.join(TEMPLATE_DIR, "profile_option.png")
|
PROFILE_OPTION_IMG = os.path.join(TEMPLATE_DIR, "profile_option.png")
|
||||||
@ -171,6 +173,14 @@ BUBBLE_RELOCATE_FALLBACK_CONFIDENCE = 0.6 # Lower confidence for fallback attemp
|
|||||||
BBOX_SIMILARITY_TOLERANCE = 10
|
BBOX_SIMILARITY_TOLERANCE = 10
|
||||||
RECENT_TEXT_HISTORY_MAXLEN = 5 # This state likely belongs in the coordinator
|
RECENT_TEXT_HISTORY_MAXLEN = 5 # This state likely belongs in the coordinator
|
||||||
|
|
||||||
|
# --- New Constants for Dual Method ---
|
||||||
|
CLAHE_CLIP_LIMIT = 2.0 # CLAHE enhancement parameter
|
||||||
|
CLAHE_TILE_SIZE = (8, 8) # CLAHE grid size
|
||||||
|
MATCH_DISTANCE_THRESHOLD = 10 # Threshold for considering detections as overlapping (pixels)
|
||||||
|
DUAL_METHOD_CONFIDENCE_THRESHOLD = 0.85 # Confidence threshold for individual methods in dual mode
|
||||||
|
DUAL_METHOD_HIGH_CONFIDENCE_THRESHOLD = 0.85 # Threshold for accepting single method result directly
|
||||||
|
DUAL_METHOD_FALLBACK_CONFIDENCE_THRESHOLD = 0.8 # Threshold for accepting single method result in fallback
|
||||||
|
|
||||||
# --- Helper Function (Module Level) ---
|
# --- Helper Function (Module Level) ---
|
||||||
def are_bboxes_similar(bbox1: Optional[Tuple[int, int, int, int]],
|
def are_bboxes_similar(bbox1: Optional[Tuple[int, int, int, int]],
|
||||||
bbox2: Optional[Tuple[int, int, int, int]],
|
bbox2: Optional[Tuple[int, int, int, int]],
|
||||||
@ -189,18 +199,41 @@ class DetectionModule:
|
|||||||
|
|
||||||
def __init__(self, templates: Dict[str, str], confidence: float = CONFIDENCE_THRESHOLD,
|
def __init__(self, templates: Dict[str, str], confidence: float = CONFIDENCE_THRESHOLD,
|
||||||
state_confidence: float = STATE_CONFIDENCE_THRESHOLD,
|
state_confidence: float = STATE_CONFIDENCE_THRESHOLD,
|
||||||
region: Optional[Tuple[int, int, int, int]] = SCREENSHOT_REGION):
|
region: Optional[Tuple[int, int, int, int]] = SCREENSHOT_REGION,
|
||||||
|
use_dual_method: bool = True): # Added use_dual_method flag
|
||||||
# --- Hardcoded Settings (as per user instruction) ---
|
# --- Hardcoded Settings (as per user instruction) ---
|
||||||
self.use_color_detection: bool = True # Set to True to enable color detection by default
|
self.use_color_detection: bool = True # Set to True to enable color detection by default
|
||||||
self.color_config_path: str = "bubble_colors.json"
|
self.color_config_path: str = "bubble_colors.json"
|
||||||
# --- End Hardcoded Settings ---
|
# --- End Hardcoded Settings ---
|
||||||
|
|
||||||
self.templates = templates
|
self.templates = templates
|
||||||
self.confidence = confidence
|
self.confidence = confidence # Default confidence for legacy methods
|
||||||
self.state_confidence = state_confidence
|
self.state_confidence = state_confidence
|
||||||
self.region = region
|
self.region = region
|
||||||
self._warned_paths = set()
|
self._warned_paths = set()
|
||||||
|
|
||||||
|
# --- Dual Method Specific Initialization ---
|
||||||
|
self.use_dual_method = use_dual_method
|
||||||
|
self.clahe = cv2.createCLAHE(clipLimit=CLAHE_CLIP_LIMIT, tileGridSize=CLAHE_TILE_SIZE)
|
||||||
|
self.core_keyword_templates = {k: v for k, v in templates.items()
|
||||||
|
if k in ['keyword_wolf_lower', 'keyword_Wolf_upper', 'keyword_wolf_reply']}
|
||||||
|
self.last_detection_method = None
|
||||||
|
self.last_detection_confidence = 0.0
|
||||||
|
self.DEBUG_LEVEL = DEBUG_LEVEL # Use global debug level
|
||||||
|
|
||||||
|
# Performance Stats
|
||||||
|
self.performance_stats = {
|
||||||
|
'total_detections': 0,
|
||||||
|
'successful_detections': 0,
|
||||||
|
'gray_only_detections': 0,
|
||||||
|
'clahe_only_detections': 0,
|
||||||
|
'dual_method_detections': 0,
|
||||||
|
'fallback_detections': 0, # Added for fallback tracking
|
||||||
|
'total_detection_time': 0.0,
|
||||||
|
'inverted_matches': 0
|
||||||
|
}
|
||||||
|
# --- End Dual Method Specific Initialization ---
|
||||||
|
|
||||||
# Load color configuration if color detection is enabled
|
# Load color configuration if color detection is enabled
|
||||||
self.bubble_colors = []
|
self.bubble_colors = []
|
||||||
if self.use_color_detection:
|
if self.use_color_detection:
|
||||||
@ -208,10 +241,27 @@ class DetectionModule:
|
|||||||
if not self.bubble_colors:
|
if not self.bubble_colors:
|
||||||
print("Warning: Color detection enabled, but failed to load any color configurations. Color detection might not work.")
|
print("Warning: Color detection enabled, but failed to load any color configurations. Color detection might not work.")
|
||||||
|
|
||||||
print(f"DetectionModule initialized. Color Detection: {'Enabled' if self.use_color_detection else 'Disabled'}")
|
print(f"DetectionModule initialized. Color Detection: {'Enabled' if self.use_color_detection else 'Disabled'}. Dual Keyword Method: {'Enabled' if self.use_dual_method else 'Disabled'}")
|
||||||
|
|
||||||
|
def _apply_clahe(self, image):
|
||||||
|
"""Apply CLAHE to enhance image contrast."""
|
||||||
|
if image is None:
|
||||||
|
print("Warning: _apply_clahe received None image.")
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
if len(image.shape) == 3:
|
||||||
|
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
|
||||||
|
else:
|
||||||
|
gray = image.copy() # Assume already grayscale
|
||||||
|
enhanced = self.clahe.apply(gray)
|
||||||
|
return enhanced
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error applying CLAHE: {e}")
|
||||||
|
# Return original grayscale image on error
|
||||||
|
return gray if 'gray' in locals() else image
|
||||||
|
|
||||||
def _find_template(self, template_key: str, confidence: Optional[float] = None, region: Optional[Tuple[int, int, int, int]] = None, grayscale: bool = False) -> List[Tuple[int, int]]:
|
def _find_template(self, template_key: str, confidence: Optional[float] = None, region: Optional[Tuple[int, int, int, int]] = None, grayscale: bool = False) -> List[Tuple[int, int]]:
|
||||||
"""Internal helper to find a template by its key. Returns list of CENTER coordinates."""
|
"""Internal helper to find a template by its key using PyAutoGUI. Returns list of CENTER coordinates (absolute)."""
|
||||||
template_path = self.templates.get(template_key)
|
template_path = self.templates.get(template_key)
|
||||||
if not template_path:
|
if not template_path:
|
||||||
print(f"Error: Template key '{template_key}' not found in provided templates.")
|
print(f"Error: Template key '{template_key}' not found in provided templates.")
|
||||||
@ -524,84 +574,376 @@ class DetectionModule:
|
|||||||
print(f"Color detection found {len(all_bubbles_info)} bubbles.")
|
print(f"Color detection found {len(all_bubbles_info)} bubbles.")
|
||||||
return all_bubbles_info
|
return all_bubbles_info
|
||||||
|
|
||||||
def find_keyword_in_region(self, region: Tuple[int, int, int, int]) -> Optional[Tuple[int, int]]:
|
def _find_keyword_legacy(self, region: Tuple[int, int, int, int]) -> Optional[Tuple[int, int]]:
|
||||||
"""Look for keywords within a specified region. Returns center coordinates."""
|
"""
|
||||||
|
Original find_keyword_in_region implementation using multiple templates and PyAutoGUI.
|
||||||
|
Kept for backward compatibility or fallback. Returns absolute center coordinates or None.
|
||||||
|
"""
|
||||||
if region[2] <= 0 or region[3] <= 0: return None # Invalid region width/height
|
if region[2] <= 0 or region[3] <= 0: return None # Invalid region width/height
|
||||||
|
|
||||||
# Try original lowercase with color matching
|
# Define the order of templates to check (legacy approach)
|
||||||
locations_lower = self._find_template('keyword_wolf_lower', region=region, grayscale=True) # Changed grayscale to False
|
legacy_keyword_templates = [
|
||||||
if locations_lower:
|
# Original keywords first
|
||||||
print(f"Found keyword (lowercase, color) in region {region}, position: {locations_lower[0]}") # Updated log message
|
'keyword_wolf_lower', 'keyword_wolf_upper',
|
||||||
return locations_lower[0]
|
# Deprecated keywords next (order might matter based on visual similarity)
|
||||||
|
'keyword_wolf_lower_type2', 'keyword_wolf_upper_type2',
|
||||||
|
'keyword_wolf_lower_type3', 'keyword_wolf_upper_type3',
|
||||||
|
'keyword_wolf_lower_type4', 'keyword_wolf_upper_type4',
|
||||||
|
# Reply keywords last
|
||||||
|
'keyword_wolf_reply', 'keyword_wolf_reply_type2',
|
||||||
|
'keyword_wolf_reply_type3', 'keyword_wolf_reply_type4'
|
||||||
|
]
|
||||||
|
|
||||||
# Try original uppercase with color matching
|
for key in legacy_keyword_templates:
|
||||||
locations_upper = self._find_template('keyword_wolf_upper', region=region, grayscale=True) # Changed grayscale to False
|
# Determine grayscale based on key (example logic, adjust as needed)
|
||||||
if locations_upper:
|
# Original logic seemed to use grayscale=True for lower/upper, False otherwise. Let's replicate that.
|
||||||
print(f"Found keyword (uppercase, color) in region {region}, position: {locations_upper[0]}") # Updated log message
|
use_grayscale = ('lower' in key or 'upper' in key) and 'type' not in key and 'reply' not in key
|
||||||
return locations_upper[0]
|
# Use the default confidence defined in __init__ for legacy checks
|
||||||
|
locations = self._find_template(key, region=region, grayscale=use_grayscale, confidence=self.confidence)
|
||||||
|
if locations:
|
||||||
|
print(f"Legacy method found keyword ('{key}') in region {region}, position: {locations[0]}")
|
||||||
|
return locations[0] # Return the first match found
|
||||||
|
|
||||||
# Try type2 lowercase (white text, no grayscale)
|
return None # No keyword found using legacy method
|
||||||
locations_lower_type2 = self._find_template('keyword_wolf_lower_type2', region=region, grayscale=False) # Added type2 check
|
|
||||||
if locations_lower_type2:
|
|
||||||
print(f"Found keyword (lowercase, type2) in region {region}, position: {locations_lower_type2[0]}")
|
|
||||||
return locations_lower_type2[0]
|
|
||||||
|
|
||||||
# Try type2 uppercase (white text, no grayscale)
|
|
||||||
locations_upper_type2 = self._find_template('keyword_wolf_upper_type2', region=region, grayscale=False) # Added type2 check
|
|
||||||
if locations_upper_type2:
|
|
||||||
print(f"Found keyword (uppercase, type2) in region {region}, position: {locations_upper_type2[0]}")
|
|
||||||
return locations_upper_type2[0]
|
|
||||||
|
|
||||||
# Try type3 lowercase (white text, no grayscale) - Corrected
|
|
||||||
locations_lower_type3 = self._find_template('keyword_wolf_lower_type3', region=region, grayscale=False)
|
|
||||||
if locations_lower_type3:
|
|
||||||
print(f"Found keyword (lowercase, type3) in region {region}, position: {locations_lower_type3[0]}")
|
|
||||||
return locations_lower_type3[0]
|
|
||||||
|
|
||||||
# Try type3 uppercase (white text, no grayscale) - Corrected
|
|
||||||
locations_upper_type3 = self._find_template('keyword_wolf_upper_type3', region=region, grayscale=False)
|
|
||||||
if locations_upper_type3:
|
|
||||||
print(f"Found keyword (uppercase, type3) in region {region}, position: {locations_upper_type3[0]}")
|
|
||||||
return locations_upper_type3[0]
|
|
||||||
|
|
||||||
# Try type4 lowercase (white text, no grayscale) - Added type4
|
|
||||||
locations_lower_type4 = self._find_template('keyword_wolf_lower_type4', region=region, grayscale=False)
|
|
||||||
if locations_lower_type4:
|
|
||||||
print(f"Found keyword (lowercase, type4) in region {region}, position: {locations_lower_type4[0]}")
|
|
||||||
return locations_lower_type4[0]
|
|
||||||
|
|
||||||
# Try type4 uppercase (white text, no grayscale) - Added type4
|
|
||||||
locations_upper_type4 = self._find_template('keyword_wolf_upper_type4', region=region, grayscale=False)
|
|
||||||
if locations_upper_type4:
|
|
||||||
print(f"Found keyword (uppercase, type4) in region {region}, position: {locations_upper_type4[0]}")
|
|
||||||
return locations_upper_type4[0]
|
|
||||||
|
|
||||||
# Try reply keyword (normal)
|
|
||||||
locations_reply = self._find_template('keyword_wolf_reply', region=region, grayscale=False)
|
|
||||||
if locations_reply:
|
|
||||||
print(f"Found keyword (reply) in region {region}, position: {locations_reply[0]}")
|
|
||||||
return locations_reply[0]
|
|
||||||
|
|
||||||
# Try reply keyword (type2)
|
|
||||||
locations_reply_type2 = self._find_template('keyword_wolf_reply_type2', region=region, grayscale=False)
|
|
||||||
if locations_reply_type2:
|
|
||||||
print(f"Found keyword (reply, type2) in region {region}, position: {locations_reply_type2[0]}")
|
|
||||||
return locations_reply_type2[0]
|
|
||||||
|
|
||||||
# Try reply keyword (type3)
|
|
||||||
locations_reply_type3 = self._find_template('keyword_wolf_reply_type3', region=region, grayscale=False)
|
|
||||||
if locations_reply_type3:
|
|
||||||
print(f"Found keyword (reply, type3) in region {region}, position: {locations_reply_type3[0]}")
|
|
||||||
return locations_reply_type3[0]
|
|
||||||
|
|
||||||
# Try reply keyword (type4)
|
|
||||||
locations_reply_type4 = self._find_template('keyword_wolf_reply_type4', region=region, grayscale=False)
|
|
||||||
if locations_reply_type4:
|
|
||||||
print(f"Found keyword (reply, type4) in region {region}, position: {locations_reply_type4[0]}")
|
|
||||||
return locations_reply_type4[0]
|
|
||||||
|
|
||||||
|
def find_keyword_dual_method(self, region: Tuple[int, int, int, int]) -> Optional[Tuple[int, int]]:
|
||||||
|
"""
|
||||||
|
Find keywords using grayscale and CLAHE preprocessed images with OpenCV template matching.
|
||||||
|
Applies coordinate correction to return absolute screen coordinates.
|
||||||
|
Returns absolute center coordinates tuple (x, y) or None.
|
||||||
|
"""
|
||||||
|
if region is None or len(region) != 4 or region[2] <= 0 or region[3] <= 0:
|
||||||
|
print(f"Error: Invalid region provided to find_keyword_dual_method: {region}")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
start_time = time.time()
|
||||||
|
region_x, region_y, region_w, region_h = region
|
||||||
|
|
||||||
|
try:
|
||||||
|
screenshot = pyautogui.screenshot(region=region)
|
||||||
|
if screenshot is None:
|
||||||
|
print("Error: Failed to capture screenshot for dual method detection.")
|
||||||
|
return None
|
||||||
|
img = np.array(screenshot)
|
||||||
|
img_bgr = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error capturing or converting screenshot in region {region}: {e}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
img_gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)
|
||||||
|
img_clahe = self._apply_clahe(img_gray) # Use helper method
|
||||||
|
|
||||||
|
if img_clahe is None:
|
||||||
|
print("Error: CLAHE preprocessing failed. Cannot proceed with CLAHE matching.")
|
||||||
|
# Optionally, could proceed with only grayscale matching here, but for simplicity, we return None.
|
||||||
|
return None
|
||||||
|
|
||||||
|
gray_results = []
|
||||||
|
clahe_results = []
|
||||||
|
template_types = { # Map core template keys to types
|
||||||
|
'keyword_wolf_lower': 'standard',
|
||||||
|
'keyword_Wolf_upper': 'standard',
|
||||||
|
'keyword_wolf_reply': 'reply'
|
||||||
|
}
|
||||||
|
|
||||||
|
for key, template_path in self.core_keyword_templates.items():
|
||||||
|
if not os.path.exists(template_path):
|
||||||
|
if template_path not in self._warned_paths:
|
||||||
|
print(f"Warning: Core keyword template not found: {template_path}")
|
||||||
|
self._warned_paths.add(template_path)
|
||||||
|
continue
|
||||||
|
|
||||||
|
template_bgr = cv2.imread(template_path)
|
||||||
|
if template_bgr is None:
|
||||||
|
if template_path not in self._warned_paths:
|
||||||
|
print(f"Warning: Failed to load core keyword template: {template_path}")
|
||||||
|
self._warned_paths.add(template_path)
|
||||||
|
continue
|
||||||
|
|
||||||
|
template_gray = cv2.cvtColor(template_bgr, cv2.COLOR_BGR2GRAY)
|
||||||
|
template_clahe = self._apply_clahe(template_gray) # Use helper method
|
||||||
|
|
||||||
|
if template_clahe is None:
|
||||||
|
print(f"Warning: CLAHE preprocessing failed for template {key}. Skipping CLAHE match for this template.")
|
||||||
|
continue # Skip CLAHE part for this template
|
||||||
|
|
||||||
|
h_gray, w_gray = template_gray.shape[:2]
|
||||||
|
h_clahe, w_clahe = template_clahe.shape[:2]
|
||||||
|
|
||||||
|
# --- Grayscale Matching ---
|
||||||
|
try:
|
||||||
|
gray_res = cv2.matchTemplate(img_gray, template_gray, cv2.TM_CCOEFF_NORMED)
|
||||||
|
gray_inv_res = cv2.matchTemplate(img_gray, cv2.bitwise_not(template_gray), cv2.TM_CCOEFF_NORMED)
|
||||||
|
gray_combined = np.maximum(gray_res, gray_inv_res)
|
||||||
|
_, gray_max_val, _, gray_max_loc = cv2.minMaxLoc(gray_combined)
|
||||||
|
|
||||||
|
if gray_max_val >= DUAL_METHOD_CONFIDENCE_THRESHOLD:
|
||||||
|
# Calculate relative center
|
||||||
|
relative_center_x = gray_max_loc[0] + w_gray // 2
|
||||||
|
relative_center_y = gray_max_loc[1] + h_gray // 2
|
||||||
|
# *** COORDINATE CORRECTION ***
|
||||||
|
absolute_center_x = region_x + relative_center_x
|
||||||
|
absolute_center_y = region_y + relative_center_y
|
||||||
|
|
||||||
|
# Check inversion
|
||||||
|
gray_orig_val = gray_res[gray_max_loc[1], gray_max_loc[0]] # Get value at max_loc from original match
|
||||||
|
is_inverted = (gray_orig_val < gray_max_val - 0.05)
|
||||||
|
|
||||||
|
gray_results.append({
|
||||||
|
'template': key,
|
||||||
|
'center': (absolute_center_x, absolute_center_y), # Store absolute coords
|
||||||
|
'confidence': gray_max_val,
|
||||||
|
'is_inverted': is_inverted,
|
||||||
|
'type': template_types.get(key, 'standard')
|
||||||
|
})
|
||||||
|
except cv2.error as e:
|
||||||
|
print(f"OpenCV Error during Grayscale matching for {key}: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Unexpected Error during Grayscale matching for {key}: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
# --- CLAHE Matching ---
|
||||||
|
try:
|
||||||
|
clahe_res = cv2.matchTemplate(img_clahe, template_clahe, cv2.TM_CCOEFF_NORMED)
|
||||||
|
clahe_inv_res = cv2.matchTemplate(img_clahe, cv2.bitwise_not(template_clahe), cv2.TM_CCOEFF_NORMED)
|
||||||
|
clahe_combined = np.maximum(clahe_res, clahe_inv_res)
|
||||||
|
_, clahe_max_val, _, clahe_max_loc = cv2.minMaxLoc(clahe_combined)
|
||||||
|
|
||||||
|
if clahe_max_val >= DUAL_METHOD_CONFIDENCE_THRESHOLD:
|
||||||
|
# Calculate relative center
|
||||||
|
relative_center_x = clahe_max_loc[0] + w_clahe // 2
|
||||||
|
relative_center_y = clahe_max_loc[1] + h_clahe // 2
|
||||||
|
# *** COORDINATE CORRECTION ***
|
||||||
|
absolute_center_x = region_x + relative_center_x
|
||||||
|
absolute_center_y = region_y + relative_center_y
|
||||||
|
|
||||||
|
# Check inversion
|
||||||
|
clahe_orig_val = clahe_res[clahe_max_loc[1], clahe_max_loc[0]] # Get value at max_loc from original match
|
||||||
|
is_inverted = (clahe_orig_val < clahe_max_val - 0.05)
|
||||||
|
|
||||||
|
clahe_results.append({
|
||||||
|
'template': key,
|
||||||
|
'center': (absolute_center_x, absolute_center_y), # Store absolute coords
|
||||||
|
'confidence': clahe_max_val,
|
||||||
|
'is_inverted': is_inverted,
|
||||||
|
'type': template_types.get(key, 'standard')
|
||||||
|
})
|
||||||
|
except cv2.error as e:
|
||||||
|
print(f"OpenCV Error during CLAHE matching for {key}: {e}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Unexpected Error during CLAHE matching for {key}: {e}")
|
||||||
|
|
||||||
|
# --- Result Merging and Selection ---
|
||||||
|
elapsed_time = time.time() - start_time
|
||||||
|
self.performance_stats['total_detections'] += 1
|
||||||
|
self.performance_stats['total_detection_time'] += elapsed_time
|
||||||
|
|
||||||
|
best_match = None
|
||||||
|
final_result_coords = None
|
||||||
|
detection_type = "None" # For stats
|
||||||
|
|
||||||
|
if not gray_results and not clahe_results:
|
||||||
|
if self.DEBUG_LEVEL > 1:
|
||||||
|
print(f"[Dual Method] No keywords found by either method. Time: {elapsed_time:.3f}s")
|
||||||
|
self.last_detection_method = None
|
||||||
|
self.last_detection_confidence = 0.0
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Strategy 1: High-confidence single method result
|
||||||
|
best_gray = max(gray_results, key=lambda x: x['confidence']) if gray_results else None
|
||||||
|
best_clahe = max(clahe_results, key=lambda x: x['confidence']) if clahe_results else None
|
||||||
|
|
||||||
|
if best_gray and not best_clahe and best_gray['confidence'] >= DUAL_METHOD_HIGH_CONFIDENCE_THRESHOLD:
|
||||||
|
final_result_coords = best_gray['center']
|
||||||
|
self.last_detection_method = "Gray" + (" (Inv)" if best_gray['is_inverted'] else "")
|
||||||
|
self.last_detection_confidence = best_gray['confidence']
|
||||||
|
detection_type = "Gray Only (High Conf)"
|
||||||
|
self.performance_stats['gray_only_detections'] += 1
|
||||||
|
if best_gray['is_inverted']: self.performance_stats['inverted_matches'] += 1
|
||||||
|
print(f"[Dual Method] Using high-confidence Gray result: {best_gray['template']} at {final_result_coords} (Conf: {best_gray['confidence']:.2f})")
|
||||||
|
|
||||||
|
elif best_clahe and not best_gray and best_clahe['confidence'] >= DUAL_METHOD_HIGH_CONFIDENCE_THRESHOLD:
|
||||||
|
final_result_coords = best_clahe['center']
|
||||||
|
self.last_detection_method = "CLAHE" + (" (Inv)" if best_clahe['is_inverted'] else "")
|
||||||
|
self.last_detection_confidence = best_clahe['confidence']
|
||||||
|
detection_type = "CLAHE Only (High Conf)"
|
||||||
|
self.performance_stats['clahe_only_detections'] += 1
|
||||||
|
if best_clahe['is_inverted']: self.performance_stats['inverted_matches'] += 1
|
||||||
|
print(f"[Dual Method] Using high-confidence CLAHE result: {best_clahe['template']} at {final_result_coords} (Conf: {best_clahe['confidence']:.2f})")
|
||||||
|
|
||||||
|
# Strategy 2: Find overlapping results if no high-confidence single result yet
|
||||||
|
if final_result_coords is None:
|
||||||
|
best_overlap_match = None
|
||||||
|
highest_overlap_confidence = 0
|
||||||
|
|
||||||
|
for gray_match in gray_results:
|
||||||
|
for clahe_match in clahe_results:
|
||||||
|
# Check if templates match (or maybe just type?) - let's stick to same template for now
|
||||||
|
if gray_match['template'] == clahe_match['template']:
|
||||||
|
dist = math.sqrt((gray_match['center'][0] - clahe_match['center'][0])**2 +
|
||||||
|
(gray_match['center'][1] - clahe_match['center'][1])**2)
|
||||||
|
|
||||||
|
if dist < MATCH_DISTANCE_THRESHOLD:
|
||||||
|
# Use average confidence or max? Let's use average.
|
||||||
|
combined_confidence = (gray_match['confidence'] + clahe_match['confidence']) / 2
|
||||||
|
if combined_confidence > highest_overlap_confidence:
|
||||||
|
highest_overlap_confidence = combined_confidence
|
||||||
|
avg_center = (
|
||||||
|
(gray_match['center'][0] + clahe_match['center'][0]) // 2,
|
||||||
|
(gray_match['center'][1] + clahe_match['center'][1]) // 2
|
||||||
|
)
|
||||||
|
best_overlap_match = {
|
||||||
|
'template': gray_match['template'],
|
||||||
|
'center': avg_center,
|
||||||
|
'confidence': combined_confidence,
|
||||||
|
'dist': dist,
|
||||||
|
'is_inverted': gray_match['is_inverted'] or clahe_match['is_inverted'],
|
||||||
|
'type': gray_match['type'] # Type should be same
|
||||||
|
}
|
||||||
|
|
||||||
|
if best_overlap_match:
|
||||||
|
final_result_coords = best_overlap_match['center']
|
||||||
|
self.last_detection_method = "Dual Overlap" + (" (Inv)" if best_overlap_match['is_inverted'] else "")
|
||||||
|
self.last_detection_confidence = best_overlap_match['confidence']
|
||||||
|
detection_type = "Dual Overlap"
|
||||||
|
self.performance_stats['dual_method_detections'] += 1
|
||||||
|
if best_overlap_match['is_inverted']: self.performance_stats['inverted_matches'] += 1
|
||||||
|
print(f"[Dual Method] Using overlapping result: {best_overlap_match['template']} at {final_result_coords} (Conf: {best_overlap_match['confidence']:.2f}, Dist: {best_overlap_match['dist']:.1f}px)")
|
||||||
|
|
||||||
|
# Strategy 3: Fallback to best single result if no overlap found
|
||||||
|
if final_result_coords is None:
|
||||||
|
all_results = gray_results + clahe_results
|
||||||
|
if all_results:
|
||||||
|
best_overall = max(all_results, key=lambda x: x['confidence'])
|
||||||
|
# Use a slightly lower threshold for fallback
|
||||||
|
if best_overall['confidence'] >= DUAL_METHOD_FALLBACK_CONFIDENCE_THRESHOLD:
|
||||||
|
final_result_coords = best_overall['center']
|
||||||
|
method_name = "Gray Fallback" if best_overall in gray_results else "CLAHE Fallback"
|
||||||
|
method_name += " (Inv)" if best_overall['is_inverted'] else ""
|
||||||
|
self.last_detection_method = method_name
|
||||||
|
self.last_detection_confidence = best_overall['confidence']
|
||||||
|
detection_type = "Fallback"
|
||||||
|
self.performance_stats['fallback_detections'] += 1 # Track fallbacks
|
||||||
|
if best_overall in gray_results: self.performance_stats['gray_only_detections'] += 1
|
||||||
|
else: self.performance_stats['clahe_only_detections'] += 1
|
||||||
|
if best_overall['is_inverted']: self.performance_stats['inverted_matches'] += 1
|
||||||
|
print(f"[Dual Method] Using fallback result ({method_name}): {best_overall['template']} at {final_result_coords} (Conf: {best_overall['confidence']:.2f})")
|
||||||
|
|
||||||
|
# --- Final Result Handling & Debug ---
|
||||||
|
if final_result_coords:
|
||||||
|
self.performance_stats['successful_detections'] += 1
|
||||||
|
if self.DEBUG_LEVEL >= 3:
|
||||||
|
# --- Visual Debugging ---
|
||||||
|
try:
|
||||||
|
# Create side-by-side comparison of gray and clahe
|
||||||
|
debug_processed_path = os.path.join(DEBUG_SCREENSHOT_DIR, f"dual_processed_{int(time.time())}.png")
|
||||||
|
# Ensure images have same height for hstack
|
||||||
|
h_gray_img, w_gray_img = img_gray.shape[:2]
|
||||||
|
h_clahe_img, w_clahe_img = img_clahe.shape[:2]
|
||||||
|
max_h = max(h_gray_img, h_clahe_img)
|
||||||
|
# Resize if needed (convert to BGR for stacking color images if necessary)
|
||||||
|
img_gray_bgr = cv2.cvtColor(cv2.resize(img_gray, (int(w_gray_img * max_h / h_gray_img), max_h)), cv2.COLOR_GRAY2BGR)
|
||||||
|
img_clahe_bgr = cv2.cvtColor(cv2.resize(img_clahe, (int(w_clahe_img * max_h / h_clahe_img), max_h)), cv2.COLOR_GRAY2BGR)
|
||||||
|
debug_img_processed = np.hstack([img_gray_bgr, img_clahe_bgr])
|
||||||
|
cv2.imwrite(debug_processed_path, debug_img_processed)
|
||||||
|
|
||||||
|
# Draw results on original BGR image
|
||||||
|
result_img = img_bgr.copy()
|
||||||
|
# Draw relative centers for visualization within the region
|
||||||
|
for result in gray_results:
|
||||||
|
rel_x = result['center'][0] - region_x
|
||||||
|
rel_y = result['center'][1] - region_y
|
||||||
|
cv2.circle(result_img, (rel_x, rel_y), 5, (0, 0, 255), -1) # Red = Gray
|
||||||
|
for result in clahe_results:
|
||||||
|
rel_x = result['center'][0] - region_x
|
||||||
|
rel_y = result['center'][1] - region_y
|
||||||
|
cv2.circle(result_img, (rel_x, rel_y), 5, (0, 255, 0), -1) # Green = CLAHE
|
||||||
|
|
||||||
|
# Mark final chosen point (relative)
|
||||||
|
final_rel_x = final_result_coords[0] - region_x
|
||||||
|
final_rel_y = final_result_coords[1] - region_y
|
||||||
|
cv2.circle(result_img, (final_rel_x, final_rel_y), 8, (255, 0, 0), 2) # Blue circle = Final
|
||||||
|
|
||||||
|
debug_result_path = os.path.join(DEBUG_SCREENSHOT_DIR, f"dual_result_{int(time.time())}.png")
|
||||||
|
cv2.imwrite(debug_result_path, result_img)
|
||||||
|
print(f"[Dual Method Debug] Saved processed image to {debug_processed_path}")
|
||||||
|
print(f"[Dual Method Debug] Saved result image to {debug_result_path}")
|
||||||
|
except Exception as debug_e:
|
||||||
|
print(f"Error during visual debugging image generation: {debug_e}")
|
||||||
|
# --- End Visual Debugging ---
|
||||||
|
|
||||||
|
return final_result_coords # Return absolute coordinates
|
||||||
|
else:
|
||||||
|
if self.DEBUG_LEVEL > 0: # Log failure only if debug level > 0
|
||||||
|
print(f"[Dual Method] No sufficiently confident match found. Time: {elapsed_time:.3f}s")
|
||||||
|
self.last_detection_method = None
|
||||||
|
self.last_detection_confidence = 0.0
|
||||||
|
return None
|
||||||
|
|
||||||
|
def find_keyword_in_region(self, region: Tuple[int, int, int, int]) -> Optional[Tuple[int, int]]:
|
||||||
|
"""
|
||||||
|
Wrapper method to find keywords in a region.
|
||||||
|
Uses either the new dual method or the legacy method based on the 'use_dual_method' flag.
|
||||||
|
Returns absolute center coordinates or None.
|
||||||
|
"""
|
||||||
|
if region is None or len(region) != 4 or region[2] <= 0 or region[3] <= 0:
|
||||||
|
print(f"Error: Invalid region provided to find_keyword_in_region: {region}")
|
||||||
|
return None
|
||||||
|
|
||||||
|
if self.use_dual_method:
|
||||||
|
result = self.find_keyword_dual_method(region)
|
||||||
|
# Debug Fallback Check
|
||||||
|
if result is None and self.DEBUG_LEVEL >= 3:
|
||||||
|
print("[DEBUG] Dual method failed. Trying legacy method for comparison...")
|
||||||
|
legacy_result = self._find_keyword_legacy(region)
|
||||||
|
if legacy_result:
|
||||||
|
print(f"[DEBUG] Legacy method succeeded where dual method failed. Legacy Coords: {legacy_result}")
|
||||||
|
else:
|
||||||
|
print("[DEBUG] Legacy method also failed.")
|
||||||
|
return result # Return the result from the dual method
|
||||||
|
else:
|
||||||
|
# Use legacy method if dual method is disabled
|
||||||
|
return self._find_keyword_legacy(region)
|
||||||
|
|
||||||
|
def print_detection_stats(self):
|
||||||
|
"""Prints the collected keyword detection performance statistics."""
|
||||||
|
stats = self.performance_stats
|
||||||
|
total = stats['total_detections']
|
||||||
|
successful = stats['successful_detections']
|
||||||
|
|
||||||
|
if total == 0:
|
||||||
|
print("\n=== Keyword Detection Performance Stats ===")
|
||||||
|
print("No detections recorded yet.")
|
||||||
|
return
|
||||||
|
|
||||||
|
print("\n=== Keyword Detection Performance Stats ===")
|
||||||
|
print(f"Total Detection Attempts: {total}")
|
||||||
|
success_rate = (successful / total * 100) if total > 0 else 0
|
||||||
|
print(f"Successful Detections: {successful} ({success_rate:.1f}%)")
|
||||||
|
avg_time = (stats['total_detection_time'] / total * 1000) if total > 0 else 0
|
||||||
|
print(f"Average Detection Time: {avg_time:.2f} ms")
|
||||||
|
|
||||||
|
if successful > 0:
|
||||||
|
dual_pct = stats['dual_method_detections'] / successful * 100
|
||||||
|
gray_pct = stats['gray_only_detections'] / successful * 100
|
||||||
|
clahe_pct = stats['clahe_only_detections'] / successful * 100
|
||||||
|
fallback_pct = stats['fallback_detections'] / successful * 100 # Percentage of successful that were fallbacks
|
||||||
|
|
||||||
|
print("\nDetection Method Distribution (Successful Detections):")
|
||||||
|
print(f" - Dual Overlap: {stats['dual_method_detections']} ({dual_pct:.1f}%)")
|
||||||
|
print(f" - Gray Only: {stats['gray_only_detections']} ({gray_pct:.1f}%)")
|
||||||
|
print(f" - CLAHE Only: {stats['clahe_only_detections']} ({clahe_pct:.1f}%)")
|
||||||
|
# Note: Gray Only + CLAHE Only might include high-confidence singles and fallbacks.
|
||||||
|
# Fallback count is a subset of Gray/CLAHE Only.
|
||||||
|
print(f" - Fallback Used:{stats['fallback_detections']} ({fallback_pct:.1f}%)")
|
||||||
|
|
||||||
|
|
||||||
|
if stats['inverted_matches'] > 0:
|
||||||
|
inv_pct = stats['inverted_matches'] / successful * 100
|
||||||
|
print(f"\nInverted Matches Detected: {stats['inverted_matches']} ({inv_pct:.1f}%)")
|
||||||
|
print("==========================================")
|
||||||
|
|
||||||
|
|
||||||
def calculate_avatar_coords(self, bubble_tl_coords: Tuple[int, int], offset_x: int = AVATAR_OFFSET_X) -> Tuple[int, int]:
|
def calculate_avatar_coords(self, bubble_tl_coords: Tuple[int, int], offset_x: int = AVATAR_OFFSET_X) -> Tuple[int, int]:
|
||||||
"""
|
"""
|
||||||
Calculate avatar coordinates based on the EXACT top-left corner coordinates of the bubble.
|
Calculate avatar coordinates based on the EXACT top-left corner coordinates of the bubble.
|
||||||
@ -1263,41 +1605,29 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
print("\n--- Starting UI Monitoring Loop (Thread) ---")
|
print("\n--- Starting UI Monitoring Loop (Thread) ---")
|
||||||
|
|
||||||
# --- Initialization (Instantiate modules within the thread) ---
|
# --- Initialization (Instantiate modules within the thread) ---
|
||||||
# Load templates directly using constants defined in this file for now
|
# --- Template Dictionary Setup (Refactored) ---
|
||||||
# Consider passing config or a template loader object in the future
|
essential_templates = {
|
||||||
templates = {
|
# Bubble Corners (All types needed for legacy/color fallback)
|
||||||
# Regular Bubble (Original + Skins) - Keys match those used in find_dialogue_bubbles
|
|
||||||
'corner_tl': CORNER_TL_IMG, 'corner_br': CORNER_BR_IMG,
|
'corner_tl': CORNER_TL_IMG, 'corner_br': CORNER_BR_IMG,
|
||||||
'corner_tl_type2': CORNER_TL_TYPE2_IMG, 'corner_br_type2': CORNER_BR_TYPE2_IMG,
|
'corner_tl_type2': CORNER_TL_TYPE2_IMG, 'corner_br_type2': CORNER_BR_TYPE2_IMG,
|
||||||
'corner_tl_type3': CORNER_TL_TYPE3_IMG, 'corner_br_type3': CORNER_BR_TYPE3_IMG,
|
'corner_tl_type3': CORNER_TL_TYPE3_IMG, 'corner_br_type3': CORNER_BR_TYPE3_IMG,
|
||||||
'corner_tl_type4': CORNER_TL_TYPE4_IMG, 'corner_br_type4': CORNER_BR_TYPE4_IMG, # Added type4
|
'corner_tl_type4': CORNER_TL_TYPE4_IMG, 'corner_br_type4': CORNER_BR_TYPE4_IMG, # Added type4
|
||||||
# Bot Bubble (Single Type)
|
|
||||||
'bot_corner_tl': BOT_CORNER_TL_IMG, 'bot_corner_br': BOT_CORNER_BR_IMG,
|
'bot_corner_tl': BOT_CORNER_TL_IMG, 'bot_corner_br': BOT_CORNER_BR_IMG,
|
||||||
# Keywords & UI Elements
|
# Core Keywords (for dual method)
|
||||||
'keyword_wolf_lower': KEYWORD_wolf_LOWER_IMG,
|
'keyword_wolf_lower': KEYWORD_wolf_LOWER_IMG,
|
||||||
'keyword_wolf_upper': KEYWORD_Wolf_UPPER_IMG,
|
'keyword_Wolf_upper': KEYWORD_Wolf_UPPER_IMG,
|
||||||
'keyword_wolf_lower_type2': KEYWORD_wolf_LOWER_TYPE2_IMG,
|
|
||||||
'keyword_wolf_upper_type2': KEYWORD_Wolf_UPPER_TYPE2_IMG,
|
|
||||||
'keyword_wolf_lower_type3': KEYWORD_wolf_LOWER_TYPE3_IMG,
|
|
||||||
'keyword_wolf_upper_type3': KEYWORD_Wolf_UPPER_TYPE3_IMG,
|
|
||||||
'keyword_wolf_lower_type4': KEYWORD_wolf_LOWER_TYPE4_IMG, # Added type4
|
|
||||||
'keyword_wolf_upper_type4': KEYWORD_Wolf_UPPER_TYPE4_IMG, # Added type4
|
|
||||||
# --- Add Reply Keywords ---
|
|
||||||
'keyword_wolf_reply': KEYWORD_WOLF_REPLY_IMG,
|
'keyword_wolf_reply': KEYWORD_WOLF_REPLY_IMG,
|
||||||
'keyword_wolf_reply_type2': KEYWORD_WOLF_REPLY_TYPE2_IMG,
|
# Essential UI Elements
|
||||||
'keyword_wolf_reply_type3': KEYWORD_WOLF_REPLY_TYPE3_IMG,
|
|
||||||
'keyword_wolf_reply_type4': KEYWORD_WOLF_REPLY_TYPE4_IMG,
|
|
||||||
# --- End Reply Keywords ---
|
|
||||||
'copy_menu_item': COPY_MENU_ITEM_IMG, 'profile_option': PROFILE_OPTION_IMG,
|
'copy_menu_item': COPY_MENU_ITEM_IMG, 'profile_option': PROFILE_OPTION_IMG,
|
||||||
'copy_name_button': COPY_NAME_BUTTON_IMG, 'send_button': SEND_BUTTON_IMG,
|
'copy_name_button': COPY_NAME_BUTTON_IMG, 'send_button': SEND_BUTTON_IMG,
|
||||||
'chat_input': CHAT_INPUT_IMG, 'profile_name_page': PROFILE_NAME_PAGE_IMG,
|
'chat_input': CHAT_INPUT_IMG, 'profile_name_page': PROFILE_NAME_PAGE_IMG,
|
||||||
'profile_page': PROFILE_PAGE_IMG, 'chat_room': CHAT_ROOM_IMG,
|
'profile_page': PROFILE_PAGE_IMG, 'chat_room': CHAT_ROOM_IMG,
|
||||||
'base_screen': BASE_SCREEN_IMG, 'world_map_screen': WORLD_MAP_IMG, # Added for navigation
|
'base_screen': BASE_SCREEN_IMG, 'world_map_screen': WORLD_MAP_IMG,
|
||||||
'world_chat': WORLD_CHAT_IMG, 'private_chat': PRIVATE_CHAT_IMG,
|
'world_chat': WORLD_CHAT_IMG, 'private_chat': PRIVATE_CHAT_IMG,
|
||||||
# Add position templates
|
# Position templates
|
||||||
'development_pos': POS_DEV_IMG, 'interior_pos': POS_INT_IMG, 'science_pos': POS_SCI_IMG,
|
'development_pos': POS_DEV_IMG, 'interior_pos': POS_INT_IMG, 'science_pos': POS_SCI_IMG,
|
||||||
'security_pos': POS_SEC_IMG, 'strategy_pos': POS_STR_IMG,
|
'security_pos': POS_SEC_IMG, 'strategy_pos': POS_STR_IMG,
|
||||||
# Add capitol templates
|
# Capitol templates
|
||||||
'capitol_button': CAPITOL_BUTTON_IMG, 'president_title': PRESIDENT_TITLE_IMG,
|
'capitol_button': CAPITOL_BUTTON_IMG, 'president_title': PRESIDENT_TITLE_IMG,
|
||||||
'pos_btn_dev': POS_BTN_DEV_IMG, 'pos_btn_int': POS_BTN_INT_IMG, 'pos_btn_sci': POS_BTN_SCI_IMG,
|
'pos_btn_dev': POS_BTN_DEV_IMG, 'pos_btn_int': POS_BTN_INT_IMG, 'pos_btn_sci': POS_BTN_SCI_IMG,
|
||||||
'pos_btn_sec': POS_BTN_SEC_IMG, 'pos_btn_str': POS_BTN_STR_IMG,
|
'pos_btn_sec': POS_BTN_SEC_IMG, 'pos_btn_str': POS_BTN_STR_IMG,
|
||||||
@ -1305,16 +1635,34 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
'page_sec': PAGE_SEC_IMG, 'page_str': PAGE_STR_IMG,
|
'page_sec': PAGE_SEC_IMG, 'page_str': PAGE_STR_IMG,
|
||||||
'dismiss_button': DISMISS_BUTTON_IMG, 'confirm_button': CONFIRM_BUTTON_IMG,
|
'dismiss_button': DISMISS_BUTTON_IMG, 'confirm_button': CONFIRM_BUTTON_IMG,
|
||||||
'close_button': CLOSE_BUTTON_IMG, 'back_arrow': BACK_ARROW_IMG,
|
'close_button': CLOSE_BUTTON_IMG, 'back_arrow': BACK_ARROW_IMG,
|
||||||
'reply_button': REPLY_BUTTON_IMG # Added reply button template key
|
'reply_button': REPLY_BUTTON_IMG
|
||||||
}
|
}
|
||||||
# Use default confidence/region settings from constants
|
legacy_templates = {
|
||||||
# Detector now loads its own color settings internally based on hardcoded values
|
# Deprecated Keywords (for legacy method fallback)
|
||||||
detector = DetectionModule(templates,
|
'keyword_wolf_lower_type2': KEYWORD_wolf_LOWER_TYPE2_IMG,
|
||||||
confidence=CONFIDENCE_THRESHOLD,
|
'keyword_wolf_upper_type2': KEYWORD_Wolf_UPPER_TYPE2_IMG,
|
||||||
|
'keyword_wolf_lower_type3': KEYWORD_wolf_LOWER_TYPE3_IMG,
|
||||||
|
'keyword_wolf_upper_type3': KEYWORD_Wolf_UPPER_TYPE3_IMG,
|
||||||
|
'keyword_wolf_lower_type4': KEYWORD_wolf_LOWER_TYPE4_IMG,
|
||||||
|
'keyword_wolf_upper_type4': KEYWORD_Wolf_UPPER_TYPE4_IMG,
|
||||||
|
'keyword_wolf_reply_type2': KEYWORD_WOLF_REPLY_TYPE2_IMG,
|
||||||
|
'keyword_wolf_reply_type3': KEYWORD_WOLF_REPLY_TYPE3_IMG,
|
||||||
|
'keyword_wolf_reply_type4': KEYWORD_WOLF_REPLY_TYPE4_IMG,
|
||||||
|
}
|
||||||
|
# Combine dictionaries
|
||||||
|
all_templates = {**essential_templates, **legacy_templates}
|
||||||
|
# --- End Template Dictionary Setup ---
|
||||||
|
|
||||||
|
# --- Instantiate Modules ---
|
||||||
|
detector = DetectionModule(all_templates,
|
||||||
|
confidence=CONFIDENCE_THRESHOLD, # Default for legacy pyautogui calls
|
||||||
state_confidence=STATE_CONFIDENCE_THRESHOLD,
|
state_confidence=STATE_CONFIDENCE_THRESHOLD,
|
||||||
region=SCREENSHOT_REGION)
|
region=SCREENSHOT_REGION,
|
||||||
# Use default input coords/keys from constants
|
use_dual_method=True) # Enable new method by default
|
||||||
interactor = InteractionModule(detector, input_coords=(CHAT_INPUT_CENTER_X, CHAT_INPUT_CENTER_Y), input_template_key='chat_input', send_button_key='send_button')
|
interactor = InteractionModule(detector,
|
||||||
|
input_coords=(CHAT_INPUT_CENTER_X, CHAT_INPUT_CENTER_Y),
|
||||||
|
input_template_key='chat_input',
|
||||||
|
send_button_key='send_button')
|
||||||
|
|
||||||
# --- State Management (Local to this monitoring thread) ---
|
# --- State Management (Local to this monitoring thread) ---
|
||||||
last_processed_bubble_info = None # Store the whole dict now
|
last_processed_bubble_info = None # Store the whole dict now
|
||||||
@ -1322,8 +1670,13 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
screenshot_counter = 0 # Initialize counter for debug screenshots
|
screenshot_counter = 0 # Initialize counter for debug screenshots
|
||||||
main_screen_click_counter = 0 # Counter for consecutive main screen clicks
|
main_screen_click_counter = 0 # Counter for consecutive main screen clicks
|
||||||
|
|
||||||
|
loop_counter = 0 # Add loop counter for debugging
|
||||||
while True:
|
while True:
|
||||||
|
loop_counter += 1
|
||||||
|
# print(f"\n--- UI Loop Iteration #{loop_counter} ---") # DEBUG REMOVED
|
||||||
|
|
||||||
# --- Process ALL Pending Commands First ---
|
# --- Process ALL Pending Commands First ---
|
||||||
|
# print("[DEBUG] UI Loop: Processing command queue...") # DEBUG REMOVED
|
||||||
commands_processed_this_cycle = False
|
commands_processed_this_cycle = False
|
||||||
try:
|
try:
|
||||||
while True: # Loop to drain the queue
|
while True: # Loop to drain the queue
|
||||||
@ -1401,22 +1754,26 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
|
|
||||||
except queue.Empty:
|
except queue.Empty:
|
||||||
# No more commands in the queue for this cycle
|
# No more commands in the queue for this cycle
|
||||||
if commands_processed_this_cycle:
|
# if commands_processed_this_cycle: # DEBUG REMOVED
|
||||||
print("UI Thread: Finished processing commands for this cycle.")
|
# print("UI Thread: Finished processing commands for this cycle.") # DEBUG REMOVED
|
||||||
pass
|
pass
|
||||||
except Exception as cmd_err:
|
except Exception as cmd_err:
|
||||||
print(f"UI Thread: Error processing command queue: {cmd_err}")
|
print(f"UI Thread: Error processing command queue: {cmd_err}")
|
||||||
# Consider if pausing is needed on error, maybe not
|
# Consider if pausing is needed on error, maybe not
|
||||||
|
|
||||||
# --- Now, Check Pause State ---
|
# --- Now, Check Pause State ---
|
||||||
|
# print("[DEBUG] UI Loop: Checking pause state...") # DEBUG REMOVED
|
||||||
if monitoring_paused_flag[0]:
|
if monitoring_paused_flag[0]:
|
||||||
|
# print("[DEBUG] UI Loop: Monitoring is paused. Sleeping...") # DEBUG REMOVED
|
||||||
# If paused, sleep and skip UI monitoring part
|
# If paused, sleep and skip UI monitoring part
|
||||||
time.sleep(0.1) # Sleep briefly while paused
|
time.sleep(0.1) # Sleep briefly while paused
|
||||||
continue # Go back to check commands again
|
continue # Go back to check commands again
|
||||||
|
|
||||||
# --- If not paused, proceed with UI Monitoring ---
|
# --- If not paused, proceed with UI Monitoring ---
|
||||||
|
# print("[DEBUG] UI Loop: Monitoring is active. Proceeding...") # DEBUG REMOVED
|
||||||
|
|
||||||
# --- Check for Main Screen Navigation ---
|
# --- Check for Main Screen Navigation ---
|
||||||
|
# print("[DEBUG] UI Loop: Checking for main screen navigation...") # DEBUG REMOVED
|
||||||
try:
|
try:
|
||||||
base_locs = detector._find_template('base_screen', confidence=0.8)
|
base_locs = detector._find_template('base_screen', confidence=0.8)
|
||||||
map_locs = detector._find_template('world_map_screen', confidence=0.8)
|
map_locs = detector._find_template('world_map_screen', confidence=0.8)
|
||||||
@ -1447,53 +1804,8 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
# Decide if you want to continue or pause after error
|
# Decide if you want to continue or pause after error
|
||||||
main_screen_click_counter = 0 # Reset counter on error too
|
main_screen_click_counter = 0 # Reset counter on error too
|
||||||
|
|
||||||
# --- Process Commands Second (Non-blocking) ---
|
|
||||||
# This block seems redundant now as commands are processed at the start of the loop.
|
|
||||||
# Keeping it commented out for now, can be removed later if confirmed unnecessary.
|
|
||||||
# try:
|
|
||||||
# command_data = command_queue.get_nowait() # Check for commands without blocking
|
|
||||||
# action = command_data.get('action')
|
|
||||||
# if action == 'send_reply':
|
|
||||||
# text_to_send = command_data.get('text')
|
|
||||||
# # reply_context_activated = command_data.get('reply_context_activated', False) # Check if reply context was set
|
|
||||||
#
|
|
||||||
# if not text_to_send:
|
|
||||||
# print("UI Thread: Received send_reply command with no text.")
|
|
||||||
# continue # Skip if no text
|
|
||||||
#
|
|
||||||
# print(f"UI Thread: Received command to send reply: '{text_to_send[:50]}...'")
|
|
||||||
# # The reply context (clicking bubble + reply button) is now handled *before* putting into queue.
|
|
||||||
# # So, we just need to send the message directly here.
|
|
||||||
# # The input field should already be focused and potentially have @Username prefix if reply context was activated.
|
|
||||||
# interactor.send_chat_message(text_to_send)
|
|
||||||
#
|
|
||||||
# elif action == 'remove_position': # <--- Handle new command
|
|
||||||
# region = command_data.get('trigger_bubble_region')
|
|
||||||
# if region:
|
|
||||||
# print(f"UI Thread: Received command to remove position triggered by bubble region: {region}")
|
|
||||||
# # Call the new UI function
|
|
||||||
# success = remove_user_position(detector, interactor, region) # Call synchronous function
|
|
||||||
# print(f"UI Thread: Position removal attempt finished. Success: {success}")
|
|
||||||
# # Note: No need to send result back unless main thread needs confirmation
|
|
||||||
# else:
|
|
||||||
# print("UI Thread: Received remove_position command without trigger_bubble_region.")
|
|
||||||
# elif action == 'pause': # <--- Handle pause command
|
|
||||||
# print("UI Thread: Received pause command. Pausing monitoring.")
|
|
||||||
# monitoring_paused_flag[0] = True
|
|
||||||
# continue # Immediately pause after receiving command
|
|
||||||
# elif action == 'resume': # <--- Handle resume command (might be redundant if checked above, but safe)
|
|
||||||
# print("UI Thread: Received resume command. Resuming monitoring.")
|
|
||||||
# monitoring_paused_flag[0] = False
|
|
||||||
# else:
|
|
||||||
# print(f"UI Thread: Received unknown command: {action}")
|
|
||||||
# except queue.Empty:
|
|
||||||
# pass # No command waiting, continue with monitoring
|
|
||||||
# except Exception as cmd_err:
|
|
||||||
# print(f"UI Thread: Error processing command queue: {cmd_err}")
|
|
||||||
# # This block is now part of the command processing loop above
|
|
||||||
# pass
|
|
||||||
|
|
||||||
# --- Verify Chat Room State Before Bubble Detection (Only if NOT paused) ---
|
# --- Verify Chat Room State Before Bubble Detection (Only if NOT paused) ---
|
||||||
|
# print("[DEBUG] UI Loop: Verifying chat room state...") # DEBUG REMOVED
|
||||||
try:
|
try:
|
||||||
# Use a slightly lower confidence maybe, or state_confidence
|
# Use a slightly lower confidence maybe, or state_confidence
|
||||||
chat_room_locs = detector._find_template('chat_room', confidence=detector.state_confidence)
|
chat_room_locs = detector._find_template('chat_room', confidence=detector.state_confidence)
|
||||||
@ -1505,8 +1817,8 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
print("UI Thread: Continuing loop after attempting chat room cleanup.")
|
print("UI Thread: Continuing loop after attempting chat room cleanup.")
|
||||||
time.sleep(0.5) # Small pause after cleanup attempt
|
time.sleep(0.5) # Small pause after cleanup attempt
|
||||||
continue
|
continue
|
||||||
# else: # Optional: Log if chat room is confirmed
|
# else: # Optional: Log if chat room is confirmed # DEBUG REMOVED
|
||||||
# print("UI Thread: Chat room state confirmed.")
|
# print("[DEBUG] UI Thread: Chat room state confirmed.") # DEBUG REMOVED
|
||||||
|
|
||||||
except Exception as state_check_err:
|
except Exception as state_check_err:
|
||||||
print(f"UI Thread: Error checking for chat room state: {state_check_err}")
|
print(f"UI Thread: Error checking for chat room state: {state_check_err}")
|
||||||
@ -1515,24 +1827,34 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
|
|
||||||
|
|
||||||
# --- Then Perform UI Monitoring (Bubble Detection) ---
|
# --- Then Perform UI Monitoring (Bubble Detection) ---
|
||||||
|
# print("[DEBUG] UI Loop: Starting bubble detection...") # DEBUG REMOVED
|
||||||
try:
|
try:
|
||||||
# 1. Detect Bubbles
|
# 1. Detect Bubbles
|
||||||
all_bubbles_data = detector.find_dialogue_bubbles() # Returns list of dicts
|
all_bubbles_data = detector.find_dialogue_bubbles() # Returns list of dicts
|
||||||
if not all_bubbles_data: time.sleep(2); continue
|
if not all_bubbles_data:
|
||||||
|
# print("[DEBUG] UI Loop: No bubbles detected.") # DEBUG REMOVED
|
||||||
|
time.sleep(2); continue
|
||||||
|
|
||||||
# Filter out bot bubbles
|
# Filter out bot bubbles
|
||||||
other_bubbles_data = [b_info for b_info in all_bubbles_data if not b_info['is_bot']]
|
other_bubbles_data = [b_info for b_info in all_bubbles_data if not b_info['is_bot']]
|
||||||
if not other_bubbles_data: time.sleep(0.2); continue
|
if not other_bubbles_data:
|
||||||
|
# print("[DEBUG] UI Loop: No non-bot bubbles detected.") # DEBUG REMOVED
|
||||||
|
time.sleep(0.2); continue
|
||||||
|
|
||||||
|
# print(f"[DEBUG] UI Loop: Found {len(other_bubbles_data)} non-bot bubbles. Sorting...") # DEBUG REMOVED
|
||||||
# Sort bubbles from bottom to top (based on bottom Y coordinate)
|
# Sort bubbles from bottom to top (based on bottom Y coordinate)
|
||||||
sorted_bubbles = sorted(other_bubbles_data, key=lambda b_info: b_info['bbox'][3], reverse=True)
|
sorted_bubbles = sorted(other_bubbles_data, key=lambda b_info: b_info['bbox'][3], reverse=True)
|
||||||
|
|
||||||
# Iterate through sorted bubbles (bottom to top)
|
# Iterate through sorted bubbles (bottom to top)
|
||||||
for target_bubble_info in sorted_bubbles:
|
# print("[DEBUG] UI Loop: Iterating through sorted bubbles...") # DEBUG REMOVED
|
||||||
|
for i, target_bubble_info in enumerate(sorted_bubbles):
|
||||||
|
# print(f"[DEBUG] UI Loop: Processing bubble #{i+1}") # DEBUG REMOVED
|
||||||
target_bbox = target_bubble_info['bbox']
|
target_bbox = target_bubble_info['bbox']
|
||||||
bubble_region = (target_bbox[0], target_bbox[1], target_bbox[2]-target_bbox[0], target_bbox[3]-target_bbox[1])
|
# Ensure bubble_region uses standard ints
|
||||||
|
bubble_region = (int(target_bbox[0]), int(target_bbox[1]), int(target_bbox[2]-target_bbox[0]), int(target_bbox[3]-target_bbox[1]))
|
||||||
|
|
||||||
# 3. Detect Keyword in Bubble
|
# 3. Detect Keyword in Bubble
|
||||||
|
# print(f"[DEBUG] UI Loop: Detecting keyword in region {bubble_region}...") # DEBUG REMOVED
|
||||||
keyword_coords = detector.find_keyword_in_region(bubble_region)
|
keyword_coords = detector.find_keyword_in_region(bubble_region)
|
||||||
|
|
||||||
if keyword_coords:
|
if keyword_coords:
|
||||||
@ -1567,6 +1889,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
print("Warning: SCREENSHOT_REGION not defined, searching full screen for bubble snapshot.")
|
print("Warning: SCREENSHOT_REGION not defined, searching full screen for bubble snapshot.")
|
||||||
|
|
||||||
# --- Take Snapshot for Re-location ---
|
# --- Take Snapshot for Re-location ---
|
||||||
|
# print("[DEBUG] UI Loop: Taking bubble snapshot...") # DEBUG REMOVED
|
||||||
try:
|
try:
|
||||||
bubble_region_tuple = (int(bubble_region[0]), int(bubble_region[1]), int(bubble_region[2]), int(bubble_region[3]))
|
bubble_region_tuple = (int(bubble_region[0]), int(bubble_region[1]), int(bubble_region[2]), int(bubble_region[3]))
|
||||||
if bubble_region_tuple[2] <= 0 or bubble_region_tuple[3] <= 0:
|
if bubble_region_tuple[2] <= 0 or bubble_region_tuple[3] <= 0:
|
||||||
@ -1594,7 +1917,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
continue # Skip to next bubble
|
continue # Skip to next bubble
|
||||||
|
|
||||||
# 4. Re-locate bubble *before* copying text
|
# 4. Re-locate bubble *before* copying text
|
||||||
print("Attempting to re-locate bubble before copying text...")
|
# print("[DEBUG] UI Loop: Re-locating bubble before copying text...") # DEBUG REMOVED
|
||||||
new_bubble_box_for_copy = None
|
new_bubble_box_for_copy = None
|
||||||
if bubble_snapshot:
|
if bubble_snapshot:
|
||||||
try:
|
try:
|
||||||
@ -1610,11 +1933,12 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
continue # Skip to the next bubble in the outer loop
|
continue # Skip to the next bubble in the outer loop
|
||||||
|
|
||||||
print(f"Successfully re-located bubble for copy at: {new_bubble_box_for_copy}")
|
print(f"Successfully re-located bubble for copy at: {new_bubble_box_for_copy}")
|
||||||
# Define the region based on the re-located bubble to find the keyword again
|
# Define the region based on the re-located bubble, casting to int
|
||||||
copy_bubble_region = (new_bubble_box_for_copy.left, new_bubble_box_for_copy.top,
|
copy_bubble_region = (int(new_bubble_box_for_copy.left), int(new_bubble_box_for_copy.top),
|
||||||
new_bubble_box_for_copy.width, new_bubble_box_for_copy.height)
|
int(new_bubble_box_for_copy.width), int(new_bubble_box_for_copy.height))
|
||||||
|
|
||||||
# Find the keyword *again* within the *new* bubble region to get current coords
|
# Find the keyword *again* within the *new* bubble region to get current coords
|
||||||
|
# print("[DEBUG] UI Loop: Finding keyword again in re-located region...") # DEBUG REMOVED
|
||||||
current_keyword_coords = detector.find_keyword_in_region(copy_bubble_region)
|
current_keyword_coords = detector.find_keyword_in_region(copy_bubble_region)
|
||||||
if not current_keyword_coords:
|
if not current_keyword_coords:
|
||||||
print("Warning: Keyword not found in the re-located bubble region. Skipping this bubble.")
|
print("Warning: Keyword not found in the re-located bubble region. Skipping this bubble.")
|
||||||
@ -1634,6 +1958,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
print(f"Detected keyword is not a reply type (current location). Click target: {click_coords_current}")
|
print(f"Detected keyword is not a reply type (current location). Click target: {click_coords_current}")
|
||||||
|
|
||||||
# Interact: Get Bubble Text using current coordinates
|
# Interact: Get Bubble Text using current coordinates
|
||||||
|
# print("[DEBUG] UI Loop: Copying text...") # DEBUG REMOVED
|
||||||
bubble_text = interactor.copy_text_at(click_coords_current)
|
bubble_text = interactor.copy_text_at(click_coords_current)
|
||||||
if not bubble_text:
|
if not bubble_text:
|
||||||
print("Error: Could not get dialogue content for this bubble (after re-location).")
|
print("Error: Could not get dialogue content for this bubble (after re-location).")
|
||||||
@ -1641,6 +1966,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
continue # Skip to next bubble
|
continue # Skip to next bubble
|
||||||
|
|
||||||
# Check recent text history
|
# Check recent text history
|
||||||
|
# print("[DEBUG] UI Loop: Checking recent text history...") # DEBUG REMOVED
|
||||||
if bubble_text in recent_texts:
|
if bubble_text in recent_texts:
|
||||||
print(f"Content '{bubble_text[:30]}...' in recent history, skipping this bubble.")
|
print(f"Content '{bubble_text[:30]}...' in recent history, skipping this bubble.")
|
||||||
continue # Skip to next bubble
|
continue # Skip to next bubble
|
||||||
@ -1650,6 +1976,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
recent_texts.append(bubble_text)
|
recent_texts.append(bubble_text)
|
||||||
|
|
||||||
# 5. Interact: Get Sender Name (uses re-location internally via retrieve_sender_name_interaction)
|
# 5. Interact: Get Sender Name (uses re-location internally via retrieve_sender_name_interaction)
|
||||||
|
# print("[DEBUG] UI Loop: Retrieving sender name...") # DEBUG REMOVED
|
||||||
sender_name = None
|
sender_name = None
|
||||||
try:
|
try:
|
||||||
# --- Bubble Re-location Logic ---
|
# --- Bubble Re-location Logic ---
|
||||||
@ -1715,6 +2042,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
continue # Skip to next bubble
|
continue # Skip to next bubble
|
||||||
|
|
||||||
# 6. Perform Cleanup
|
# 6. Perform Cleanup
|
||||||
|
# print("[DEBUG] UI Loop: Performing cleanup after getting name...") # DEBUG REMOVED
|
||||||
cleanup_successful = perform_state_cleanup(detector, interactor)
|
cleanup_successful = perform_state_cleanup(detector, interactor)
|
||||||
if not cleanup_successful:
|
if not cleanup_successful:
|
||||||
print("Error: Failed to return to chat screen after getting name. Skipping this bubble.")
|
print("Error: Failed to return to chat screen after getting name. Skipping this bubble.")
|
||||||
@ -1725,6 +2053,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
continue # Skip to next bubble
|
continue # Skip to next bubble
|
||||||
|
|
||||||
# --- Attempt to activate reply context ---
|
# --- Attempt to activate reply context ---
|
||||||
|
# print("[DEBUG] UI Loop: Attempting to activate reply context...") # DEBUG REMOVED
|
||||||
reply_context_activated = False
|
reply_context_activated = False
|
||||||
try:
|
try:
|
||||||
print("Attempting to activate reply context...")
|
print("Attempting to activate reply context...")
|
||||||
@ -1801,6 +2130,7 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu
|
|||||||
|
|
||||||
# If the loop finished without breaking (i.e., no trigger processed), wait the full interval.
|
# If the loop finished without breaking (i.e., no trigger processed), wait the full interval.
|
||||||
# If it broke, the sleep still happens here before the next cycle.
|
# If it broke, the sleep still happens here before the next cycle.
|
||||||
|
# print("[DEBUG] UI Loop: Finished bubble iteration or broke early. Sleeping...") # DEBUG REMOVED
|
||||||
time.sleep(1.5) # Polling interval after checking all bubbles or processing one
|
time.sleep(1.5) # Polling interval after checking all bubbles or processing one
|
||||||
|
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user