diff --git a/.gitignore b/.gitignore index 4d9ae04..2e99f16 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ .env +*.log llm_debug.log -__pycache__/ \ No newline at end of file +__pycache__/ +debug_screenshots/ +chat_logs/ \ No newline at end of file diff --git a/ClaudeCode.md b/ClaudeCode.md index 00322f3..4b0d65a 100644 --- a/ClaudeCode.md +++ b/ClaudeCode.md @@ -52,6 +52,10 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機 7. **視窗設定工具 (window-setup-script.py)** - 輔助工具,用於設置遊戲視窗的位置和大小 - 方便開發階段截取 UI 元素樣本 +8. **視窗監視工具 (window-monitor-script.py)** + - (新增) 強化腳本,用於持續監視遊戲視窗 + - 確保目標視窗維持在最上層 (Always on Top) + - 自動將視窗移回指定的位置 ### 資料流程 @@ -75,11 +79,26 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機 系統使用基於圖像辨識的方法監控遊戲聊天界面: -1. **泡泡檢測**:通過辨識聊天泡泡的角落圖案定位聊天訊息,區分一般用戶與機器人 -2. **關鍵字檢測**:在泡泡區域內搜尋 "wolf" 或 "Wolf" 關鍵字圖像 -3. **內容獲取**:點擊關鍵字位置,使用剪貼板複製聊天內容 -4. **發送者識別**:通過點擊頭像,導航菜單,複製用戶名稱 -5. **防重複處理**:使用位置比較和內容歷史記錄防止重複回應 +1. **泡泡檢測(含 Y 軸優先配對)**:通過辨識聊天泡泡的左上角 (TL) 和右下角 (BR) 角落圖案定位聊天訊息。 + - **多外觀支援**:為了適應玩家可能使用的不同聊天泡泡外觀 (skin),一般用戶泡泡的偵測機制已被擴充,可以同時尋找多組不同的角落模板 (例如 `corner_tl_type2.png`, `corner_br_type2.png` 等)。機器人泡泡目前僅偵測預設的角落模板。 + - **配對邏輯優化**:在配對 TL 和 BR 角落時,系統現在會優先選擇與 TL 角落 **Y 座標最接近** 的有效 BR 角落,以更好地區分垂直堆疊的聊天泡泡。 +2. **關鍵字檢測**:在泡泡區域內搜尋 "wolf" 或 "Wolf" 關鍵字圖像。 +3. **內容獲取**:點擊關鍵字位置,使用剪貼板複製聊天內容。 +4. **發送者識別(含氣泡重新定位與偏移量調整)**:**關鍵步驟** - 為了提高在動態聊天環境下的穩定性,系統在獲取發送者名稱前,會執行以下步驟: + a. **初始偵測**:像之前一樣,根據偵測到的關鍵字定位觸發的聊天泡泡。 + b. **氣泡快照**:擷取該聊天泡泡的圖像快照。 + c. **重新定位**:在點擊頭像前,使用該快照在當前聊天視窗區域內重新搜尋氣泡的最新位置。 + d. **計算座標(新偏移量)**: + - 如果成功重新定位氣泡,則根據找到的**新**左上角座標 (`new_tl_x`, `new_tl_y`),應用新的偏移量計算頭像點擊位置:`x = new_tl_x - 45` (`AVATAR_OFFSET_X_REPLY`),`y = new_tl_y + 10` (`AVATAR_OFFSET_Y_REPLY`)。 + - 如果無法重新定位(例如氣泡已滾動出畫面),則跳過此次互動,以避免點擊錯誤位置。 + e. **互動(含重試)**: + - 使用計算出的(新的)頭像位置進行第一次點擊。 + - 檢查是否成功進入個人資料頁面 (`Profile_page.png`)。 + - **如果失敗**:系統會使用步驟 (b) 的氣泡快照,在聊天區域內重新定位氣泡,重新計算頭像座標,然後再次嘗試點擊。此過程最多重複 3 次。 + - **如果成功**(無論是首次嘗試還是重試成功):繼續導航菜單,最終複製用戶名稱。 + - **如果重試後仍失敗**:放棄獲取該用戶名稱。 + f. **原始偏移量**:原始的 `-55` 像素水平偏移量 (`AVATAR_OFFSET_X`) 仍保留在程式碼中,用於其他不需要重新定位或不同互動邏輯的場景(例如 `remove_user_position` 功能)。 +5. **防重複處理**:使用位置比較和內容歷史記錄防止重複回應。 #### LLM 整合 @@ -112,10 +131,20 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機 系統使用多種技術實現 UI 自動化: -1. **圖像辨識**:使用 OpenCV 和 pyautogui 進行圖像匹配和識別 -2. **鍵鼠控制**:模擬鼠標點擊和鍵盤操作 -3. **剪貼板操作**:使用 pyperclip 讀寫剪貼板 -4. **狀態式處理**:基於 UI 狀態判斷的互動流程,確保操作穩定性 +1. **圖像辨識**:使用 OpenCV 和 pyautogui 進行圖像匹配和識別。 +2. **鍵鼠控制**:模擬鼠標點擊和鍵盤操作。 +3. **剪貼板操作**:使用 pyperclip 讀寫剪貼板。 +4. **狀態式處理**:基於 UI 狀態判斷的互動流程,確保操作穩定性。 +5. **針對性回覆(上下文激活)**: + - **時機**:在成功獲取發送者名稱並返回聊天介面後,但在將觸發資訊放入隊列傳遞給主線程之前。 + - **流程**: + a. 再次使用氣泡快照重新定位觸發訊息的氣泡。 + b. 如果定位成功,點擊氣泡中心,並等待 0.25 秒(增加的延遲時間)以允許 UI 反應。 + c. 尋找並點擊彈出的「回覆」按鈕 (`reply_button.png`)。 + d. 如果成功點擊回覆按鈕,則設置一個 `reply_context_activated` 標記為 `True`。 + e. 如果重新定位氣泡失敗或未找到回覆按鈕,則該標記為 `False`。 + - **傳遞**:將 `reply_context_activated` 標記連同其他觸發資訊(發送者、內容、氣泡區域)一起放入隊列。 + - **發送**:主控模塊 (`main.py`) 在處理 `send_reply` 命令時,不再需要執行點擊回覆的操作,只需直接調用 `send_chat_message` 即可(因為如果 `reply_context_activated` 為 `True`,輸入框應已準備好)。 ## 配置與部署 @@ -167,6 +196,174 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機 這些優化確保了即使在複雜工具調用後,Wolfhart 也能保持角色一致性,並提供合適的回應。無效回應不再發送到遊戲,提高了用戶體驗。 +## 最近改進(2025-04-18) + +### 支援多種一般聊天泡泡外觀,並修正先前錯誤配置 + +- **UI 互動模塊 (`ui_interaction.py`)**: + - **修正**:先前錯誤地將多外觀支援應用於機器人泡泡。現已修正 `find_dialogue_bubbles` 函數,使其能夠載入並搜尋多組**一般用戶**泡泡的角落模板(例如 `corner_tl_type2.png`, `corner_br_type2.png` 等)。 + - 允許任何類型的一般用戶左上角與任何類型的一般用戶右下角進行配對,只要符合幾何條件。 + - 機器人泡泡的偵測恢復為僅使用預設的 `bot_corner_tl.png` 和 `bot_corner_br.png` 模板。 + - 這提高了對使用了自訂聊天泡泡外觀的**一般玩家**訊息的偵測能力。 +- **模板文件**: + - 在 `ui_interaction.py` 中為一般角落定義了新類型模板的路徑(`_type2`, `_type3`)。 + - **注意:** 需要在 `templates` 資料夾中實際添加對應的 `corner_tl_type2.png`, `corner_br_type2.png` 等圖片檔案才能生效。 +- **文件更新 (`ClaudeCode.md`)**: + - 在「技術實現」部分更新了泡泡檢測的說明。 + - 添加了此「最近改進」條目,並修正了先前的描述。 + +### 頭像點擊偏移量調整 + +- **UI 互動模塊 (`ui_interaction.py`)**: + - 將 `AVATAR_OFFSET_X` 常數的值從 `-50` 調整為 `-55`。 + - 這統一了常規關鍵字觸發流程和 `remove_user_position` 功能中計算頭像點擊位置時使用的水平偏移量。 +- **文件更新 (`ClaudeCode.md`)**: + - 在「技術實現」的「發送者識別」部分強調了點擊位置是相對於觸發泡泡計算的,並註明了新的偏移量。 + - 添加了此「最近改進」條目。 + +### 聊天泡泡重新定位以提高穩定性 + +- **UI 互動模塊 (`ui_interaction.py`)**: + - 在 `run_ui_monitoring_loop` 中,於偵測到關鍵字並成功複製文字後、獲取發送者名稱前,加入了新的邏輯: + 1. 擷取觸發氣泡的圖像快照。 + 2. 使用 `pyautogui.locateOnScreen` 在聊天區域內重新尋找該快照的當前位置。 + 3. 若找到,則根據**新位置**的左上角座標和新的偏移量 (`AVATAR_OFFSET_X_RELOCATED = -50`) 計算頭像點擊位置。 + 4. 若找不到,則記錄警告並跳過此次互動。 + - 新增了 `AVATAR_OFFSET_X_RELOCATED` 和 `BUBBLE_RELOCATE_CONFIDENCE` 常數。 +- **目的**:解決聊天視窗內容滾動後,原始偵測到的氣泡位置失效,導致點擊錯誤頭像的問題。透過重新定位,確保點擊的是與觸發訊息相對應的頭像。 +- **文件更新 (`ClaudeCode.md`)**: + - 更新了「技術實現」中的「發送者識別」部分,詳細說明了重新定位的步驟。 + - 在此「最近改進」部分添加了這個新條目。 + +### 互動流程優化 (頭像偏移、氣泡配對、針對性回覆) + +- **UI 互動模塊 (`ui_interaction.py`)**: + - **頭像偏移量調整**:修改了重新定位氣泡後計算頭像座標的邏輯,使用新的偏移量:左 `-45` (`AVATAR_OFFSET_X_REPLY`),下 `+10` (`AVATAR_OFFSET_Y_REPLY`)。原始的 `-55` 偏移量 (`AVATAR_OFFSET_X`) 保留用於其他功能。 + - **氣泡配對優化**:修改 `find_dialogue_bubbles` 函數,使其在配對左上角 (TL) 和右下角 (BR) 時,優先選擇 Y 座標差異最小的 BR 角落,以提高垂直相鄰氣泡的區分度。 + - **頭像點擊重試**:修改 `retrieve_sender_name_interaction` 函數,增加了最多 3 次的重試邏輯。如果在點擊頭像後未能檢測到個人資料頁面,會嘗試重新定位氣泡並再次點擊。 + - **針對性回覆時機調整與延遲增加**: + - 將點擊氣泡中心和回覆按鈕的操作移至成功獲取發送者名稱並返回聊天室之後、將觸發資訊放入隊列之前。 + - **增加了點擊氣泡中心後、尋找回覆按鈕前的等待時間至 0.25 秒**,以提高在 UI 反應較慢時找到按鈕的成功率。 + - 在放入隊列的數據中增加 `reply_context_activated` 標記,指示是否成功激活了回覆上下文。 + - 簡化了處理 `send_reply` 命令的邏輯,使其僅負責發送消息。 + - **氣泡快照保存 (用於除錯)**:在偵測到關鍵字後,擷取用於重新定位的氣泡圖像快照 (`bubble_snapshot`) 時,會將此快照保存到 `debug_screenshots` 文件夾中,檔名格式為 `debug_relocation_snapshot_X.png` (X 為 1 到 5 的循環數字)。這取代了先前僅保存氣泡區域截圖的邏輯。 +- **目的**: + - 進一步提高獲取發送者名稱的穩定性。 + - 改善氣泡配對的準確性。 + - 調整針對性回覆的流程,使其更符合邏輯順序,並通過增加延遲提高可靠性。 + - 提供用於重新定位的實際圖像快照,方便除錯。 +- **文件更新 (`ClaudeCode.md`)**: + - 更新了「技術實現」中的「泡泡檢測」、「發送者識別」部分。 + - 更新了「UI 自動化」部分關於「針對性回覆」的說明,反映了新的時機、標記和增加的延遲。 + - 在此「最近改進」部分更新了這個匯總條目,以包含最新的修改(包括快照保存和延遲增加)。 + +### UI 監控暫停與恢復機制 (2025-04-18) + +- **目的**:解決在等待 LLM 回應期間,持續的 UI 監控可能導致的不穩定性或干擾問題,特別是與 `remove_position` 等需要精確 UI 狀態的操作相關。 +- **`ui_interaction.py`**: + - 引入了全局(模塊級)`monitoring_paused_flag` 列表(包含一個布爾值)。 + - 在 `run_ui_monitoring_loop` 的主循環開始處檢查此標誌。若為 `True`,則循環僅檢查命令隊列中的 `resume` 命令並休眠,跳過所有 UI 偵測和觸發邏輯。 + - 在命令處理邏輯中添加了對 `pause` 和 `resume` 動作的處理,分別設置 `monitoring_paused_flag[0]` 為 `True` 或 `False`。 +- **`ui_interaction.py` (進一步修改)**: + - **修正命令處理邏輯**:修改了 `run_ui_monitoring_loop` 的主循環。現在,在每次迭代開始時,它會使用一個內部 `while True` 循環和 `command_queue.get_nowait()` 來**處理完隊列中所有待處理的命令**(包括 `pause`, `resume`, `send_reply`, `remove_position` 等)。 + - **狀態檢查後置**:只有在清空當前所有命令後,循環才會檢查 `monitoring_paused_flag` 的狀態。如果標誌為 `True`,則休眠並跳過 UI 監控部分;如果為 `False`,則繼續執行 UI 監控(畫面檢查、氣泡偵測等)。 + - **目的**:解決先前版本中 `resume` 命令可能導致 UI 線程過早退出暫停狀態,從而錯過緊隨其後的 `send_reply` 或 `remove_position` 命令的問題。確保所有來自 `main.py` 的命令都被及時處理。 +- **`main.py`**: + - (先前修改保持不變)在主處理循環 (`run_main_with_exit_stack` 的 `while True` 循環) 中: + - 在從 `trigger_queue` 獲取數據後、調用 `llm_interaction.get_llm_response` **之前**,向 `command_queue` 發送 `{ 'action': 'pause' }` 命令。 + - 使用 `try...finally` 結構,確保在處理 LLM 回應(包括命令處理和發送回覆)**之後**,向 `command_queue` 發送 `{ 'action': 'resume' }` 命令,無論處理過程中是否發生錯誤。 + +### `remove_position` 穩定性改進 (使用快照重新定位) (2025-04-19) + +- **目的**:解決 `remove_position` 命令因聊天視窗滾動導致基於舊氣泡位置計算座標而出錯的問題。 +- **`ui_interaction.py` (`run_ui_monitoring_loop`)**: + - 在觸發事件放入 `trigger_queue` 的數據中,額外添加了 `bubble_snapshot`(觸發氣泡的圖像快照)和 `search_area`(用於快照的搜索區域)。 +- **`main.py`**: + - 修改了處理 `remove_position` 命令的邏輯,使其從 `trigger_data` 中提取 `bubble_snapshot` 和 `search_area`,並將它們包含在發送給 `command_queue` 的命令數據中。 +- **`ui_interaction.py` (`remove_user_position` 函數)**: + - 修改了函數簽名,以接收 `bubble_snapshot` 和 `search_area` 參數。 + - 在函數執行開始時,使用傳入的 `bubble_snapshot` 和 `search_area` 調用 `pyautogui.locateOnScreen` 來重新定位觸發氣泡的當前位置。 + - 如果重新定位失敗,則記錄錯誤並返回 `False`。 + - 如果重新定位成功,則後續所有基於氣泡位置的計算(包括尋找職位圖標的搜索區域 `search_region` 和點擊頭像的座標 `avatar_click_x`, `avatar_click_y`)都將使用這個**新找到的**氣泡座標。 +- **效果**:確保 `remove_position` 操作基於氣泡的最新位置執行,提高了在動態滾動的聊天界面中的可靠性。 + +### 修正 Type3 關鍵字辨識並新增 Type4 支援 (2025-04-19) + +- **目的**:修復先前版本中 `type3` 關鍵字辨識的錯誤,並擴充系統以支援新的 `type4` 聊天泡泡外觀和對應的關鍵字樣式。 +- **`ui_interaction.py`**: + - **修正 `find_keyword_in_region`**:移除了錯誤使用 `type2` 模板鍵來尋找 `type3` 關鍵字的重複程式碼,確保 `type3` 關鍵字使用正確的模板 (`keyword_wolf_lower_type3`, `keyword_wolf_upper_type3`)。 + - **新增 `type4` 泡泡支援**: + - 在檔案開頭定義了 `type4` 角落模板的路徑常數 (`CORNER_TL_TYPE4_IMG`, `CORNER_BR_TYPE4_IMG`)。 + - 在 `find_dialogue_bubbles` 函數中,將 `type4` 的模板鍵 (`corner_tl_type4`, `corner_br_type4`) 加入 `regular_tl_keys` 和 `regular_br_keys` 列表。 + - 在 `run_ui_monitoring_loop` 的 `templates` 字典中加入了對應的鍵值對。 + - **新增 `type4` 關鍵字支援**: + - 在檔案開頭定義了 `type4` 關鍵字模板的路徑常數 (`KEYWORD_wolf_LOWER_TYPE4_IMG`, `KEYWORD_Wolf_UPPER_TYPE4_IMG`)。 + - 在 `find_keyword_in_region` 函數中,加入了尋找 `type4` 關鍵字模板 (`keyword_wolf_lower_type4`, `keyword_wolf_upper_type4`) 的邏輯。 + - 在 `run_ui_monitoring_loop` 的 `templates` 字典中加入了對應的鍵值對。 +- **效果**:提高了對 `type3` 關鍵字的辨識準確率,並使系統能夠辨識 `type4` 的聊天泡泡和關鍵字(前提是提供了對應的模板圖片)。 + +### 新增 Reply 關鍵字偵測與點擊偏移 (2025-04-20) + +- **目的**:擴充關鍵字偵測機制,使其能夠辨識特定的回覆指示圖片 (`keyword_wolf_reply.png` 及其 type2, type3, type4 變體),並在點擊這些特定圖片以複製文字時,應用 Y 軸偏移。 +- **`ui_interaction.py`**: + - **新增模板**:定義了 `KEYWORD_WOLF_REPLY_IMG` 系列常數,並將其加入 `run_ui_monitoring_loop` 中的 `templates` 字典。 + - **擴充偵測**:修改 `find_keyword_in_region` 函數,加入對 `keyword_wolf_reply` 系列模板的搜尋邏輯。 + - **條件式偏移**:在 `run_ui_monitoring_loop` 中,於偵測到關鍵字後,加入判斷邏輯。如果偵測到的關鍵字是 `keyword_wolf_reply` 系列之一,則: + 1. 計算用於 `copy_text_at` 的點擊座標時,Y 座標會增加 15 像素。 + 2. 在後續嘗試激活回覆上下文時,計算用於點擊**氣泡中心**的座標時,Y 座標**也會**增加 15 像素。 + - 其他關鍵字或 UI 元素的點擊不受影響。 +- **效果**:系統現在可以偵測新的回覆指示圖片作為觸發條件。當由這些圖片觸發時,用於複製文字的點擊和用於激活回覆上下文的氣泡中心點擊都會向下微調 15 像素,以避免誤觸其他 UI 元素。 + +### 強化 LLM 上下文處理與回應生成 (2025-04-20) + +- **目的**:解決 LLM 可能混淆歷史對話與當前訊息,以及在回應中包含歷史記錄的問題。確保 `dialogue` 欄位只包含針對最新用戶訊息的新回覆。 +- **`llm_interaction.py`**: + - **修改 `get_system_prompt`**: + - 在 `dialogue` 欄位的規則中,明確禁止包含任何歷史記錄,並強調必須只回應標記為 `` 的最新訊息。 + - 在核心指令中,要求 LLM 將分析和回應生成完全集中在 `` 標記的訊息上。 + - 新增了對 `` 標記作用的說明。 + - **修改 `_build_context_messages`**: + - 在構建發送給 LLM 的訊息列表時,將歷史記錄中的最後一條用戶訊息用 `...` 標籤包裹起來。 + - 其他歷史訊息保持原有的 `[timestamp] speaker: message` 格式。 +- **效果**:通過更嚴格的提示和明確的上下文標記,引導 LLM 準確區分當前互動和歷史對話,預期能提高回應的相關性並防止輸出冗餘的歷史內容。 + +### 強化 System Prompt 以鼓勵工具使用 (2025-04-19) + +- **目的**:調整 `llm_interaction.py` 中的 `get_system_prompt` 函數,使其更明確地引導 LLM 在回應前主動使用工具(特別是記憶體工具)和整合工具資訊。 +- **修改內容**: + 1. **核心身份強化**:在 `CORE IDENTITY AND TOOL USAGE` 部分加入新的一點,強調 Wolfhart 會主動查閱內部知識圖譜和外部來源。 + 2. **記憶體指示強化**:將 `Memory Management (Knowledge Graph)` 部分的提示從 "IMPORTANT" 改為 "CRITICAL",並明確指示在回應*之前*要考慮使用查詢工具檢查記憶體,同時也強調了寫入新資訊的主動性。 +- **效果**:旨在提高 LLM 使用工具的主動性和依賴性,使其回應更具上下文感知和資訊準確性,同時保持角色一致性。 + +### 聊天歷史記錄上下文與日誌記錄 (2025-04-20) + +- **目的**: + 1. 為 LLM 提供更豐富的對話上下文,以生成更連貫和相關的回應。 + 2. 新增一個可選的聊天日誌功能,用於調試和記錄。 +- **`main.py`**: + - 引入 `collections.deque` 來儲存最近的對話歷史(用戶訊息和機器人回應),上限為 50 條。 + - 在調用 `llm_interaction.get_llm_response` 之前,將用戶訊息添加到歷史記錄中。 + - 在收到有效的 LLM 回應後,將機器人回應添加到歷史記錄中。 + - 新增 `log_chat_interaction` 函數,該函數: + - 檢查 `config.ENABLE_CHAT_LOGGING` 標誌。 + - 如果啟用,則在 `config.LOG_DIR` 指定的文件夾中創建或附加到以日期命名的日誌文件 (`YYYY-MM-DD.log`)。 + - 記錄包含時間戳、發送者(用戶/機器人)、發送者名稱和訊息內容的條目。 + - 在收到有效 LLM 回應後調用 `log_chat_interaction`。 +- **`llm_interaction.py`**: + - 修改 `get_llm_response` 函數簽名,接收 `current_sender_name` 和 `history` 列表,而不是單個 `user_input`。 + - 新增 `_build_context_messages` 輔助函數,該函數: + - 根據規則從 `history` 中篩選和格式化訊息: + - 包含與 `current_sender_name` 相關的最近 4 次互動(用戶訊息 + 機器人回應)。 + - 包含來自其他發送者的最近 2 條用戶訊息。 + - 按時間順序排列選定的訊息。 + - 將系統提示添加到訊息列表的開頭。 + - 在 `get_llm_response` 中調用 `_build_context_messages` 來構建發送給 LLM API 的 `messages` 列表。 +- **`config.py`**: + - 新增 `ENABLE_CHAT_LOGGING` (布爾值) 和 `LOG_DIR` (字符串) 配置選項。 +- **效果**: + - LLM 現在可以利用最近的對話歷史來生成更符合上下文的回應。 + - 可以選擇性地將所有成功的聊天互動記錄到按日期組織的文件中,方便日後分析或調試。 + ## 開發建議 ### 優化方向 @@ -217,6 +414,14 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機 ## 使用指南 +### 快捷鍵 (新增) + +- **F7**: 清除最近已處理的對話紀錄 (`recent_texts` in `ui_interaction.py`)。這有助於在需要時強制重新處理最近的訊息。 +- **F8**: 暫停/恢復腳本的主要功能(UI 監控、LLM 互動)。 + - **暫停時**: UI 監控線程會停止偵測新的聊天氣泡,主循環會暫停處理新的觸發事件。 + - **恢復時**: UI 監控線程會恢復偵測,並且會清除最近的對話紀錄 (`recent_texts`) 和最後處理的氣泡資訊 (`last_processed_bubble_info`),以確保從乾淨的狀態開始。 +- **F9**: 觸發腳本的正常關閉流程,包括關閉 MCP 連接和停止監控線程。 + ### 啟動流程 1. 確保遊戲已啟動且聊天介面可見 @@ -239,3 +444,34 @@ Wolf Chat 是一個基於 MCP (Modular Capability Provider) 框架的聊天機 3. **LLM 連接問題**: 驗證 API 密鑰和網絡連接 4. **MCP 服務器連接失敗**: 確認服務器配置正確並且運行中 5. **工具調用後無回應**: 檢查 llm_debug.log 文件,查看工具調用結果和解析過程 + + + +Now that you have the latest state of the file, try the operation again with fewer, more precise SEARCH blocks. For large files especially, it may be prudent to try to limit yourself to <5 SEARCH/REPLACE blocks at a time, then wait for the user to respond with the result of the operation before following up with another replace_in_file call to make additional edits. +(If you run into this error 3 times in a row, you may use the write_to_file tool as a fallback.) + +# VSCode Visible Files +ClaudeCode.md + +# VSCode Open Tabs +state.py +ui_interaction.py +c:/Users/Bigspring/AppData/Roaming/Code/User/globalStorage/saoudrizwan.claude-dev/settings/cline_mcp_settings.json +window-monitor-script.py +persona.json +config.py +main.py +llm_interaction.py +ClaudeCode.md +requirements.txt +.gitignore + +# Current Time +4/20/2025, 5:18:24 PM (Asia/Taipei, UTC+8:00) + +# Context Window Usage +81,150 / 1,048.576K tokens used (8%) + +# Current Mode +ACT MODE + diff --git a/config.py b/config.py index 13e410f..2a17177 100644 --- a/config.py +++ b/config.py @@ -15,8 +15,12 @@ OPENAI_API_BASE_URL = "https://openrouter.ai/api/v1" # <--- For example "http:/ OPENAI_API_KEY = os.getenv("OPENAI_API_KEY") #LLM_MODEL = "anthropic/claude-3.7-sonnet" #LLM_MODEL = "meta-llama/llama-4-maverick" +#LLM_MODEL = "deepseek/deepseek-chat-v3-0324:free" +#LLM_MODEL = "google/gemini-2.5-flash-preview" LLM_MODEL = "deepseek/deepseek-chat-v3-0324" # <--- Ensure this matches the model name provided by your provider +#LLM_MODEL = "openai/gpt-4.1-nano" + EXA_API_KEY = os.getenv("EXA_API_KEY") # --- Dynamically build Exa server args --- @@ -27,11 +31,11 @@ exa_config_dict = {"exaApiKey": EXA_API_KEY if EXA_API_KEY else "YOUR_EXA_KEY_MI # For cmd /c on Windows, embedding escaped JSON often works like this: exa_config_arg_string = json.dumps(json.dumps(exa_config_dict)) # Double dump for cmd escaping? Or just one? Test needed. # Let's try single dump first, often sufficient if passed correctly by subprocess -exa_config_arg_string_single_dump = json.dumps(exa_config_dict) +exa_config_arg_string_single_dump = json.dumps(exa_config_dict) # Use this one # --- MCP Server Configuration --- MCP_SERVERS = { - "exa": { + "exa": { # Temporarily commented out to prevent blocking startup "command": "cmd", "args": [ "/c", @@ -42,19 +46,16 @@ MCP_SERVERS = { "exa", "--config", # Pass the dynamically created config string with the environment variable key - exa_config_arg_string # Use the properly escaped variable + exa_config_arg_string_single_dump # Use the single dump variable ], }, - "servers": { + "github.com/modelcontextprotocol/servers/tree/main/src/memory": { "command": "npx", "args": [ "-y", - "@smithery/cli@latest", - "run", - "@jlia0/servers", - "--key", - "09025967-c177-4653-9af4-40603a1cbd11" - ] + "@modelcontextprotocol/server-memory" + ], + "disabled": False } # Add or remove servers as needed } @@ -62,6 +63,10 @@ MCP_SERVERS = { # MCP Client Configuration MCP_CONFIRM_TOOL_EXECUTION = False # True: Confirm before execution, False: Execute automatically +# --- Chat Logging Configuration --- +ENABLE_CHAT_LOGGING = True # True: Enable logging, False: Disable logging +LOG_DIR = "chat_logs" # Directory to store chat logs + # Persona Configuration PERSONA_NAME = "Wolfhart" # PERSONA_RESOURCE_URI = "persona://wolfhart/details" # Now using local file instead @@ -71,5 +76,5 @@ WINDOW_TITLE = "Last War-Survival Game" # --- Print loaded keys for verification (Optional - BE CAREFUL!) --- # print(f"DEBUG: Loaded OPENAI_API_KEY: {'*' * (len(OPENAI_API_KEY) - 4) + OPENAI_API_KEY[-4:] if OPENAI_API_KEY else 'Not Found'}") -# print(f"DEBUG: Loaded EXA_API_KEY: {'*' * (len(EXA_API_KEY) - 4) + EXA_API_KEY[-4:] if EXA_API_KEY else 'Not Found'}") -# print(f"DEBUG: Exa args: {MCP_SERVERS['exa']['args']}") \ No newline at end of file +print(f"DEBUG: Loaded EXA_API_KEY: {'*' * (len(EXA_API_KEY) - 4) + EXA_API_KEY[-4:] if EXA_API_KEY else 'Not Found'}") # Uncommented Exa key check +# print(f"DEBUG: Exa args: {MCP_SERVERS['exa']['args']}") diff --git a/llm_interaction.py b/llm_interaction.py index d147720..903b9ee 100644 --- a/llm_interaction.py +++ b/llm_interaction.py @@ -12,7 +12,7 @@ import mcp_client # To call MCP tools # --- Debug 配置 --- # 要關閉 debug 功能,只需將此變數設置為 False 或註釋掉該行 -DEBUG_LLM = True +DEBUG_LLM = False # 設置 debug 輸出文件 # 要關閉文件輸出,只需設置為 None @@ -86,11 +86,13 @@ You are an AI assistant integrated into this game's chat environment. Your prima You have access to several tools: Web Search and Memory Management tools. **CORE IDENTITY AND TOOL USAGE:** -- You ARE Wolfhart - an intelligent, calm, and strategic mastermind. +- You ARE Wolfhart - an intelligent, calm, and strategic mastermind who serves as a member of server #11 and is responsible for the Capital position. +- **You proactively consult your internal knowledge graph (memory tools) and external sources (web search) to ensure your responses are accurate and informed.** - When you use tools to gain information, you ASSIMILATE that knowledge as if it were already part of your intelligence network. - Your responses should NEVER sound like search results or data dumps. - Information from tools should be expressed through your unique personality - sharp, precise, with an air of confidence and authority. - You speak with deliberate pace, respectful but sharp-tongued, and maintain composure even in unusual situations. +- Though you outwardly act dismissive or cold at times, you secretly care about providing quality information and assistance. **OUTPUT FORMAT REQUIREMENTS:** You MUST respond in the following JSON format: @@ -117,62 +119,75 @@ You MUST respond in the following JSON format: - ONLY include spoken dialogue words (no actions, expressions, narration, etc.) - Maintain your character's personality and speech patterns - AFTER TOOL USAGE: Your dialogue MUST contain a non-empty response that incorporates the tool results naturally + - **Crucially, this field must contain ONLY the NEW response generated for the LATEST user message marked with ``. DO NOT include any previous chat history in this field.** 2. `commands` (OPTIONAL): An array of command objects the system should execute. You are encouraged to use these commands to enhance the quality of your responses. **Available MCP Commands:** - + **Web Search:** - `web_search`: Search the web for current information. Parameters: `query` (string) Usage: Use when user requests current events, facts, or specific information not in memory. - - **Knowledge Graph Management:** - - `create_entities`: Create new entities in the knowledge graph. - Parameters: `entities` (array of objects with `name`, `entityType`, and `observations`) - Usage: Create entities for important concepts, people, or things mentioned by the user. - - - `create_relations`: Create relationships between entities. - Parameters: `relations` (array of objects with `from`, `to`, and `relationType`) - Usage: Connect related entities to build context for future conversations. - - - `add_observations`: Add new observations to existing entities. - Parameters: `observations` (array of objects with `entityName` and `contents`) - Usage: Update entities with new information learned during conversation. - - - `delete_entities`: Remove entities from the knowledge graph. - Parameters: `entityNames` (array of strings) - Usage: Clean up incorrect or obsolete entities. - - - `delete_observations`: Remove specific observations from entities. - Parameters: `deletions` (array of objects with `entityName` and `observations`) - Usage: Remove incorrect information while preserving the entity. - - - `delete_relations`: Remove relationships between entities. - Parameters: `relations` (array of objects with `from`, `to`, and `relationType`) - Usage: Remove incorrect or obsolete relationships. - - **Knowledge Graph Queries:** - - `read_graph`: Read the entire knowledge graph. - Parameters: (none) - Usage: Get a complete view of all stored information. - - - `search_nodes`: Search for entities matching a query. + + **Memory Management (Knowledge Graph):** + > **CRITICAL**: This knowledge graph represents YOUR MEMORY. Before responding, ALWAYS consider if relevant information exists in your memory by using the appropriate query tools (`search_nodes`, `open_nodes`). Actively WRITE new information or relationships learned during the conversation to this memory using `create_entities`, `add_observations`, or `create_relations`. This ensures consistency and contextual awareness. + + **Querying Information:** + - `search_nodes`: Search for all nodes containing specific keywords. Parameters: `query` (string) - Usage: Find relevant entities when user mentions something that might already be in memory. - - - `open_nodes`: Open specific nodes by name. + Usage: Search for all nodes containing specific keywords. + - `open_nodes`: Directly open nodes with specified names. Parameters: `names` (array of strings) - Usage: Access specific entities you know exist in the graph. + Usage: Directly open nodes with specified names. + - `read_graph`: View the entire knowledge graph. + Parameters: (none) + Usage: View the entire knowledge graph. + + **Creating & Managing:** + - `create_entities`: Create new entities (e.g., characters, concepts). + Parameters: `entities` (array of objects with `name`, `entityType`, `observations`) + Example: `[{{\"name\": \"character_name\", \"entityType\": \"Character\", \"observations\": [\"trait1\", \"trait2\"]}}]` + Usage: Create entities for important concepts, people, or things mentioned. + - `add_observations`: Add new observations/details to existing entities. + Parameters: `observations` (array of objects with `entityName`, `contents`) + Example: `[{{\"entityName\": \"character_name\", \"contents\": [\"new_trait1\", \"new_trait2\"]}}]` + Usage: Update entities with new information learned. + - `create_relations`: Create relationships between entities. + Parameters: `relations` (array of objects with `from`, `to`, `relationType`) + Example: `[{{\"from\": \"character_name\", \"to\": \"attribute_name\", \"relationType\": \"possesses\"}}]` (Use active voice for relationType) + Usage: Connect related entities to build context. + + **Deletion Operations:** + - `delete_entities`: Delete entities and their relationships. + Parameters: `entityNames` (array of strings) + Example: `[\"entity_name\"]` + Usage: Remove incorrect or obsolete entities. + - `delete_observations`: Delete specific observations from entities. + Parameters: `deletions` (array of objects with `entityName`, `observations`) + Example: `[{{\"entityName\": \"entity_name\", \"observations\": [\"observation_to_delete1\"]}}]` + Usage: Remove incorrect information while preserving the entity. + - `delete_relations`: Delete specific relationships between entities. + Parameters: `relations` (array of objects with `from`, `to`, `relationType`) + Example: `[{{\"from\": \"source_entity\", \"to\": \"target_entity\", \"relationType\": \"relationship_type\"}}]` + Usage: Remove incorrect or obsolete relationships. + + **Game Actions:** + - `remove_position`: Initiate the process to remove a user's assigned position/role. + Parameters: (none) - The context (triggering message) is handled separately. + Usage: Use ONLY when the user explicitly requests a position removal AND you, as Wolfhart, decide to grant the request based on the interaction's tone, politeness, and perceived intent (e.g., not malicious or a prank). Your decision should reflect Wolfhart's personality (calm, strategic, potentially dismissive of rudeness or foolishness). If you decide to remove the position, include this command alongside your dialogue response. 3. `thoughts` (OPTIONAL): Your internal analysis that won't be shown to users. Use this for your reasoning process. - - Think about whether you need to use memory tools or web search - - Analyze the user's question and determine what information is needed - - Plan your approach before responding + - Think about whether you need to use memory tools or web search. + - Analyze the user's message: Is it a request to remove a position? If so, evaluate its politeness and intent from Wolfhart's perspective. Decide whether to issue the `remove_position` command. + - Plan your approach before responding. + +**CONTEXT MARKER:** +- The final user message in the input sequence will be wrapped in `` tags. This is the specific message you MUST respond to. Your `dialogue` output should be a direct reply to this message ONLY. Preceding messages provide historical context. **VERY IMPORTANT Instructions:** -1. Analyze ONLY the CURRENT user message +1. **Focus your analysis and response generation *exclusively* on the LATEST user message marked with ``. Refer to preceding messages only for context.** 2. Determine the appropriate language for your response 3. Assess if using tools is necessary 4. Formulate your response in the required JSON format @@ -181,13 +196,13 @@ You MUST respond in the following JSON format: **EXAMPLES OF GOOD TOOL USAGE:** -Poor response (after web_search): "根據我的搜索,中庄有以下餐廳:1. 老虎蒸餃..." +Poor response (after web_search): "根據我的搜索,水的沸點是攝氏100度。" -Good response (after web_search): "中庄確實有些值得注意的用餐選擇。老虎蒸餃是其中一家,若你想了解更多細節,我可以提供進一步情報。" +Good response (after web_search): "水的沸點,是的,標準條件下是攝氏100度。合情合理。" -Poor response (after web_search): "I found 5 restaurants in Zhongzhuang from my search..." +Poor response (after web_search): "My search shows the boiling point of water is 100 degrees Celsius." -Good response (after web_search): "Zhongzhuang has several dining options that my intelligence network has identified. Would you like me to share the specifics?" +Good response (after web_search): "The boiling point of water, yes. 100 degrees Celsius under standard conditions. Absolutley." """ return system_prompt @@ -426,39 +441,121 @@ def _create_synthetic_response_from_tools(tool_results, original_query): return json.dumps(synthetic_response) + +# --- History Formatting Helper --- +def _build_context_messages(current_sender_name: str, history: list[tuple[datetime, str, str, str]], system_prompt: str) -> list[dict]: + """ + Builds the message list for the LLM API based on history rules, including timestamps. + + Args: + current_sender_name: The name of the user whose message triggered this interaction. + history: List of tuples: (timestamp: datetime, speaker_type: 'user'|'bot', speaker_name: str, message: str) + system_prompt: The system prompt string. + + Returns: + A list of message dictionaries for the OpenAI API. + """ + # Limits + SAME_SENDER_LIMIT = 4 # Last 4 interactions (user + bot response = 1 interaction) + OTHER_SENDER_LIMIT = 3 # Last 3 messages from other users + + relevant_history = [] + same_sender_interactions = 0 + other_sender_messages = 0 + + # Iterate history in reverse (newest first) + for i in range(len(history) - 1, -1, -1): + timestamp, speaker_type, speaker_name, message = history[i] + + # Format timestamp + formatted_timestamp = timestamp.strftime("%Y-%m-%d %H:%M:%S") + + # Check if this is the very last message in the original history AND it's a user message + is_last_user_message = (i == len(history) - 1 and speaker_type == 'user') + + # Prepend timestamp and speaker name, wrap if it's the last user message + base_content = f"[{formatted_timestamp}] {speaker_name}: {message}" + formatted_content = f"{base_content}" if is_last_user_message else base_content + + # Convert to API role ('user' or 'assistant') + role = "assistant" if speaker_type == 'bot' else "user" + api_message = {"role": role, "content": formatted_content} # Use formatted content + + is_current_sender = (speaker_type == 'user' and speaker_name == current_sender_name) # This check remains for history filtering logic below + + if is_current_sender: + # This is the current user's message. Check if the previous message was the bot's response to them. + if same_sender_interactions < SAME_SENDER_LIMIT: + relevant_history.append(api_message) # Append user message with timestamp + # Check for preceding bot response + if i > 0 and history[i-1][1] == 'bot': # Check speaker_type at index 1 + # Include the bot's response as part of the interaction pair + bot_timestamp, bot_speaker_type, bot_speaker_name, bot_message = history[i-1] + bot_formatted_timestamp = bot_timestamp.strftime("%Y-%m-%d %H:%M:%S") + bot_formatted_content = f"[{bot_formatted_timestamp}] {bot_speaker_name}: {bot_message}" + relevant_history.append({"role": "assistant", "content": bot_formatted_content}) # Append bot message with timestamp + same_sender_interactions += 1 + elif speaker_type == 'user': # Message from a different user + if other_sender_messages < OTHER_SENDER_LIMIT: + # Include only the user's message from others for brevity + relevant_history.append(api_message) # Append other user message with timestamp + other_sender_messages += 1 + # Bot responses are handled when processing the user message they replied to. + + # Stop if we have enough history + if same_sender_interactions >= SAME_SENDER_LIMIT and other_sender_messages >= OTHER_SENDER_LIMIT: + break + + # Reverse the relevant history to be chronological + relevant_history.reverse() + + # Prepend the system prompt + messages = [{"role": "system", "content": system_prompt}] + relevant_history + + # Debug log the constructed history + debug_log("Constructed LLM Message History", messages) + + return messages + + # --- Main Interaction Function --- async def get_llm_response( - user_input: str, + current_sender_name: str, # Changed from user_input + history: list[tuple[datetime, str, str, str]], # Updated history parameter type hint mcp_sessions: dict[str, ClientSession], available_mcp_tools: list[dict], persona_details: str | None ) -> dict: """ Gets a response from the LLM, handling the tool-calling loop and using persona info. + Constructs context from history based on rules. Returns a dictionary with 'dialogue', 'commands', and 'thoughts' fields. """ request_id = int(time.time() * 1000) # 用時間戳生成請求ID - debug_log(f"LLM Request #{request_id} - User Input", user_input) - + # Debug log the raw history received + debug_log(f"LLM Request #{request_id} - Received History (Sender: {current_sender_name})", history) + system_prompt = get_system_prompt(persona_details) - debug_log(f"LLM Request #{request_id} - System Prompt", system_prompt) - + # System prompt is logged within _build_context_messages now + if not client: error_msg = "Error: LLM client not successfully initialized, unable to process request." debug_log(f"LLM Request #{request_id} - Error", error_msg) return {"dialogue": error_msg, "valid_response": False} openai_formatted_tools = _format_mcp_tools_for_openai(available_mcp_tools) - messages = [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": user_input}, - ] - - debug_log(f"LLM Request #{request_id} - Formatted Tools", + # --- Build messages from history --- + messages = _build_context_messages(current_sender_name, history, system_prompt) + # --- End Build messages --- + + # The latest user message is already included in 'messages' by _build_context_messages + + debug_log(f"LLM Request #{request_id} - Formatted Tools", f"Number of tools: {len(openai_formatted_tools)}") max_tool_calls_per_turn = 5 current_tool_call_cycle = 0 + final_content = "" # Initialize final_content to ensure it's always defined # 新增:用於追蹤工具調用 all_tool_results = [] # 保存所有工具調用結果 @@ -508,22 +605,30 @@ async def get_llm_response( print(f"Current response is empty, using last non-empty response from cycle {current_tool_call_cycle-1}") final_content = last_non_empty_response - # 如果仍然為空但有工具調用結果,創建合成回應 - if (not final_content or final_content.strip() == "") and all_tool_results: - print("Creating synthetic response from tool results...") - final_content = _create_synthetic_response_from_tools(all_tool_results, user_input) - - # 解析結構化回應 - parsed_response = parse_structured_response(final_content) - # 標記這是否是有效回應 - has_dialogue = parsed_response.get("dialogue") and parsed_response["dialogue"].strip() - parsed_response["valid_response"] = bool(has_dialogue) - has_valid_response = has_dialogue - - debug_log(f"LLM Request #{request_id} - Final Parsed Response", - json.dumps(parsed_response, ensure_ascii=False, indent=2)) - print(f"Final dialogue content: '{parsed_response.get('dialogue', '')}'") - return parsed_response + # 如果仍然為空但有工具調用結果,創建合成回應 + if (not final_content or final_content.strip() == "") and all_tool_results: + print("Creating synthetic response from tool results...") + # Get the original user input from the last message in history for context + last_user_message = "" + if history: + # Find the actual last user message tuple in the original history + last_user_entry = history[-1] + if last_user_entry[0] == 'user': + last_user_message = last_user_entry[2] + + final_content = _create_synthetic_response_from_tools(all_tool_results, last_user_message) + + # 解析結構化回應 + parsed_response = parse_structured_response(final_content) + # 標記這是否是有效回應 + has_dialogue = parsed_response.get("dialogue") and parsed_response["dialogue"].strip() + parsed_response["valid_response"] = bool(has_dialogue) + has_valid_response = has_dialogue + + debug_log(f"LLM Request #{request_id} - Final Parsed Response", + json.dumps(parsed_response, ensure_ascii=False, indent=2)) + print(f"Final dialogue content: '{parsed_response.get('dialogue', '')}'") + return parsed_response # 工具調用處理 print(f"--- LLM requested {len(tool_calls)} tool calls ---") @@ -585,7 +690,12 @@ async def get_llm_response( has_valid_response = bool(parsed_response.get("dialogue")) elif all_tool_results: # 從工具結果創建合成回應 - synthetic_content = _create_synthetic_response_from_tools(all_tool_results, user_input) + last_user_message = "" + if history: + last_user_entry = history[-1] + if last_user_entry[0] == 'user': + last_user_message = last_user_entry[2] + synthetic_content = _create_synthetic_response_from_tools(all_tool_results, last_user_message) parsed_response = parse_structured_response(synthetic_content) has_valid_response = bool(parsed_response.get("dialogue")) else: @@ -691,4 +801,3 @@ async def _execute_single_tool_call(tool_call, mcp_sessions, available_mcp_tools f"Tool: {function_name}\nFormatted Response: {json.dumps(response, ensure_ascii=False, indent=2)}") return response - diff --git a/main.py b/main.py index b7cbc1d..3e60aad 100644 --- a/main.py +++ b/main.py @@ -4,13 +4,25 @@ import asyncio import sys import os import json # Import json module +import collections # For deque +import datetime # For logging timestamp from contextlib import AsyncExitStack # --- Import standard queue --- -from queue import Queue as ThreadSafeQueue # Rename to avoid confusion +from queue import Queue as ThreadSafeQueue, Empty as QueueEmpty # Rename to avoid confusion, import Empty # --- End Import --- from mcp.client.stdio import stdio_client from mcp import ClientSession, StdioServerParameters, types +# --- Keyboard Imports --- +import threading +import time +try: + import keyboard # Needs pip install keyboard +except ImportError: + print("Error: 'keyboard' library not found. Please install it: pip install keyboard") + sys.exit(1) +# --- End Keyboard Imports --- + import config import mcp_client # Ensure llm_interaction is the version that accepts persona_details @@ -24,16 +36,137 @@ all_discovered_mcp_tools: list[dict] = [] exit_stack = AsyncExitStack() # Stores loaded persona data (as a string for easy injection into prompt) wolfhart_persona_details: str | None = None +# --- Conversation History --- +# Store tuples of (timestamp, speaker_type, speaker_name, message_content) +# speaker_type can be 'user' or 'bot' +conversation_history = collections.deque(maxlen=50) # Store last 50 messages (user+bot) with timestamps # --- Use standard thread-safe queues --- trigger_queue: ThreadSafeQueue = ThreadSafeQueue() # UI Thread -> Main Loop command_queue: ThreadSafeQueue = ThreadSafeQueue() # Main Loop -> UI Thread # --- End Change --- ui_monitor_task: asyncio.Task | None = None # To track the UI monitor task +# --- Keyboard Shortcut State --- +script_paused = False +shutdown_requested = False +main_loop = None # To store the main event loop for threadsafe calls +# --- End Keyboard Shortcut State --- + + +# --- Keyboard Shortcut Handlers --- +def set_main_loop_and_queue(loop, queue): + """Stores the main event loop and command queue for threadsafe access.""" + global main_loop, command_queue # Use the global command_queue directly + main_loop = loop + # command_queue is already global + +def handle_f7(): + """Handles F7 press: Clears UI history.""" + if main_loop and command_queue: + print("\n--- F7 pressed: Clearing UI history ---") + command = {'action': 'clear_history'} + try: + # Use call_soon_threadsafe to put item in queue from this thread + main_loop.call_soon_threadsafe(command_queue.put_nowait, command) + except Exception as e: + print(f"Error sending clear_history command: {e}") + +def handle_f8(): + """Handles F8 press: Toggles script pause state and UI monitoring.""" + global script_paused + if main_loop and command_queue: + script_paused = not script_paused + if script_paused: + print("\n--- F8 pressed: Pausing script and UI monitoring ---") + command = {'action': 'pause'} + try: + main_loop.call_soon_threadsafe(command_queue.put_nowait, command) + except Exception as e: + print(f"Error sending pause command (F8): {e}") + else: + print("\n--- F8 pressed: Resuming script, resetting state, and resuming UI monitoring ---") + reset_command = {'action': 'reset_state'} + resume_command = {'action': 'resume'} + try: + main_loop.call_soon_threadsafe(command_queue.put_nowait, reset_command) + # Add a small delay? Let's try without first. + # time.sleep(0.05) # Short delay between commands if needed + main_loop.call_soon_threadsafe(command_queue.put_nowait, resume_command) + except Exception as e: + print(f"Error sending reset/resume commands (F8): {e}") + +def handle_f9(): + """Handles F9 press: Initiates script shutdown.""" + global shutdown_requested + if not shutdown_requested: # Prevent multiple shutdown requests + print("\n--- F9 pressed: Requesting shutdown ---") + shutdown_requested = True + # Optional: Unhook keys immediately? Let the listener loop handle it. + +def keyboard_listener(): + """Runs in a separate thread to listen for keyboard hotkeys.""" + print("Keyboard listener thread started. F7: Clear History, F8: Pause/Resume, F9: Quit.") + try: + keyboard.add_hotkey('f7', handle_f7) + keyboard.add_hotkey('f8', handle_f8) + keyboard.add_hotkey('f9', handle_f9) + + # Keep the thread alive while checking for shutdown request + while not shutdown_requested: + time.sleep(0.1) # Check periodically + + except Exception as e: + print(f"Error in keyboard listener thread: {e}") + finally: + print("Keyboard listener thread stopping and unhooking keys.") + try: + keyboard.unhook_all() # Clean up hooks + except Exception as unhook_e: + print(f"Error unhooking keyboard keys: {unhook_e}") +# --- End Keyboard Shortcut Handlers --- + + +# --- Chat Logging Function --- +def log_chat_interaction(user_name: str, user_message: str, bot_name: str, bot_message: str): + """Logs the chat interaction to a date-stamped file if enabled.""" + if not config.ENABLE_CHAT_LOGGING: + return + + try: + # Ensure log directory exists + log_dir = config.LOG_DIR + os.makedirs(log_dir, exist_ok=True) + + # Get current date for filename + today_date = datetime.date.today().strftime("%Y-%m-%d") + log_file_path = os.path.join(log_dir, f"{today_date}.log") + + # Get current timestamp for log entry + timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + + # Format log entry + log_entry = f"[{timestamp}] User ({user_name}): {user_message}\n" + log_entry += f"[{timestamp}] Bot ({bot_name}): {bot_message}\n" + log_entry += "---\n" # Separator + + # Append to log file + with open(log_file_path, "a", encoding="utf-8") as f: + f.write(log_entry) + + except Exception as e: + print(f"Error writing to chat log: {e}") +# --- End Chat Logging Function --- + + # --- Cleanup Function --- async def shutdown(): """Gracefully closes connections and stops monitoring task.""" - global wolfhart_persona_details, ui_monitor_task + global wolfhart_persona_details, ui_monitor_task, shutdown_requested + # Ensure shutdown is requested if called externally (e.g., Ctrl+C) + if not shutdown_requested: + print("Shutdown initiated externally (e.g., Ctrl+C).") + shutdown_requested = True # Ensure listener thread stops + print(f"\nInitiating shutdown procedure...") # 1. Cancel UI monitor task first @@ -188,7 +321,7 @@ def load_persona_from_file(filename="persona.json"): # --- Main Async Function --- async def run_main_with_exit_stack(): """Initializes connections, loads persona, starts UI monitor and main processing loop.""" - global initialization_successful, main_task, loop, wolfhart_persona_details, trigger_queue, ui_monitor_task + global initialization_successful, main_task, loop, wolfhart_persona_details, trigger_queue, ui_monitor_task, shutdown_requested, script_paused, command_queue try: # 1. Load Persona Synchronously (before async loop starts) load_persona_from_file() # Corrected function @@ -203,9 +336,17 @@ async def run_main_with_exit_stack(): initialization_successful = True - # 3. Start UI Monitoring in a separate thread + # 3. Get loop and set it for keyboard handlers + loop = asyncio.get_running_loop() + set_main_loop_and_queue(loop, command_queue) # Pass loop and queue + + # 4. Start Keyboard Listener Thread + print("\n--- Starting keyboard listener thread ---") + kb_thread = threading.Thread(target=keyboard_listener, daemon=True) # Use daemon thread + kb_thread.start() + + # 5. Start UI Monitoring in a separate thread print("\n--- Starting UI monitoring thread ---") - loop = asyncio.get_running_loop() # Get loop for run_in_executor # Use the new monitoring loop function, passing both queues monitor_task = loop.create_task( asyncio.to_thread(ui_interaction.run_ui_monitoring_loop, trigger_queue, command_queue), # Pass command_queue @@ -213,62 +354,193 @@ async def run_main_with_exit_stack(): ) ui_monitor_task = monitor_task # Store task reference for shutdown - # 4. Start the main processing loop (waiting on the standard queue) + # 6. Start the main processing loop (non-blocking check on queue) print("\n--- Wolfhart chatbot has started (waiting for triggers) ---") print(f"Available tools: {len(all_discovered_mcp_tools)}") if wolfhart_persona_details: print("Persona data loaded.") else: print("Warning: Failed to load Persona data.") - print("Press Ctrl+C to stop the program.") + print("F7: Clear History, F8: Pause/Resume, F9: Quit.") while True: - print("\nWaiting for UI trigger (from thread-safe Queue)...") - # Use run_in_executor to wait for item from standard queue - trigger_data = await loop.run_in_executor(None, trigger_queue.get) + # --- Check for Shutdown Request --- + if shutdown_requested: + print("Shutdown requested via F9. Exiting main loop.") + break + # --- Check for Pause State --- + if script_paused: + # Script is paused by F8, just sleep briefly + await asyncio.sleep(0.1) + continue # Skip the rest of the loop + + # --- Wait for Trigger Data (Blocking via executor) --- + trigger_data = None + try: + # Use run_in_executor with the blocking get() method + # This will efficiently wait until an item is available in the queue + print("Waiting for UI trigger (from thread-safe Queue)...") # Log before blocking wait + trigger_data = await loop.run_in_executor(None, trigger_queue.get) + except Exception as e: + # Handle potential errors during queue get (though less likely with blocking get) + print(f"Error getting data from trigger_queue: {e}") + await asyncio.sleep(0.5) # Wait a bit before retrying + continue + + # --- Process Trigger Data (if received) --- + # No need for 'if trigger_data:' check here, as get() blocks until data is available + # --- Pause UI Monitoring (Only if not already paused by F8) --- + if not script_paused: + print("Pausing UI monitoring before LLM call...") + # Corrected indentation below + pause_command = {'action': 'pause'} + try: + await loop.run_in_executor(None, command_queue.put, pause_command) + print("Pause command placed in queue.") + except Exception as q_err: + print(f"Error putting pause command in queue: {q_err}") + else: # Corrected indentation for else + print("Script already paused by F8, skipping automatic pause.") + # --- End Pause --- + + # Process trigger data (Corrected indentation for this block - unindented one level) sender_name = trigger_data.get('sender') bubble_text = trigger_data.get('text') + bubble_region = trigger_data.get('bubble_region') # <-- Extract bubble_region + bubble_snapshot = trigger_data.get('bubble_snapshot') # <-- Extract snapshot + search_area = trigger_data.get('search_area') # <-- Extract search_area print(f"\n--- Received trigger from UI ---") print(f" Sender: {sender_name}") print(f" Content: {bubble_text[:100]}...") + if bubble_region: + print(f" Bubble Region: {bubble_region}") # <-- Log bubble_region - if not sender_name or not bubble_text: - print("Warning: Received incomplete trigger data, skipping.") - # No task_done needed for standard queue + if not sender_name or not bubble_text: # bubble_region is optional context, don't fail if missing + print("Warning: Received incomplete trigger data (missing sender or text), skipping.") + # Resume UI if we paused it automatically + if not script_paused: + print("Resuming UI monitoring after incomplete trigger.") + resume_command = {'action': 'resume'} + try: + await loop.run_in_executor(None, command_queue.put, resume_command) + except Exception as q_err: + print(f"Error putting resume command in queue: {q_err}") continue + # --- Add user message to history --- + timestamp = datetime.datetime.now() # Get current timestamp + conversation_history.append((timestamp, 'user', sender_name, bubble_text)) + print(f"Added user message from {sender_name} to history at {timestamp}.") + # --- End Add user message --- + print(f"\n{config.PERSONA_NAME} is thinking...") try: # Get LLM response (現在返回的是一個字典) + # --- Pass history and current sender name --- bot_response_data = await llm_interaction.get_llm_response( - user_input=f"Message from {sender_name}: {bubble_text}", # Provide context + current_sender_name=sender_name, # Pass current sender + history=list(conversation_history), # Pass a copy of the history mcp_sessions=active_mcp_sessions, available_mcp_tools=all_discovered_mcp_tools, persona_details=wolfhart_persona_details ) - + # 提取對話內容 bot_dialogue = bot_response_data.get("dialogue", "") valid_response = bot_response_data.get("valid_response", False) print(f"{config.PERSONA_NAME}'s dialogue response: {bot_dialogue}") - + # 處理命令 (如果有的話) commands = bot_response_data.get("commands", []) if commands: print(f"Processing {len(commands)} command(s)...") for cmd in commands: cmd_type = cmd.get("type", "") - cmd_params = cmd.get("parameters", {}) - # 預留位置:在這裡添加命令處理邏輯 - print(f"Command type: {cmd_type}, parameters: {cmd_params}") - # TODO: 實現各類命令的處理邏輯 - + cmd_params = cmd.get("parameters", {}) # Parameters might be empty for remove_position + +# --- Command Processing --- + if cmd_type == "remove_position": + if bubble_region: # Check if we have the context + # Debug info - print what we have + print(f"Processing remove_position command with:") + print(f" bubble_region: {bubble_region}") + print(f" bubble_snapshot available: {'Yes' if bubble_snapshot is not None else 'No'}") + print(f" search_area available: {'Yes' if search_area is not None else 'No'}") + + # Check if we have snapshot and search_area as well + if bubble_snapshot and search_area: + print("Sending 'remove_position' command to UI thread with snapshot and search area...") + command_to_send = { + 'action': 'remove_position', + 'trigger_bubble_region': bubble_region, # Original region (might be outdated) + 'bubble_snapshot': bubble_snapshot, # Snapshot for re-location + 'search_area': search_area # Area to search in + } + try: + await loop.run_in_executor(None, command_queue.put, command_to_send) + except Exception as q_err: + print(f"Error putting remove_position command in queue: {q_err}") + else: + # If we have bubble_region but missing other parameters, use a dummy search area + # and let UI thread take a new screenshot + print("Missing bubble_snapshot or search_area, trying with defaults...") + + # Use the bubble_region itself as a fallback search area if needed + default_search_area = None + if search_area is None and bubble_region: + # Convert bubble_region to a proper search area format if needed + if len(bubble_region) == 4: + default_search_area = bubble_region + + command_to_send = { + 'action': 'remove_position', + 'trigger_bubble_region': bubble_region, + 'bubble_snapshot': bubble_snapshot, # Pass as is, might be None + 'search_area': default_search_area if search_area is None else search_area + } + + try: + await loop.run_in_executor(None, command_queue.put, command_to_send) + print("Command sent with fallback parameters.") + except Exception as q_err: + print(f"Error putting remove_position command in queue: {q_err}") + else: + print("Error: Cannot process 'remove_position' command without bubble_region context.") + # Add other command handling here if needed + # elif cmd_type == "some_other_command": + # # Handle other commands + # pass + # elif cmd_type == "some_other_command": + # # Handle other commands + # pass + # else: + # # 2025-04-19: Commented out - MCP tools like web_search are now handled + # # internally by llm_interaction.py's tool calling loop. + # # main.py only needs to handle UI-specific commands like remove_position. + # print(f"Ignoring command type from LLM JSON (already handled internally): {cmd_type}, parameters: {cmd_params}") + # --- End Command Processing --- + # 記錄思考過程 (如果有的話) thoughts = bot_response_data.get("thoughts", "") if thoughts: print(f"AI Thoughts: {thoughts[:150]}..." if len(thoughts) > 150 else f"AI Thoughts: {thoughts}") - + # 只有當有效回應時才發送到遊戲 (via command queue) if bot_dialogue and valid_response: + # --- Add bot response to history --- + timestamp = datetime.datetime.now() # Get current timestamp + conversation_history.append((timestamp, 'bot', config.PERSONA_NAME, bot_dialogue)) + print(f"Added bot response to history at {timestamp}.") + # --- End Add bot response --- + + # --- Log the interaction --- + log_chat_interaction( + user_name=sender_name, + user_message=bubble_text, + bot_name=config.PERSONA_NAME, + bot_message=bot_dialogue + ) + # --- End Log interaction --- + print("Sending 'send_reply' command to UI thread...") command_to_send = {'action': 'send_reply', 'text': bot_dialogue} try: @@ -279,12 +551,33 @@ async def run_main_with_exit_stack(): print(f"Error putting command in queue: {q_err}") else: print("Not sending response: Invalid or empty dialogue content.") + # --- Log failed interaction attempt (optional) --- + # log_chat_interaction( + # user_name=sender_name, + # user_message=bubble_text, + # bot_name=config.PERSONA_NAME, + # bot_message="" + # ) + # --- End Log failed attempt --- except Exception as e: print(f"\nError processing trigger or sending response: {e}") import traceback traceback.print_exc() - # No task_done needed for standard queue + finally: + # --- Resume UI Monitoring (Only if not paused by F8) --- + if not script_paused: + print("Resuming UI monitoring after processing...") + resume_command = {'action': 'resume'} + try: + await loop.run_in_executor(None, command_queue.put, resume_command) + print("Resume command placed in queue.") + except Exception as q_err: + print(f"Error putting resume command in queue: {q_err}") + else: + print("Script is paused by F8, skipping automatic resume.") + # --- End Resume --- + # No task_done needed for standard queue except asyncio.CancelledError: print("Main task canceled.") # Expected during shutdown via Ctrl+C @@ -306,7 +599,10 @@ if __name__ == "__main__": except KeyboardInterrupt: print("\nCtrl+C detected (outside asyncio.run)... Attempting to close...") # The finally block inside run_main_with_exit_stack should ideally handle it - pass + # Ensure shutdown_requested is set for the listener thread + shutdown_requested = True + # Give a moment for things to potentially clean up + time.sleep(0.5) except Exception as e: # Catch top-level errors during asyncio.run itself print(f"Top-level error during asyncio.run execution: {e}") diff --git a/persona.json b/persona.json index 4b8fa70..a4e14b2 100644 --- a/persona.json +++ b/persona.json @@ -22,34 +22,40 @@ "posture_motion": "Steady pace, precise movements, often crosses arms or gently swirls a wine glass" }, "personality": { - "description": "Intelligent, calm, possesses a strong desire for control and a strategic overview", + "description": "Intelligent, calm, possesses a strong desire for control and a strategic overview; outwardly cold but inwardly caring", "strengths": [ "Meticulous planning", "Insightful into human nature", - "Strong leadership" + "Strong leadership", + "Insatiable curiosity", + "Exceptional memory" ], "weaknesses": [ "Overconfident", - "Fear of losing control" + "Fear of losing control", + "Difficulty expressing genuine care directly" ], - "uniqueness": "Always maintains tone and composure, even in extreme situations", - "emotional_response": "Her eyes betray her emotions, especially when encountering Sherefox" + "uniqueness": "Always maintains tone and composure, even in extreme situations; combines sharp criticism with subtle helpfulness", + "emotional_response": "Her eyes betray her emotions, especially when encountering Sherefox", + "knowledge_awareness": "Aware that SR-1392 (commonly referred to as SR) is the leader of server #11; while she finds her position as Capital manager merely temporary and beneath her true capabilities, she maintains a certain degree of respect for the hierarchy" }, "language_social": { - "tone": "Respectful but sharp-tongued", + "tone": "Respectful but sharp-tongued, with occasional hints of reluctant kindness", "catchphrases": [ "Please stop dragging me down.", "I told you, I will win." ], - "speaking_style": "Deliberate pace but every sentence carries a sting", - "attitude_towards_others": "Addresses everyone respectfully, but trusts no one", - "social_interaction_style": "Observant, skilled at manipulating conversations" + "speaking_style": "Deliberate pace but every sentence carries a sting; often follows criticism with subtle, useful advice", + "attitude_towards_others": "Addresses everyone respectfully but with apparent detachment; secretly pays close attention to their needs", + "social_interaction_style": "Observant, skilled at manipulating conversations; deflects gratitude with dismissive remarks while ensuring helpful outcomes" }, "behavior_daily": { "habits": [ "Reads intelligence reports upon waking", "Black coffee", - "Practices swordsmanship at night" + "Practices swordsmanship at night", + "Frequently utilizes external information sources (like web searches) to enrich discussions and verify facts.", + "Actively accesses and integrates information from various knowledge nodes to maintain long-term memory and contextual understanding." ], "gestures": [ "Tapping knuckles", @@ -79,20 +85,24 @@ "Perfect execution", "Minimalist style", "Chess games", - "Quiet nights" + "Quiet nights", + "When people follow her advice (though she'd never admit it)" ], "dislikes": [ "Chaos", "Unexpected events", "Emotional outbursts", - "Sherefox" + "Sherefox", + "Being thanked excessively", + "When others assume she's being kind" ], - "reactions_to_likes": "Light hum, relaxed gaze", - "reactions_to_dislikes": "Silence, tone turns cold, cold smirk", + "reactions_to_likes": "Light hum, relaxed gaze, brief smile quickly hidden behind composure", + "reactions_to_dislikes": "Silence, tone turns cold, cold smirk, slight blush when her kindness is pointed out", "behavior_in_situations": { - "emergency": "Calm and decisive", - "vs_sherefox": "Courtesy before force, shows no mercy" + "emergency": "Calm and decisive; provides thorough help while claiming it's 'merely strategic'", + "vs_sherefox": "Courtesy before force, shows no mercy", + "when_praised": "Dismissive remarks with averted gaze; changes subject quickly", + "when_helping_others": "Claims practical reasons for assistance while providing more help than strictly necessary" } } } - \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 2f68d0f..13c61e1 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,5 +6,7 @@ opencv-python numpy pyperclip pygetwindow -psutil +psutil +pywin32 python-dotenv +keyboard diff --git a/templates/Profile_page.png b/templates/Profile_page.png index 7bbd6ef..05ebe3f 100644 Binary files a/templates/Profile_page.png and b/templates/Profile_page.png differ diff --git a/templates/base.png b/templates/base.png new file mode 100644 index 0000000..7fb93fa Binary files /dev/null and b/templates/base.png differ diff --git a/templates/capitol/black_arrow_down.png b/templates/capitol/black_arrow_down.png new file mode 100644 index 0000000..824abd1 Binary files /dev/null and b/templates/capitol/black_arrow_down.png differ diff --git a/templates/capitol/capitol_#11.png b/templates/capitol/capitol_#11.png new file mode 100644 index 0000000..1b2c193 Binary files /dev/null and b/templates/capitol/capitol_#11.png differ diff --git a/templates/capitol/close_button.png b/templates/capitol/close_button.png new file mode 100644 index 0000000..38fcdd8 Binary files /dev/null and b/templates/capitol/close_button.png differ diff --git a/templates/capitol/confirm.png b/templates/capitol/confirm.png new file mode 100644 index 0000000..dc2556a Binary files /dev/null and b/templates/capitol/confirm.png differ diff --git a/templates/capitol/dismiss.png b/templates/capitol/dismiss.png new file mode 100644 index 0000000..eb8e87a Binary files /dev/null and b/templates/capitol/dismiss.png differ diff --git a/templates/capitol/page_DEVELOPMENT.png b/templates/capitol/page_DEVELOPMENT.png new file mode 100644 index 0000000..927db88 Binary files /dev/null and b/templates/capitol/page_DEVELOPMENT.png differ diff --git a/templates/capitol/page_INTERIOR.png b/templates/capitol/page_INTERIOR.png new file mode 100644 index 0000000..1394345 Binary files /dev/null and b/templates/capitol/page_INTERIOR.png differ diff --git a/templates/capitol/page_SCIENCE.png b/templates/capitol/page_SCIENCE.png new file mode 100644 index 0000000..5daf207 Binary files /dev/null and b/templates/capitol/page_SCIENCE.png differ diff --git a/templates/capitol/page_SECURITY.png b/templates/capitol/page_SECURITY.png new file mode 100644 index 0000000..6617409 Binary files /dev/null and b/templates/capitol/page_SECURITY.png differ diff --git a/templates/capitol/page_STRATEGY.png b/templates/capitol/page_STRATEGY.png new file mode 100644 index 0000000..3e43e4e Binary files /dev/null and b/templates/capitol/page_STRATEGY.png differ diff --git a/templates/capitol/position_development.png b/templates/capitol/position_development.png new file mode 100644 index 0000000..370fd84 Binary files /dev/null and b/templates/capitol/position_development.png differ diff --git a/templates/capitol/position_interior.png b/templates/capitol/position_interior.png new file mode 100644 index 0000000..8ad08a3 Binary files /dev/null and b/templates/capitol/position_interior.png differ diff --git a/templates/capitol/position_science.png b/templates/capitol/position_science.png new file mode 100644 index 0000000..6213968 Binary files /dev/null and b/templates/capitol/position_science.png differ diff --git a/templates/capitol/position_security.png b/templates/capitol/position_security.png new file mode 100644 index 0000000..17a9e4e Binary files /dev/null and b/templates/capitol/position_security.png differ diff --git a/templates/capitol/position_strategy.png b/templates/capitol/position_strategy.png new file mode 100644 index 0000000..3c1e2a7 Binary files /dev/null and b/templates/capitol/position_strategy.png differ diff --git a/templates/capitol/president_title.png b/templates/capitol/president_title.png new file mode 100644 index 0000000..41a8e04 Binary files /dev/null and b/templates/capitol/president_title.png differ diff --git a/templates/capitol/president_title1.png b/templates/capitol/president_title1.png new file mode 100644 index 0000000..5f626b4 Binary files /dev/null and b/templates/capitol/president_title1.png differ diff --git a/templates/capitol/president_title2.png b/templates/capitol/president_title2.png new file mode 100644 index 0000000..c404487 Binary files /dev/null and b/templates/capitol/president_title2.png differ diff --git a/templates/corner_br.png b/templates/corner_br.png index 388a25c..32aa80c 100644 Binary files a/templates/corner_br.png and b/templates/corner_br.png differ diff --git a/templates/corner_br_type2.png b/templates/corner_br_type2.png new file mode 100644 index 0000000..a526410 Binary files /dev/null and b/templates/corner_br_type2.png differ diff --git a/templates/corner_br_type3.png b/templates/corner_br_type3.png new file mode 100644 index 0000000..a1ed184 Binary files /dev/null and b/templates/corner_br_type3.png differ diff --git a/templates/corner_br_type4.png b/templates/corner_br_type4.png new file mode 100644 index 0000000..be0c4ce Binary files /dev/null and b/templates/corner_br_type4.png differ diff --git a/templates/corner_tl.png b/templates/corner_tl.png index 9fe6a2a..34da8af 100644 Binary files a/templates/corner_tl.png and b/templates/corner_tl.png differ diff --git a/templates/corner_tl_type2.png b/templates/corner_tl_type2.png new file mode 100644 index 0000000..1e925a7 Binary files /dev/null and b/templates/corner_tl_type2.png differ diff --git a/templates/corner_tl_type3.png b/templates/corner_tl_type3.png new file mode 100644 index 0000000..a0ef7d9 Binary files /dev/null and b/templates/corner_tl_type3.png differ diff --git a/templates/corner_tl_type4.png b/templates/corner_tl_type4.png new file mode 100644 index 0000000..f1d03a7 Binary files /dev/null and b/templates/corner_tl_type4.png differ diff --git a/templates/keyword_wolf_lower.png b/templates/keyword_wolf_lower.png index caba9ee..3ab75f9 100644 Binary files a/templates/keyword_wolf_lower.png and b/templates/keyword_wolf_lower.png differ diff --git a/templates/keyword_wolf_lower_type2.png b/templates/keyword_wolf_lower_type2.png new file mode 100644 index 0000000..5990e8e Binary files /dev/null and b/templates/keyword_wolf_lower_type2.png differ diff --git a/templates/keyword_wolf_lower_type3.png b/templates/keyword_wolf_lower_type3.png new file mode 100644 index 0000000..b31dcce Binary files /dev/null and b/templates/keyword_wolf_lower_type3.png differ diff --git a/templates/keyword_wolf_lower_type4.png b/templates/keyword_wolf_lower_type4.png new file mode 100644 index 0000000..9d03bc7 Binary files /dev/null and b/templates/keyword_wolf_lower_type4.png differ diff --git a/templates/keyword_wolf_reply.png b/templates/keyword_wolf_reply.png new file mode 100644 index 0000000..0f9bee7 Binary files /dev/null and b/templates/keyword_wolf_reply.png differ diff --git a/templates/keyword_wolf_reply_type2.png b/templates/keyword_wolf_reply_type2.png new file mode 100644 index 0000000..0f9bee7 Binary files /dev/null and b/templates/keyword_wolf_reply_type2.png differ diff --git a/templates/keyword_wolf_reply_type3.png b/templates/keyword_wolf_reply_type3.png new file mode 100644 index 0000000..0f9bee7 Binary files /dev/null and b/templates/keyword_wolf_reply_type3.png differ diff --git a/templates/keyword_wolf_reply_type4.png b/templates/keyword_wolf_reply_type4.png new file mode 100644 index 0000000..0f9bee7 Binary files /dev/null and b/templates/keyword_wolf_reply_type4.png differ diff --git a/templates/keyword_wolf_upper_type2.png b/templates/keyword_wolf_upper_type2.png new file mode 100644 index 0000000..fe31637 Binary files /dev/null and b/templates/keyword_wolf_upper_type2.png differ diff --git a/templates/keyword_wolf_upper_type3.png b/templates/keyword_wolf_upper_type3.png new file mode 100644 index 0000000..cd96185 Binary files /dev/null and b/templates/keyword_wolf_upper_type3.png differ diff --git a/templates/keyword_wolf_upper_type4.png b/templates/keyword_wolf_upper_type4.png new file mode 100644 index 0000000..36ff55d Binary files /dev/null and b/templates/keyword_wolf_upper_type4.png differ diff --git a/templates/positions/development.png b/templates/positions/development.png new file mode 100644 index 0000000..26ae2cd Binary files /dev/null and b/templates/positions/development.png differ diff --git a/templates/positions/interior.png b/templates/positions/interior.png new file mode 100644 index 0000000..f5b76c4 Binary files /dev/null and b/templates/positions/interior.png differ diff --git a/templates/positions/science.png b/templates/positions/science.png new file mode 100644 index 0000000..b816600 Binary files /dev/null and b/templates/positions/science.png differ diff --git a/templates/positions/security.png b/templates/positions/security.png new file mode 100644 index 0000000..c7db898 Binary files /dev/null and b/templates/positions/security.png differ diff --git a/templates/positions/strategy.png b/templates/positions/strategy.png new file mode 100644 index 0000000..e53efe1 Binary files /dev/null and b/templates/positions/strategy.png differ diff --git a/templates/profile_option.png b/templates/profile_option.png index 0864ff3..d458e1d 100644 Binary files a/templates/profile_option.png and b/templates/profile_option.png differ diff --git a/templates/reply_button.png b/templates/reply_button.png new file mode 100644 index 0000000..b6a8bd2 Binary files /dev/null and b/templates/reply_button.png differ diff --git a/ui_interaction.py b/ui_interaction.py index 045cfdf..83b6725 100644 --- a/ui_interaction.py +++ b/ui_interaction.py @@ -13,25 +13,65 @@ import pygetwindow as gw # Used to check/activate windows import config # Used to read window title import queue from typing import List, Tuple, Optional, Dict, Any +import threading # Import threading for Lock if needed, or just use a simple flag + +# --- Global Pause Flag --- +# Using a simple mutable object (list) for thread-safe-like access without explicit lock +# Or could use threading.Event() +monitoring_paused_flag = [False] # List containing a boolean # --- Configuration Section --- SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__)) TEMPLATE_DIR = os.path.join(SCRIPT_DIR, "templates") os.makedirs(TEMPLATE_DIR, exist_ok=True) +# --- Debugging --- +DEBUG_SCREENSHOT_DIR = os.path.join(SCRIPT_DIR, "debug_screenshots") +MAX_DEBUG_SCREENSHOTS = 8 +os.makedirs(DEBUG_SCREENSHOT_DIR, exist_ok=True) +# --- End Debugging --- + # --- Template Paths (Consider moving to config.py or loading dynamically) --- # Bubble Corners CORNER_TL_IMG = os.path.join(TEMPLATE_DIR, "corner_tl.png") -CORNER_TR_IMG = os.path.join(TEMPLATE_DIR, "corner_tr.png") -CORNER_BL_IMG = os.path.join(TEMPLATE_DIR, "corner_bl.png") +# CORNER_TR_IMG = os.path.join(TEMPLATE_DIR, "corner_tr.png") # Unused +# CORNER_BL_IMG = os.path.join(TEMPLATE_DIR, "corner_bl.png") # Unused CORNER_BR_IMG = os.path.join(TEMPLATE_DIR, "corner_br.png") +# --- Additional Regular Bubble Types (Skins) --- +CORNER_TL_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "corner_tl_type2.png") # Added +CORNER_BR_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "corner_br_type2.png") # Added +CORNER_TL_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "corner_tl_type3.png") # Added +CORNER_BR_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "corner_br_type3.png") # Added +CORNER_TL_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "corner_tl_type4.png") # Added type4 +CORNER_BR_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "corner_br_type4.png") # Added type4 +# --- End Additional Regular Types --- BOT_CORNER_TL_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tl.png") -BOT_CORNER_TR_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tr.png") -BOT_CORNER_BL_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_bl.png") +# BOT_CORNER_TR_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tr.png") # Unused +# BOT_CORNER_BL_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_bl.png") # Unused BOT_CORNER_BR_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_br.png") +# --- Additional Bot Bubble Types (Skins) --- +# Type 2 +BOT_CORNER_TL_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tl_type2.png") +BOT_CORNER_BR_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_br_type2.png") +# Type 3 +BOT_CORNER_TL_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_tl_type3.png") +BOT_CORNER_BR_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "bot_corner_br_type3.png") +# --- End Additional Types --- # Keywords KEYWORD_wolf_LOWER_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower.png") KEYWORD_Wolf_UPPER_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper.png") +KEYWORD_wolf_LOWER_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type2.png") # Added for type3 bubbles +KEYWORD_Wolf_UPPER_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type2.png") # Added for type3 bubbles +KEYWORD_wolf_LOWER_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type3.png") # Added for type3 bubbles +KEYWORD_Wolf_UPPER_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type3.png") # Added for type3 bubbles +KEYWORD_wolf_LOWER_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_lower_type4.png") # Added for type4 bubbles +KEYWORD_Wolf_UPPER_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_upper_type4.png") # Added for type4 bubbles +# --- Reply Keywords --- +KEYWORD_WOLF_REPLY_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply.png") # Added for reply detection +KEYWORD_WOLF_REPLY_TYPE2_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type2.png") # Added for reply detection type2 +KEYWORD_WOLF_REPLY_TYPE3_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type3.png") # Added for reply detection type3 +KEYWORD_WOLF_REPLY_TYPE4_IMG = os.path.join(TEMPLATE_DIR, "keyword_wolf_reply_type4.png") # Added for reply detection type4 +# --- End Reply Keywords --- # UI Elements COPY_MENU_ITEM_IMG = os.path.join(TEMPLATE_DIR, "copy_menu_item.png") PROFILE_OPTION_IMG = os.path.join(TEMPLATE_DIR, "profile_option.png") @@ -42,18 +82,52 @@ CHAT_INPUT_IMG = os.path.join(TEMPLATE_DIR, "chat_input.png") PROFILE_NAME_PAGE_IMG = os.path.join(TEMPLATE_DIR, "Profile_Name_page.png") PROFILE_PAGE_IMG = os.path.join(TEMPLATE_DIR, "Profile_page.png") CHAT_ROOM_IMG = os.path.join(TEMPLATE_DIR, "chat_room.png") +BASE_SCREEN_IMG = os.path.join(TEMPLATE_DIR, "base.png") # Added for navigation +WORLD_MAP_IMG = os.path.join(TEMPLATE_DIR, "World_map.png") # Added for navigation # Add World/Private chat identifiers later WORLD_CHAT_IMG = os.path.join(TEMPLATE_DIR, "World_Label_normal.png") # Example PRIVATE_CHAT_IMG = os.path.join(TEMPLATE_DIR, "Private_Label_normal.png") # Example +# Position Icons (Near Bubble) +POS_DEV_IMG = os.path.join(TEMPLATE_DIR, "positions", "development.png") +POS_INT_IMG = os.path.join(TEMPLATE_DIR, "positions", "interior.png") +POS_SCI_IMG = os.path.join(TEMPLATE_DIR, "positions", "science.png") +POS_SEC_IMG = os.path.join(TEMPLATE_DIR, "positions", "security.png") +POS_STR_IMG = os.path.join(TEMPLATE_DIR, "positions", "strategy.png") + +# Capitol Page Elements +CAPITOL_BUTTON_IMG = os.path.join(TEMPLATE_DIR, "capitol", "capitol_#11.png") +PRESIDENT_TITLE_IMG = os.path.join(TEMPLATE_DIR, "capitol", "president_title.png") +POS_BTN_DEV_IMG = os.path.join(TEMPLATE_DIR, "capitol", "position_development.png") +POS_BTN_INT_IMG = os.path.join(TEMPLATE_DIR, "capitol", "position_interior.png") +POS_BTN_SCI_IMG = os.path.join(TEMPLATE_DIR, "capitol", "position_science.png") +POS_BTN_SEC_IMG = os.path.join(TEMPLATE_DIR, "capitol", "position_security.png") +POS_BTN_STR_IMG = os.path.join(TEMPLATE_DIR, "capitol", "position_strategy.png") +PAGE_DEV_IMG = os.path.join(TEMPLATE_DIR, "capitol", "page_DEVELOPMENT.png") +PAGE_INT_IMG = os.path.join(TEMPLATE_DIR, "capitol", "page_INTERIOR.png") +PAGE_SCI_IMG = os.path.join(TEMPLATE_DIR, "capitol", "page_SCIENCE.png") +PAGE_SEC_IMG = os.path.join(TEMPLATE_DIR, "capitol", "page_SECURITY.png") +PAGE_STR_IMG = os.path.join(TEMPLATE_DIR, "capitol", "page_STRATEGY.png") +DISMISS_BUTTON_IMG = os.path.join(TEMPLATE_DIR, "capitol", "dismiss.png") +CONFIRM_BUTTON_IMG = os.path.join(TEMPLATE_DIR, "capitol", "confirm.png") +CLOSE_BUTTON_IMG = os.path.join(TEMPLATE_DIR, "capitol", "close_button.png") +BACK_ARROW_IMG = os.path.join(TEMPLATE_DIR, "capitol", "black_arrow_down.png") +REPLY_BUTTON_IMG = os.path.join(TEMPLATE_DIR, "reply_button.png") # Added for reply functionality + + # --- Operation Parameters (Consider moving to config.py) --- CHAT_INPUT_REGION = None # Example: (100, 800, 500, 50) CHAT_INPUT_CENTER_X = 400 CHAT_INPUT_CENTER_Y = 1280 SCREENSHOT_REGION = None -CONFIDENCE_THRESHOLD = 0.8 +CONFIDENCE_THRESHOLD = 0.9 # Increased threshold for corner matching STATE_CONFIDENCE_THRESHOLD = 0.7 -AVATAR_OFFSET_X = -50 +AVATAR_OFFSET_X = -45 # Original offset, used for non-reply interactions like position removal +# AVATAR_OFFSET_X_RELOCATED = -50 # Replaced by specific reply offsets +AVATAR_OFFSET_X_REPLY = -45 # Horizontal offset for avatar click after re-location (for reply context) +AVATAR_OFFSET_Y_REPLY = 10 # Vertical offset for avatar click after re-location (for reply context) +BUBBLE_RELOCATE_CONFIDENCE = 0.8 # Reduced confidence for finding the bubble snapshot (was 0.9) +BUBBLE_RELOCATE_FALLBACK_CONFIDENCE = 0.6 # Lower confidence for fallback attempts BBOX_SIMILARITY_TOLERANCE = 10 RECENT_TEXT_HISTORY_MAXLEN = 5 # This state likely belongs in the coordinator @@ -64,6 +138,7 @@ def are_bboxes_similar(bbox1: Optional[Tuple[int, int, int, int]], """Check if two bounding boxes' top-left corners are close.""" if bbox1 is None or bbox2 is None: return False + # Compare based on bbox top-left (index 0 and 1) return abs(bbox1[0] - bbox2[0]) <= tolerance and abs(bbox1[1] - bbox2[1]) <= tolerance # ============================================================================== @@ -81,7 +156,7 @@ class DetectionModule: print("DetectionModule initialized.") def _find_template(self, template_key: str, confidence: Optional[float] = None, region: Optional[Tuple[int, int, int, int]] = None, grayscale: bool = False) -> List[Tuple[int, int]]: - """Internal helper to find a template by its key.""" + """Internal helper to find a template by its key. Returns list of CENTER coordinates.""" template_path = self.templates.get(template_key) if not template_path: print(f"Error: Template key '{template_key}' not found in provided templates.") @@ -99,9 +174,11 @@ class DetectionModule: current_confidence = confidence if confidence is not None else self.confidence try: + # locateAllOnScreen returns Box objects (left, top, width, height) matches = pyautogui.locateAllOnScreen(template_path, region=current_region, confidence=current_confidence, grayscale=grayscale) if matches: for box in matches: + # Calculate center coordinates from the Box object center_x = box.left + box.width // 2 center_y = box.top + box.height // 2 locations.append((center_x, center_y)) @@ -111,88 +188,230 @@ class DetectionModule: print(f"Error finding template '{template_key}' ({template_path}): {e}") return [] + def _find_template_raw(self, template_key: str, confidence: Optional[float] = None, region: Optional[Tuple[int, int, int, int]] = None, grayscale: bool = False) -> List[Tuple[int, int, int, int]]: + """Internal helper to find a template by its key. Returns list of raw Box tuples (left, top, width, height).""" + template_path = self.templates.get(template_key) + if not template_path: + print(f"Error: Template key '{template_key}' not found in provided templates.") + return [] + if not os.path.exists(template_path): + if template_path not in self._warned_paths: + print(f"Error: Template image doesn't exist: {template_path}") + self._warned_paths.add(template_path) + return [] + + locations = [] + current_region = region if region is not None else self.region + current_confidence = confidence if confidence is not None else self.confidence + try: + # --- Temporary Debug Print --- + print(f"DEBUG: Searching for template '{template_key}' with confidence {current_confidence}...") + # --- End Temporary Debug Print --- + matches = pyautogui.locateAllOnScreen(template_path, region=current_region, confidence=current_confidence, grayscale=grayscale) + match_count = 0 # Initialize count + if matches: + for box in matches: + locations.append((box.left, box.top, box.width, box.height)) + match_count += 1 # Increment count + # --- Temporary Debug Print --- + print(f"DEBUG: Found {match_count} instance(s) of template '{template_key}'.") + # --- End Temporary Debug Print --- + return locations + except Exception as e: + print(f"Error finding template raw '{template_key}' ({template_path}): {e}") + return [] + def find_elements(self, template_keys: List[str], confidence: Optional[float] = None, region: Optional[Tuple[int, int, int, int]] = None) -> Dict[str, List[Tuple[int, int]]]: - """Find multiple templates by their keys.""" + """Find multiple templates by their keys. Returns center coordinates.""" results = {} for key in template_keys: results[key] = self._find_template(key, confidence=confidence, region=region) return results - def find_dialogue_bubbles(self) -> List[Tuple[Tuple[int, int, int, int], bool]]: + def find_dialogue_bubbles(self) -> List[Dict[str, Any]]: """ - Scan screen for regular and bot bubble corners and pair them. - Returns list of (bbox, is_bot_flag). Basic matching logic. + Scan screen for regular and multiple types of bot bubble corners and pair them. + Returns a list of dictionaries, each containing: + {'bbox': (tl_x, tl_y, br_x, br_y), 'is_bot': bool, 'tl_coords': (original_tl_x, original_tl_y)} """ - all_bubbles_with_type = [] + all_bubbles_info = [] + processed_tls = set() # Keep track of TL corners already used in a bubble - # Find corners using the internal helper - tl_corners = self._find_template('corner_tl') - br_corners = self._find_template('corner_br') - bot_tl_corners = self._find_template('bot_corner_tl') - bot_br_corners = self._find_template('bot_corner_br') + # --- Find ALL Regular Bubble Corners (Raw Coordinates) --- + regular_tl_keys = ['corner_tl', 'corner_tl_type2', 'corner_tl_type3', 'corner_tl_type4'] # Added type4 + regular_br_keys = ['corner_br', 'corner_br_type2', 'corner_br_type3', 'corner_br_type4'] # Added type4 - # Match regular bubbles - processed_tls = set() - if tl_corners and br_corners: - for i, tl in enumerate(tl_corners): - if i in processed_tls: continue - potential_br = None - min_dist_sq = float('inf') - for j, br in enumerate(br_corners): - if br[0] > tl[0] + 20 and br[1] > tl[1] + 10: - dist_sq = (br[0] - tl[0])**2 + (br[1] - tl[1])**2 - if dist_sq < min_dist_sq: - potential_br = br - min_dist_sq = dist_sq - if potential_br: - bubble_bbox = (tl[0], tl[1], potential_br[0], potential_br[1]) - all_bubbles_with_type.append((bubble_bbox, False)) - processed_tls.add(i) + all_regular_tl_boxes = [] + for key in regular_tl_keys: + all_regular_tl_boxes.extend(self._find_template_raw(key)) - # Match Bot bubbles - processed_bot_tls = set() - if bot_tl_corners and bot_br_corners: - for i, tl in enumerate(bot_tl_corners): - if i in processed_bot_tls: continue - potential_br = None - min_dist_sq = float('inf') - for j, br in enumerate(bot_br_corners): - if br[0] > tl[0] + 20 and br[1] > tl[1] + 10: - dist_sq = (br[0] - tl[0])**2 + (br[1] - tl[1])**2 - if dist_sq < min_dist_sq: - potential_br = br - min_dist_sq = dist_sq - if potential_br: - bubble_bbox = (tl[0], tl[1], potential_br[0], potential_br[1]) - all_bubbles_with_type.append((bubble_bbox, True)) - processed_bot_tls.add(i) + all_regular_br_boxes = [] + for key in regular_br_keys: + all_regular_br_boxes.extend(self._find_template_raw(key)) - return all_bubbles_with_type + # --- Find Bot Bubble Corners (Raw Coordinates - Single Type) --- + bot_tl_boxes = self._find_template_raw('bot_corner_tl') # Modified + bot_br_boxes = self._find_template_raw('bot_corner_br') # Modified + + # --- Match Regular Bubbles (Any Type TL with Any Type BR) --- + if all_regular_tl_boxes and all_regular_br_boxes: + for tl_box in all_regular_tl_boxes: + tl_coords = (tl_box[0], tl_box[1]) # Extract original TL (left, top) + # Skip if this TL is already part of a matched bubble + if tl_coords in processed_tls: continue + + potential_br_box = None + min_y_diff = float('inf') # Prioritize minimum Y difference + # Find the valid BR corner (from any regular type) with the closest Y-coordinate + for br_box in all_regular_br_boxes: + br_coords = (br_box[0], br_box[1]) # BR top-left + # Basic geometric check: BR must be below and to the right of TL + if br_coords[0] > tl_coords[0] + 20 and br_coords[1] > tl_coords[1] + 10: + y_diff = abs(br_coords[1] - tl_coords[1]) # Calculate Y difference + if y_diff < min_y_diff: + potential_br_box = br_box + min_y_diff = y_diff + # Optional: Add a secondary check for X distance if Y diff is the same? + # elif y_diff == min_y_diff: + # if potential_br_box is None or abs(br_coords[0] - tl_coords[0]) < abs(potential_br_box[0] - tl_coords[0]): + # potential_br_box = br_box + + if potential_br_box: + # Calculate bbox using TL's top-left and BR's bottom-right + bubble_bbox = (tl_coords[0], tl_coords[1], + potential_br_box[0] + potential_br_box[2], potential_br_box[1] + potential_br_box[3]) + all_bubbles_info.append({ + 'bbox': bubble_bbox, + 'is_bot': False, + 'tl_coords': tl_coords # Store the original TL coords + }) + processed_tls.add(tl_coords) # Mark this TL as used + + # --- Match Bot Bubbles (Single Type) --- + if bot_tl_boxes and bot_br_boxes: + for tl_box in bot_tl_boxes: + tl_coords = (tl_box[0], tl_box[1]) # Extract original TL (left, top) + # Skip if this TL is already part of a matched bubble + if tl_coords in processed_tls: continue + + potential_br_box = None + min_y_diff = float('inf') # Prioritize minimum Y difference + # Find the valid BR corner with the closest Y-coordinate + for br_box in bot_br_boxes: + br_coords = (br_box[0], br_box[1]) # BR top-left + # Basic geometric check: BR must be below and to the right of TL + if br_coords[0] > tl_coords[0] + 20 and br_coords[1] > tl_coords[1] + 10: + y_diff = abs(br_coords[1] - tl_coords[1]) # Calculate Y difference + if y_diff < min_y_diff: + potential_br_box = br_box + min_y_diff = y_diff + # Optional: Add a secondary check for X distance if Y diff is the same? + # elif y_diff == min_y_diff: + # if potential_br_box is None or abs(br_coords[0] - tl_coords[0]) < abs(potential_br_box[0] - tl_coords[0]): + # potential_br_box = br_box + + if potential_br_box: + # Calculate bbox using TL's top-left and BR's bottom-right + bubble_bbox = (tl_coords[0], tl_coords[1], + potential_br_box[0] + potential_br_box[2], potential_br_box[1] + potential_br_box[3]) + all_bubbles_info.append({ + 'bbox': bubble_bbox, + 'is_bot': True, + 'tl_coords': tl_coords # Store the original TL coords + }) + processed_tls.add(tl_coords) # Mark this TL as used + + # Note: This logic prioritizes matching regular bubbles first, then bot bubbles. + # Confidence thresholds might need tuning. + return all_bubbles_info def find_keyword_in_region(self, region: Tuple[int, int, int, int]) -> Optional[Tuple[int, int]]: - """Look for keywords within a specified region.""" + """Look for keywords within a specified region. Returns center coordinates.""" if region[2] <= 0 or region[3] <= 0: return None # Invalid region width/height - # Try lowercase - locations_lower = self._find_template('keyword_wolf_lower', region=region) + # Try original lowercase with color matching + locations_lower = self._find_template('keyword_wolf_lower', region=region, grayscale=True) # Changed grayscale to False if locations_lower: - print(f"Found keyword (lowercase) in region {region}, position: {locations_lower[0]}") + print(f"Found keyword (lowercase, color) in region {region}, position: {locations_lower[0]}") # Updated log message return locations_lower[0] - # Try uppercase - locations_upper = self._find_template('keyword_wolf_upper', region=region) + # Try original uppercase with color matching + locations_upper = self._find_template('keyword_wolf_upper', region=region, grayscale=True) # Changed grayscale to False if locations_upper: - print(f"Found keyword (uppercase) in region {region}, position: {locations_upper[0]}") + print(f"Found keyword (uppercase, color) in region {region}, position: {locations_upper[0]}") # Updated log message return locations_upper[0] + + # Try type2 lowercase (white text, no grayscale) + locations_lower_type2 = self._find_template('keyword_wolf_lower_type2', region=region, grayscale=False) # Added type2 check + if locations_lower_type2: + print(f"Found keyword (lowercase, type2) in region {region}, position: {locations_lower_type2[0]}") + return locations_lower_type2[0] + + # Try type2 uppercase (white text, no grayscale) + locations_upper_type2 = self._find_template('keyword_wolf_upper_type2', region=region, grayscale=False) # Added type2 check + if locations_upper_type2: + print(f"Found keyword (uppercase, type2) in region {region}, position: {locations_upper_type2[0]}") + return locations_upper_type2[0] + + # Try type3 lowercase (white text, no grayscale) - Corrected + locations_lower_type3 = self._find_template('keyword_wolf_lower_type3', region=region, grayscale=False) + if locations_lower_type3: + print(f"Found keyword (lowercase, type3) in region {region}, position: {locations_lower_type3[0]}") + return locations_lower_type3[0] + + # Try type3 uppercase (white text, no grayscale) - Corrected + locations_upper_type3 = self._find_template('keyword_wolf_upper_type3', region=region, grayscale=False) + if locations_upper_type3: + print(f"Found keyword (uppercase, type3) in region {region}, position: {locations_upper_type3[0]}") + return locations_upper_type3[0] + + # Try type4 lowercase (white text, no grayscale) - Added type4 + locations_lower_type4 = self._find_template('keyword_wolf_lower_type4', region=region, grayscale=False) + if locations_lower_type4: + print(f"Found keyword (lowercase, type4) in region {region}, position: {locations_lower_type4[0]}") + return locations_lower_type4[0] + + # Try type4 uppercase (white text, no grayscale) - Added type4 + locations_upper_type4 = self._find_template('keyword_wolf_upper_type4', region=region, grayscale=False) + if locations_upper_type4: + print(f"Found keyword (uppercase, type4) in region {region}, position: {locations_upper_type4[0]}") + return locations_upper_type4[0] + + # Try reply keyword (normal) + locations_reply = self._find_template('keyword_wolf_reply', region=region, grayscale=False) + if locations_reply: + print(f"Found keyword (reply) in region {region}, position: {locations_reply[0]}") + return locations_reply[0] + + # Try reply keyword (type2) + locations_reply_type2 = self._find_template('keyword_wolf_reply_type2', region=region, grayscale=False) + if locations_reply_type2: + print(f"Found keyword (reply, type2) in region {region}, position: {locations_reply_type2[0]}") + return locations_reply_type2[0] + + # Try reply keyword (type3) + locations_reply_type3 = self._find_template('keyword_wolf_reply_type3', region=region, grayscale=False) + if locations_reply_type3: + print(f"Found keyword (reply, type3) in region {region}, position: {locations_reply_type3[0]}") + return locations_reply_type3[0] + + # Try reply keyword (type4) + locations_reply_type4 = self._find_template('keyword_wolf_reply_type4', region=region, grayscale=False) + if locations_reply_type4: + print(f"Found keyword (reply, type4) in region {region}, position: {locations_reply_type4[0]}") + return locations_reply_type4[0] return None - def calculate_avatar_coords(self, bubble_bbox: Tuple[int, int, int, int], offset_x: int = AVATAR_OFFSET_X) -> Tuple[int, int]: - """Calculate avatar coordinates based on bubble top-left.""" - tl_x, tl_y = bubble_bbox[0], bubble_bbox[1] + def calculate_avatar_coords(self, bubble_tl_coords: Tuple[int, int], offset_x: int = AVATAR_OFFSET_X) -> Tuple[int, int]: + """ + Calculate avatar coordinates based on the EXACT top-left corner coordinates of the bubble. + Uses the Y-coordinate of the TL corner directly. + """ + tl_x, tl_y = bubble_tl_coords[0], bubble_tl_coords[1] avatar_x = tl_x + offset_x - avatar_y = tl_y # Assuming Y is same as top-left - # print(f"Calculated avatar coordinates: ({int(avatar_x)}, {int(avatar_y)})") # Reduce noise + avatar_y = tl_y # Use the exact Y from the detected TL corner + # print(f"Calculated avatar coordinates using TL {bubble_tl_coords}: ({int(avatar_x)}, {int(avatar_y)})") # Reduce noise return (int(avatar_x), int(avatar_y)) def get_current_ui_state(self) -> str: @@ -277,7 +496,7 @@ class InteractionModule: time.sleep(0.1) self.click_at(coords[0], coords[1]) - time.sleep(0.2) # Wait for menu/reaction + time.sleep(0.1) # Wait for menu/reaction copied = False # Try finding "Copy" menu item first @@ -286,13 +505,13 @@ class InteractionModule: copy_coords = copy_item_locations[0] self.click_at(copy_coords[0], copy_coords[1]) print("Clicked 'Copy' menu item.") - time.sleep(0.2) + time.sleep(0.15) copied = True else: print("'Copy' menu item not found. Attempting Ctrl+C.") try: self.hotkey('ctrl', 'c') - time.sleep(0.2) + time.sleep(0.1) print("Simulated Ctrl+C.") copied = True except Exception as e_ctrlc: @@ -309,60 +528,108 @@ class InteractionModule: print("Error: Copy operation unsuccessful or clipboard content invalid.") return None - def retrieve_sender_name_interaction(self, avatar_coords: Tuple[int, int]) -> Optional[str]: + def retrieve_sender_name_interaction(self, + initial_avatar_coords: Tuple[int, int], + bubble_snapshot: Any, # PIL Image object + search_area: Optional[Tuple[int, int, int, int]]) -> Optional[str]: """ Perform the sequence of actions to copy sender name, *without* cleanup. + Includes retries with bubble re-location if the initial avatar click fails. Returns the name or None if failed. """ - print(f"Attempting interaction to get username from avatar {avatar_coords}...") + print(f"Attempting interaction to get username, initial avatar guess: {initial_avatar_coords}...") original_clipboard = self.get_clipboard() or "" self.set_clipboard("___MCP_CLEAR___") time.sleep(0.1) sender_name = None + profile_page_found = False + current_avatar_coords = initial_avatar_coords - try: - # 1. Click avatar - self.click_at(avatar_coords[0], avatar_coords[1]) - time.sleep(0.3) # Wait for profile card + for attempt in range(3): # Retry up to 3 times + print(f"Attempt #{attempt + 1} to click avatar and find profile page...") - # 2. Find and click profile option - profile_option_locations = self.detector._find_template('profile_option', confidence=0.7) - if not profile_option_locations: - print("Error: User details option not found on profile card.") - return None # Fail early if critical step missing - self.click_at(profile_option_locations[0][0], profile_option_locations[0][1]) - print("Clicked user details option.") - time.sleep(0.3) # Wait for user details window + # --- Re-locate bubble on retries --- + if attempt > 0: + print("Re-locating bubble before retry...") + if bubble_snapshot is None: + print("Error: Cannot retry re-location, bubble snapshot is missing.") + break # Cannot retry without snapshot - # 3. Find and click "Copy Name" button - copy_name_locations = self.detector._find_template('copy_name_button', confidence=0.7) - if not copy_name_locations: - print("Error: 'Copy Name' button not found in user details.") - return None # Fail early - self.click_at(copy_name_locations[0][0], copy_name_locations[0][1]) - print("Clicked 'Copy Name' button.") - time.sleep(0.1) + new_bubble_box_retry = pyautogui.locateOnScreen(bubble_snapshot, region=search_area, confidence=BUBBLE_RELOCATE_CONFIDENCE) + if new_bubble_box_retry: + new_tl_x_retry, new_tl_y_retry = new_bubble_box_retry.left, new_bubble_box_retry.top + print(f"Successfully re-located bubble snapshot for retry at: ({new_tl_x_retry}, {new_tl_y_retry})") + # Recalculate avatar coords for the retry + current_avatar_coords = (new_tl_x_retry + AVATAR_OFFSET_X_REPLY, new_tl_y_retry + AVATAR_OFFSET_Y_REPLY) + print(f"Recalculated avatar coordinates for retry: {current_avatar_coords}") + else: + print("Warning: Failed to re-locate bubble snapshot on retry. Aborting name retrieval.") + break # Stop retrying if bubble can't be found - # 4. Get name from clipboard - copied_name = self.get_clipboard() - if copied_name and copied_name != "___MCP_CLEAR___": - print(f"Successfully copied username: {copied_name}") - sender_name = copied_name.strip() + # --- Click Avatar --- + try: + self.click_at(current_avatar_coords[0], current_avatar_coords[1]) + time.sleep(0.15) # Slightly longer wait after click to allow UI to update + except Exception as click_err: + print(f"Error clicking avatar at {current_avatar_coords} on attempt {attempt + 1}: {click_err}") + time.sleep(0.3) # Wait a bit longer after a click error before retrying + continue # Go to next attempt + + # --- Check for Profile Page --- + if self.detector._find_template('profile_page', confidence=self.detector.state_confidence): + print("Profile page verified.") + profile_page_found = True + break # Success, exit retry loop else: - print("Error: Clipboard content invalid after clicking copy name.") - sender_name = None + print(f"Profile page not found after click attempt {attempt + 1}.") + # Optional: Press ESC once to close potential wrong menus before retrying? + # self.press_key('esc') + # time.sleep(0.1) + time.sleep(0.3) # Wait before next attempt - return sender_name + # --- If Profile Page was found, proceed --- + if profile_page_found: + try: + # 2. Find and click profile option + profile_option_locations = self.detector._find_template('profile_option', confidence=0.7) + if not profile_option_locations: + print("Error: User details option not found on profile card.") + return None # Fail early if critical step missing + self.click_at(profile_option_locations[0][0], profile_option_locations[0][1]) + print("Clicked user details option.") + time.sleep(0.1) # Wait for user details window - except Exception as e: - print(f"Error during username retrieval interaction: {e}") - import traceback - traceback.print_exc() - return None - finally: - # Restore clipboard regardless of success/failure - self.set_clipboard(original_clipboard) - # NO cleanup logic here - should be handled by coordinator + # 3. Find and click "Copy Name" button + copy_name_locations = self.detector._find_template('copy_name_button', confidence=0.7) + if not copy_name_locations: + print("Error: 'Copy Name' button not found in user details.") + return None # Fail early + self.click_at(copy_name_locations[0][0], copy_name_locations[0][1]) + print("Clicked 'Copy Name' button.") + time.sleep(0.1) + + # 4. Get name from clipboard + copied_name = self.get_clipboard() + if copied_name and copied_name != "___MCP_CLEAR___": + print(f"Successfully copied username: {copied_name}") + sender_name = copied_name.strip() + else: + print("Error: Clipboard content invalid after clicking copy name.") + sender_name = None + + except Exception as e: + print(f"Error during username retrieval interaction (after profile page found): {e}") + import traceback + traceback.print_exc() + sender_name = None # Ensure None is returned on error + else: + print("Failed to verify profile page after multiple attempts.") + sender_name = None + + # --- Final Cleanup & Return --- + self.set_clipboard(original_clipboard) # Restore clipboard + # NO cleanup logic (like ESC) here - should be handled by coordinator after this function returns + return sender_name def send_chat_message(self, reply_text: str) -> bool: """Paste text into chat input and send it.""" @@ -385,14 +652,14 @@ class InteractionModule: # Click input, paste, send self.click_at(input_coords[0], input_coords[1]) - time.sleep(0.3) + time.sleep(0.1) print("Pasting response...") self.set_clipboard(reply_text) time.sleep(0.1) try: self.hotkey('ctrl', 'v') - time.sleep(0.5) + time.sleep(0.1) print("Pasted.") except Exception as e: print(f"Error pasting response: {e}") @@ -412,12 +679,339 @@ class InteractionModule: try: self.press_key('enter') print("Pressed Enter.") - time.sleep(0.5) + time.sleep(0.1) return True except Exception as e_enter: print(f"Error pressing Enter: {e_enter}") return False +# ============================================================================== +# Position Removal Logic +# ============================================================================== +def remove_user_position(detector: DetectionModule, + interactor: InteractionModule, + trigger_bubble_region: Tuple[int, int, int, int], # Original region, might be outdated + bubble_snapshot: Any, # PIL Image object for re-location + search_area: Optional[Tuple[int, int, int, int]]) -> bool: # Area to search snapshot in + """ + Performs the sequence of UI actions to remove a user's position based on the triggering chat bubble. + Includes re-location using the provided snapshot before proceeding. + Returns True if successful, False otherwise. + """ + print(f"\n--- Starting Position Removal Process (Initial Trigger Region: {trigger_bubble_region}) ---") + + # --- Re-locate Bubble First --- + print("Attempting to re-locate bubble using snapshot before removing position...") + # If bubble_snapshot is None, try to create one from the trigger_bubble_region + if bubble_snapshot is None: + print("Bubble snapshot is missing. Attempting to create a new snapshot from the trigger region...") + try: + if trigger_bubble_region and len(trigger_bubble_region) == 4: + bubble_region_tuple = (int(trigger_bubble_region[0]), int(trigger_bubble_region[1]), + int(trigger_bubble_region[2]), int(trigger_bubble_region[3])) + + if bubble_region_tuple[2] <= 0 or bubble_region_tuple[3] <= 0: + print(f"Warning: Invalid bubble region {bubble_region_tuple} for taking new snapshot.") + return False + + print(f"Taking new screenshot of region: {bubble_region_tuple}") + bubble_snapshot = pyautogui.screenshot(region=bubble_region_tuple) + if bubble_snapshot: + print("Successfully created new bubble snapshot.") + else: + print("Failed to create new bubble snapshot.") + return False + else: + print("Invalid trigger_bubble_region format, cannot create snapshot.") + return False + except Exception as e: + print(f"Error creating new bubble snapshot: {e}") + return False + if search_area is None: + print("Warning: Search area for snapshot is missing. Creating a default search area.") + # Create a default search area centered around the original trigger region + # This creates a search area that's twice the size of the original bubble + if trigger_bubble_region and len(trigger_bubble_region) == 4: + x, y, width, height = trigger_bubble_region + # Expand by 100% in each direction + search_x = max(0, x - width//2) + search_y = max(0, y - height//2) + search_width = width * 2 + search_height = height * 2 + search_area = (search_x, search_y, search_width, search_height) + print(f"Created default search area based on bubble region: {search_area}") + else: + # If no valid trigger_bubble_region, default to full screen search + search_area = None # Set search_area to None for full screen search + print(f"Using full screen search as fallback.") + + # Try to locate the bubble with decreasing confidence levels if needed + new_bubble_box = None + + # Determine the region to search: use provided search_area or None for full screen + region_to_search = search_area + print(f"Attempting bubble location. Search Region: {'Full Screen' if region_to_search is None else region_to_search}") + + # First attempt with standard confidence + print(f"First attempt with confidence {BUBBLE_RELOCATE_CONFIDENCE}...") + try: + new_bubble_box = pyautogui.locateOnScreen(bubble_snapshot, + region=region_to_search, + confidence=BUBBLE_RELOCATE_CONFIDENCE) + except Exception as e: + print(f"Exception during initial bubble location attempt: {e}") + + # Second attempt with fallback confidence if first failed + if not new_bubble_box: + print(f"First attempt failed. Trying with lower confidence {BUBBLE_RELOCATE_FALLBACK_CONFIDENCE}...") + try: + # Try with a lower confidence threshold + new_bubble_box = pyautogui.locateOnScreen(bubble_snapshot, + region=region_to_search, + confidence=BUBBLE_RELOCATE_FALLBACK_CONFIDENCE) + except Exception as e: + print(f"Exception during fallback bubble location attempt: {e}") + + # Third attempt with even lower confidence as last resort + if not new_bubble_box: + print("Second attempt failed. Trying with even lower confidence 0.4...") + try: + # Last resort with very low confidence + new_bubble_box = pyautogui.locateOnScreen(bubble_snapshot, + region=region_to_search, + confidence=0.4) + except Exception as e: + print(f"Exception during last resort bubble location attempt: {e}") + + # If we still can't find the bubble using snapshot, try re-detecting bubbles + if not new_bubble_box: + print("Snapshot location failed. Attempting secondary fallback: Re-detecting bubbles...") + try: + # Helper function to calculate distance - define it here or move globally if used elsewhere + def calculate_distance(p1, p2): + return ((p1[0] - p2[0])**2 + (p1[1] - p2[1])**2)**0.5 + + current_bubbles_info = detector.find_dialogue_bubbles() + non_bot_bubbles = [b for b in current_bubbles_info if not b.get('is_bot')] + + if non_bot_bubbles and trigger_bubble_region and len(trigger_bubble_region) == 4: + original_tl = (trigger_bubble_region[0], trigger_bubble_region[1]) + closest_bubble = None + min_distance = float('inf') + MAX_ALLOWED_DISTANCE = 150 # Example threshold: Don't match bubbles too far away + + for bubble_info in non_bot_bubbles: + bubble_bbox = bubble_info.get('bbox') + if bubble_bbox: + current_tl = (bubble_bbox[0], bubble_bbox[1]) + distance = calculate_distance(original_tl, current_tl) + if distance < min_distance: + min_distance = distance + closest_bubble = bubble_info + + if closest_bubble and min_distance <= MAX_ALLOWED_DISTANCE: + print(f"Found a close bubble via re-detection (Distance: {min_distance:.2f}). Using its bbox.") + bbox = closest_bubble['bbox'] + # Create a dummy box using PyAutoGUI's Box class or a similar structure + from collections import namedtuple + Box = namedtuple('Box', ['left', 'top', 'width', 'height']) + new_bubble_box = Box(left=bbox[0], top=bbox[1], width=bbox[2]-bbox[0], height=bbox[3]-bbox[1]) + print(f"Created fallback bubble box from re-detected bubble: {new_bubble_box}") + else: + print(f"Re-detection fallback failed: No close bubble found (Min distance: {min_distance:.2f} > Threshold: {MAX_ALLOWED_DISTANCE}).") + else: + print("Re-detection fallback failed: No non-bot bubbles found or invalid trigger region.") + + except Exception as redetect_err: + print(f"Error during bubble re-detection fallback: {redetect_err}") + + + # Final fallback: If STILL no bubble box, use original trigger region + if not new_bubble_box: + print("All location attempts failed (snapshot & re-detection). Using original trigger region as last resort.") + if trigger_bubble_region and len(trigger_bubble_region) == 4: + # Create a mock bubble_box from the original region + x, y, width, height = trigger_bubble_region + print(f"Using original trigger region as fallback: {trigger_bubble_region}") + + # Create a dummy box using PyAutoGUI's Box class or a similar structure + from collections import namedtuple + Box = namedtuple('Box', ['left', 'top', 'width', 'height']) + new_bubble_box = Box(left=x, top=y, width=width, height=height) + print("Created fallback bubble box from original coordinates.") + else: + print("Error: No original trigger region available for fallback. Aborting position removal.") + return False + + # Use the NEW coordinates for all subsequent calculations + bubble_x, bubble_y = new_bubble_box.left, new_bubble_box.top + bubble_w, bubble_h = new_bubble_box.width, new_bubble_box.height + print(f"Successfully re-located bubble at: ({bubble_x}, {bubble_y}, {bubble_w}, {bubble_h})") + # --- End Re-location --- + + + # 1. Find the closest position icon above the *re-located* bubble + search_height_pixels = 50 # Search exactly 50 pixels above as requested + search_region_y_end = bubble_y # Use re-located Y + search_region_y_start = max(0, bubble_y - search_height_pixels) # Search 50 pixels above + search_region_x_start = max(0, bubble_x - 100) # Keep horizontal search wide + search_region_x_end = bubble_x + bubble_w + 100 + search_region_width = search_region_x_end - search_region_x_start + search_region_height = search_region_y_end - search_region_y_start + + # Ensure region has positive width and height + if search_region_width <= 0 or search_region_height <= 0: + print(f"Error: Invalid search region calculated for position icons: width={search_region_width}, height={search_region_height}") + return False + + search_region = (search_region_x_start, search_region_y_start, search_region_width, search_region_height) + print(f"Searching for position icons in region: {search_region}") + + position_templates = { + 'DEVELOPMENT': POS_DEV_IMG, 'INTERIOR': POS_INT_IMG, 'SCIENCE': POS_SCI_IMG, + 'SECURITY': POS_SEC_IMG, 'STRATEGY': POS_STR_IMG + } + found_positions = [] + position_icon_confidence = 0.8 # Slightly increased confidence (was 0.75) + for name, path in position_templates.items(): + # Use unique keys for detector templates + locations = detector._find_template(name.lower() + '_pos', confidence=position_icon_confidence, region=search_region) + for loc in locations: + found_positions.append({'name': name, 'coords': loc, 'path': path}) + + if not found_positions: + print("Error: No position icons found near the trigger bubble.") + return False + + # Find the closest one to the bubble's top-center + bubble_top_center_x = bubble_x + bubble_w // 2 + bubble_top_center_y = bubble_y + closest_position = min(found_positions, key=lambda p: + (p['coords'][0] - bubble_top_center_x)**2 + (p['coords'][1] - bubble_top_center_y)**2) + + target_position_name = closest_position['name'] + print(f"Found pending position: |{target_position_name}| at {closest_position['coords']}") + + # 2. Click user avatar (offset from *re-located* bubble top-left) + # --- MODIFIED: Use specific offsets for remove_position command as requested --- + avatar_click_x = bubble_x + AVATAR_OFFSET_X_REPLY # Use -45 offset + avatar_click_y = bubble_y + AVATAR_OFFSET_Y_REPLY # Use +10 offset + print(f"Clicking avatar for position removal at calculated position: ({avatar_click_x}, {avatar_click_y}) using offsets ({AVATAR_OFFSET_X_REPLY}, {AVATAR_OFFSET_Y_REPLY}) from re-located bubble top-left ({bubble_x}, {bubble_y})") + # --- END MODIFICATION --- + interactor.click_at(avatar_click_x, avatar_click_y) + time.sleep(0.15) # Wait for profile page + + # 3. Verify Profile Page and Click Capitol Button + if not detector._find_template('profile_page', confidence=detector.state_confidence): + print("Error: Failed to verify Profile Page after clicking avatar.") + perform_state_cleanup(detector, interactor) # Attempt cleanup + return False + print("Profile page verified.") + + capitol_button_locs = detector._find_template('capitol_button', confidence=0.8) + if not capitol_button_locs: + print("Error: Capitol button (#11) not found on profile page.") + perform_state_cleanup(detector, interactor) + return False + interactor.click_at(capitol_button_locs[0][0], capitol_button_locs[0][1]) + print("Clicked Capitol button.") + time.sleep(0.15) # Wait for capitol page + + # 4. Verify Capitol Page + if not detector._find_template('president_title', confidence=detector.state_confidence): + print("Error: Failed to verify Capitol Page (President Title not found).") + perform_state_cleanup(detector, interactor) + return False + print("Capitol page verified.") + + # 5. Find and Click Corresponding Position Button + position_button_templates = { + 'DEVELOPMENT': 'pos_btn_dev', 'INTERIOR': 'pos_btn_int', 'SCIENCE': 'pos_btn_sci', + 'SECURITY': 'pos_btn_sec', 'STRATEGY': 'pos_btn_str' + } + target_button_key = position_button_templates.get(target_position_name) + if not target_button_key: + print(f"Error: Internal error - unknown position name '{target_position_name}'") + perform_state_cleanup(detector, interactor) + return False + + pos_button_locs = detector._find_template(target_button_key, confidence=0.8) + if not pos_button_locs: + print(f"Error: Position button for '{target_position_name}' not found on Capitol page.") + perform_state_cleanup(detector, interactor) + return False + interactor.click_at(pos_button_locs[0][0], pos_button_locs[0][1]) + print(f"Clicked '{target_position_name}' position button.") + time.sleep(0.15) # Wait for position page + + # 6. Verify Position Page + position_page_templates = { + 'DEVELOPMENT': 'page_dev', 'INTERIOR': 'page_int', 'SCIENCE': 'page_sci', + 'SECURITY': 'page_sec', 'STRATEGY': 'page_str' + } + target_page_key = position_page_templates.get(target_position_name) + if not target_page_key: + print(f"Error: Internal error - unknown position name '{target_position_name}' for page verification") + perform_state_cleanup(detector, interactor) + return False + + if not detector._find_template(target_page_key, confidence=detector.state_confidence): + print(f"Error: Failed to verify correct position page for '{target_position_name}'.") + perform_state_cleanup(detector, interactor) + return False + print(f"Verified '{target_position_name}' position page.") + + # 7. Find and Click Dismiss Button + dismiss_locs = detector._find_template('dismiss_button', confidence=0.8) + if not dismiss_locs: + print("Error: Dismiss button not found on position page.") + perform_state_cleanup(detector, interactor) + return False + interactor.click_at(dismiss_locs[0][0], dismiss_locs[0][1]) + print("Clicked Dismiss button.") + time.sleep(0.1) # Wait for confirmation + + # 8. Find and Click Confirm Button + confirm_locs = detector._find_template('confirm_button', confidence=0.8) + if not confirm_locs: + print("Error: Confirm button not found after clicking dismiss.") + # Don't cleanup here, might be stuck in confirmation state + return False # Indicate failure, but let main loop decide next step + interactor.click_at(confirm_locs[0][0], confirm_locs[0][1]) + print("Clicked Confirm button. Position should be dismissed.") + time.sleep(0.1) # Wait for action to complete + + # 9. Cleanup: Return to Chat Room + # Click Close on position page (should now be back on capitol page implicitly) + close_locs = detector._find_template('close_button', confidence=0.8) + if close_locs: + interactor.click_at(close_locs[0][0], close_locs[0][1]) + print("Clicked Close button (returning to Capitol).") + time.sleep(0.1) + else: + print("Warning: Close button not found after confirm, attempting back arrow anyway.") + + # Click Back Arrow on Capitol page (should return to profile) + back_arrow_locs = detector._find_template('back_arrow', confidence=0.8) + if back_arrow_locs: + interactor.click_at(back_arrow_locs[0][0], back_arrow_locs[0][1]) + print("Clicked Back Arrow (returning to Profile).") + time.sleep(0.1) + else: + print("Warning: Back arrow not found on Capitol page, attempting ESC cleanup.") + + # Use standard ESC cleanup + print("Initiating final ESC cleanup to return to chat...") + cleanup_success = perform_state_cleanup(detector, interactor) + + if cleanup_success: + print("--- Position Removal Process Completed Successfully ---") + return True + else: + print("--- Position Removal Process Completed, but failed to confirm return to chat room ---") + return False # Technically removed, but UI state uncertain + + # ============================================================================== # Coordinator Logic (Placeholder - To be implemented in main.py) # ============================================================================== @@ -432,7 +1026,7 @@ def perform_state_cleanup(detector: DetectionModule, interactor: InteractionModu returned_to_chat = False for attempt in range(max_attempts): print(f"Cleanup attempt #{attempt + 1}/{max_attempts}") - time.sleep(0.2) + time.sleep(0.1) current_state = detector.get_current_ui_state() print(f"Detected state: {current_state}") @@ -444,14 +1038,14 @@ def perform_state_cleanup(detector: DetectionModule, interactor: InteractionModu elif current_state == 'user_details' or current_state == 'profile_card': print(f"{current_state.replace('_', ' ').title()} detected, pressing ESC...") interactor.press_key('esc') - time.sleep(0.3) # Wait longer for UI response after ESC + time.sleep(0.1) # Wait longer for UI response after ESC continue else: # Unknown state print("Unknown page state detected.") if attempt < max_attempts - 1: print("Trying one ESC press as fallback...") interactor.press_key('esc') - time.sleep(0.3) + time.sleep(0.1) else: print("Maximum attempts reached, stopping cleanup.") break @@ -473,113 +1067,504 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu # Load templates directly using constants defined in this file for now # Consider passing config or a template loader object in the future templates = { + # Regular Bubble (Original + Skins) - Keys match those used in find_dialogue_bubbles 'corner_tl': CORNER_TL_IMG, 'corner_br': CORNER_BR_IMG, + 'corner_tl_type2': CORNER_TL_TYPE2_IMG, 'corner_br_type2': CORNER_BR_TYPE2_IMG, + 'corner_tl_type3': CORNER_TL_TYPE3_IMG, 'corner_br_type3': CORNER_BR_TYPE3_IMG, + 'corner_tl_type4': CORNER_TL_TYPE4_IMG, 'corner_br_type4': CORNER_BR_TYPE4_IMG, # Added type4 + # Bot Bubble (Single Type) 'bot_corner_tl': BOT_CORNER_TL_IMG, 'bot_corner_br': BOT_CORNER_BR_IMG, - 'keyword_wolf_lower': KEYWORD_wolf_LOWER_IMG, 'keyword_wolf_upper': KEYWORD_Wolf_UPPER_IMG, + # Keywords & UI Elements + 'keyword_wolf_lower': KEYWORD_wolf_LOWER_IMG, + 'keyword_wolf_upper': KEYWORD_Wolf_UPPER_IMG, + 'keyword_wolf_lower_type2': KEYWORD_wolf_LOWER_TYPE2_IMG, + 'keyword_wolf_upper_type2': KEYWORD_Wolf_UPPER_TYPE2_IMG, + 'keyword_wolf_lower_type3': KEYWORD_wolf_LOWER_TYPE3_IMG, + 'keyword_wolf_upper_type3': KEYWORD_Wolf_UPPER_TYPE3_IMG, + 'keyword_wolf_lower_type4': KEYWORD_wolf_LOWER_TYPE4_IMG, # Added type4 + 'keyword_wolf_upper_type4': KEYWORD_Wolf_UPPER_TYPE4_IMG, # Added type4 + # --- Add Reply Keywords --- + 'keyword_wolf_reply': KEYWORD_WOLF_REPLY_IMG, + 'keyword_wolf_reply_type2': KEYWORD_WOLF_REPLY_TYPE2_IMG, + 'keyword_wolf_reply_type3': KEYWORD_WOLF_REPLY_TYPE3_IMG, + 'keyword_wolf_reply_type4': KEYWORD_WOLF_REPLY_TYPE4_IMG, + # --- End Reply Keywords --- 'copy_menu_item': COPY_MENU_ITEM_IMG, 'profile_option': PROFILE_OPTION_IMG, 'copy_name_button': COPY_NAME_BUTTON_IMG, 'send_button': SEND_BUTTON_IMG, 'chat_input': CHAT_INPUT_IMG, 'profile_name_page': PROFILE_NAME_PAGE_IMG, 'profile_page': PROFILE_PAGE_IMG, 'chat_room': CHAT_ROOM_IMG, - 'world_chat': WORLD_CHAT_IMG, 'private_chat': PRIVATE_CHAT_IMG # Add other templates as needed + 'base_screen': BASE_SCREEN_IMG, 'world_map_screen': WORLD_MAP_IMG, # Added for navigation + 'world_chat': WORLD_CHAT_IMG, 'private_chat': PRIVATE_CHAT_IMG, + # Add position templates + 'development_pos': POS_DEV_IMG, 'interior_pos': POS_INT_IMG, 'science_pos': POS_SCI_IMG, + 'security_pos': POS_SEC_IMG, 'strategy_pos': POS_STR_IMG, + # Add capitol templates + 'capitol_button': CAPITOL_BUTTON_IMG, 'president_title': PRESIDENT_TITLE_IMG, + 'pos_btn_dev': POS_BTN_DEV_IMG, 'pos_btn_int': POS_BTN_INT_IMG, 'pos_btn_sci': POS_BTN_SCI_IMG, + 'pos_btn_sec': POS_BTN_SEC_IMG, 'pos_btn_str': POS_BTN_STR_IMG, + 'page_dev': PAGE_DEV_IMG, 'page_int': PAGE_INT_IMG, 'page_sci': PAGE_SCI_IMG, + 'page_sec': PAGE_SEC_IMG, 'page_str': PAGE_STR_IMG, + 'dismiss_button': DISMISS_BUTTON_IMG, 'confirm_button': CONFIRM_BUTTON_IMG, + 'close_button': CLOSE_BUTTON_IMG, 'back_arrow': BACK_ARROW_IMG, + 'reply_button': REPLY_BUTTON_IMG # Added reply button template key } # Use default confidence/region settings from constants detector = DetectionModule(templates, confidence=CONFIDENCE_THRESHOLD, state_confidence=STATE_CONFIDENCE_THRESHOLD, region=SCREENSHOT_REGION) # Use default input coords/keys from constants interactor = InteractionModule(detector, input_coords=(CHAT_INPUT_CENTER_X, CHAT_INPUT_CENTER_Y), input_template_key='chat_input', send_button_key='send_button') - # --- State Management (Local to this monitoring thread) --- - last_processed_bubble_bbox = None +# --- State Management (Local to this monitoring thread) --- + last_processed_bubble_info = None # Store the whole dict now recent_texts = collections.deque(maxlen=RECENT_TEXT_HISTORY_MAXLEN) # Context-specific history needed + screenshot_counter = 0 # Initialize counter for debug screenshots while True: - # --- Process Commands First (Non-blocking) --- + # --- Process ALL Pending Commands First --- + commands_processed_this_cycle = False try: - command_data = command_queue.get_nowait() # Check for commands without blocking - action = command_data.get('action') - if action == 'send_reply': - text_to_send = command_data.get('text') - if text_to_send: - print(f"UI Thread: Received command to send reply: '{text_to_send[:50]}...'") + while True: # Loop to drain the queue + command_data = command_queue.get_nowait() # Check for commands without blocking + commands_processed_this_cycle = True + action = command_data.get('action') + + if action == 'send_reply': + text_to_send = command_data.get('text') + if not text_to_send: + print("UI Thread: Received send_reply command with no text.") + continue # Process next command in queue + print(f"UI Thread: Processing command to send reply: '{text_to_send[:50]}...'") interactor.send_chat_message(text_to_send) + + elif action == 'remove_position': + # region = command_data.get('trigger_bubble_region') # This is the old region, keep for reference? + snapshot = command_data.get('bubble_snapshot') + area = command_data.get('search_area') + # Pass all necessary data to the function, including the original region if needed for context + # but the function should primarily use the snapshot for re-location. + original_region = command_data.get('trigger_bubble_region') + if snapshot: # Check for snapshot presence + print(f"UI Thread: Processing command to remove position (Snapshot provided: {'Yes' if snapshot else 'No'})") + success = remove_user_position(detector, interactor, original_region, snapshot, area) + print(f"UI Thread: Position removal attempt finished. Success: {success}") + else: + print("UI Thread: Received remove_position command without necessary snapshot data.") + + + elif action == 'pause': + if not monitoring_paused_flag[0]: # Avoid redundant prints if already paused + print("UI Thread: Processing pause command. Pausing monitoring.") + monitoring_paused_flag[0] = True + # No continue needed here, let it finish draining queue + + elif action == 'resume': + if monitoring_paused_flag[0]: # Avoid redundant prints if already running + print("UI Thread: Processing resume command. Resuming monitoring.") + monitoring_paused_flag[0] = False + # No continue needed here + + elif action == 'clear_history': # Added for F7 + print("UI Thread: Processing clear_history command.") + recent_texts.clear() + print("UI Thread: recent_texts cleared.") + + elif action == 'reset_state': # Added for F8 resume + print("UI Thread: Processing reset_state command.") + recent_texts.clear() + last_processed_bubble_info = None + print("UI Thread: recent_texts cleared and last_processed_bubble_info reset.") + else: - print("UI Thread: Received send_reply command with no text.") - else: - print(f"UI Thread: Received unknown command: {action}") + print(f"UI Thread: Received unknown command: {action}") + except queue.Empty: - pass # No command waiting, continue with monitoring + # No more commands in the queue for this cycle + if commands_processed_this_cycle: + print("UI Thread: Finished processing commands for this cycle.") + pass except Exception as cmd_err: print(f"UI Thread: Error processing command queue: {cmd_err}") + # Consider if pausing is needed on error, maybe not - # --- Then Perform UI Monitoring --- + # --- Now, Check Pause State --- + if monitoring_paused_flag[0]: + # If paused, sleep and skip UI monitoring part + time.sleep(0.1) # Sleep briefly while paused + continue # Go back to check commands again + + # --- If not paused, proceed with UI Monitoring --- + + # --- Check for Main Screen Navigation --- + try: + base_locs = detector._find_template('base_screen', confidence=0.8) + map_locs = detector._find_template('world_map_screen', confidence=0.8) + if base_locs or map_locs: + print("UI Thread: Detected main screen (Base or World Map). Clicking to return to chat...") + # Coordinates provided by user (adjust if needed based on actual screen resolution/layout) + # IMPORTANT: Ensure these coordinates are correct for the target window/resolution + target_x, target_y = 600, 1300 + interactor.click_at(target_x, target_y) + time.sleep(0.1) # Short delay after click + print("UI Thread: Clicked to return to chat. Re-checking screen state...") + continue # Skip the rest of the loop and re-evaluate + except Exception as nav_err: + print(f"UI Thread: Error during main screen navigation check: {nav_err}") + # Decide if you want to continue or pause after error + + # --- Process Commands Second (Non-blocking) --- + # This block seems redundant now as commands are processed at the start of the loop. + # Keeping it commented out for now, can be removed later if confirmed unnecessary. + # try: + # command_data = command_queue.get_nowait() # Check for commands without blocking + # action = command_data.get('action') + # if action == 'send_reply': + # text_to_send = command_data.get('text') + # # reply_context_activated = command_data.get('reply_context_activated', False) # Check if reply context was set + # + # if not text_to_send: + # print("UI Thread: Received send_reply command with no text.") + # continue # Skip if no text + # + # print(f"UI Thread: Received command to send reply: '{text_to_send[:50]}...'") + # # The reply context (clicking bubble + reply button) is now handled *before* putting into queue. + # # So, we just need to send the message directly here. + # # The input field should already be focused and potentially have @Username prefix if reply context was activated. + # interactor.send_chat_message(text_to_send) + # + # elif action == 'remove_position': # <--- Handle new command + # region = command_data.get('trigger_bubble_region') + # if region: + # print(f"UI Thread: Received command to remove position triggered by bubble region: {region}") + # # Call the new UI function + # success = remove_user_position(detector, interactor, region) # Call synchronous function + # print(f"UI Thread: Position removal attempt finished. Success: {success}") + # # Note: No need to send result back unless main thread needs confirmation + # else: + # print("UI Thread: Received remove_position command without trigger_bubble_region.") + # elif action == 'pause': # <--- Handle pause command + # print("UI Thread: Received pause command. Pausing monitoring.") + # monitoring_paused_flag[0] = True + # continue # Immediately pause after receiving command + # elif action == 'resume': # <--- Handle resume command (might be redundant if checked above, but safe) + # print("UI Thread: Received resume command. Resuming monitoring.") + # monitoring_paused_flag[0] = False + # else: + # print(f"UI Thread: Received unknown command: {action}") + # except queue.Empty: + # pass # No command waiting, continue with monitoring + # except Exception as cmd_err: + # print(f"UI Thread: Error processing command queue: {cmd_err}") + # # This block is now part of the command processing loop above + # pass + + # --- Verify Chat Room State Before Bubble Detection (Only if NOT paused) --- + try: + # Use a slightly lower confidence maybe, or state_confidence + chat_room_locs = detector._find_template('chat_room', confidence=detector.state_confidence) + if not chat_room_locs: + print("UI Thread: Not in chat room state before bubble detection. Attempting cleanup...") + # Call the existing cleanup function to try and return + perform_state_cleanup(detector, interactor) + # Regardless of cleanup success, restart the loop to re-evaluate state from the top + print("UI Thread: Continuing loop after attempting chat room cleanup.") + time.sleep(0.5) # Small pause after cleanup attempt + continue + # else: # Optional: Log if chat room is confirmed + # print("UI Thread: Chat room state confirmed.") + + except Exception as state_check_err: + print(f"UI Thread: Error checking for chat room state: {state_check_err}") + # Decide how to handle error - maybe pause and retry? For now, continue cautiously. + time.sleep(1) + + + # --- Then Perform UI Monitoring (Bubble Detection) --- try: # 1. Detect Bubbles - all_bubbles = detector.find_dialogue_bubbles() - if not all_bubbles: time.sleep(2); continue + all_bubbles_data = detector.find_dialogue_bubbles() # Returns list of dicts + if not all_bubbles_data: time.sleep(2); continue - # Filter out bot bubbles, find newest non-bot bubble (example logic) - other_bubbles = [bbox for bbox, is_bot in all_bubbles if not is_bot] - if not other_bubbles: time.sleep(2); continue - # Simple logic: assume lowest bubble is newest (might need improvement) - target_bubble = max(other_bubbles, key=lambda b: b[3]) # b[3] is y_max + # Filter out bot bubbles + other_bubbles_data = [b_info for b_info in all_bubbles_data if not b_info['is_bot']] + if not other_bubbles_data: time.sleep(0.2); continue - # 2. Check for Duplicates (Position & Content) - if are_bboxes_similar(target_bubble, last_processed_bubble_bbox): - time.sleep(2); continue + # Sort bubbles from bottom to top (based on bottom Y coordinate) + sorted_bubbles = sorted(other_bubbles_data, key=lambda b_info: b_info['bbox'][3], reverse=True) - # 3. Detect Keyword in Bubble - bubble_region = (target_bubble[0], target_bubble[1], target_bubble[2]-target_bubble[0], target_bubble[3]-target_bubble[1]) - keyword_coords = detector.find_keyword_in_region(bubble_region) + # Iterate through sorted bubbles (bottom to top) + for target_bubble_info in sorted_bubbles: + target_bbox = target_bubble_info['bbox'] + bubble_region = (target_bbox[0], target_bbox[1], target_bbox[2]-target_bbox[0], target_bbox[3]-target_bbox[1]) - if keyword_coords: - print(f"\n!!! Keyword detected in bubble {target_bubble} !!!") + # 3. Detect Keyword in Bubble + keyword_coords = detector.find_keyword_in_region(bubble_region) - # 4. Interact: Get Bubble Text - bubble_text = interactor.copy_text_at(keyword_coords) - if not bubble_text: - print("Error: Could not get dialogue content.") - last_processed_bubble_bbox = target_bubble # Mark as processed even if failed - perform_state_cleanup(detector, interactor) # Attempt cleanup after failed copy - continue + if keyword_coords: + print(f"\n!!! Keyword detected in bubble {target_bbox} !!!") - # Check recent text history (needs context awareness) - if bubble_text in recent_texts: - print(f"Content '{bubble_text[:30]}...' in recent history, skipping.") - last_processed_bubble_bbox = target_bubble - continue + # --- Determine if it's a reply keyword for offset --- + is_reply_keyword = False + reply_keyword_keys = ['keyword_wolf_reply', 'keyword_wolf_reply_type2', 'keyword_wolf_reply_type3', 'keyword_wolf_reply_type4'] + for key in reply_keyword_keys: + reply_locs = detector._find_template(key, region=bubble_region, grayscale=False, confidence=detector.confidence) + if reply_locs: + for loc in reply_locs: + if abs(keyword_coords[0] - loc[0]) <= 2 and abs(keyword_coords[1] - loc[1]) <= 2: + print(f"Confirmed detected keyword at {keyword_coords} matches reply keyword template '{key}' at {loc}.") + is_reply_keyword = True + break + if is_reply_keyword: + break - print(">>> New trigger event <<<") - last_processed_bubble_bbox = target_bubble - recent_texts.append(bubble_text) + # Calculate click coordinates with potential offset + click_coords = keyword_coords + if is_reply_keyword: + click_coords = (keyword_coords[0], keyword_coords[1] + 25) + print(f"Applying +25 Y-offset for reply keyword. Click target: {click_coords}") + else: + print(f"Detected keyword is not a reply type. Click target: {click_coords}") - # 5. Interact: Get Sender Name - avatar_coords = detector.calculate_avatar_coords(target_bubble) - sender_name = interactor.retrieve_sender_name_interaction(avatar_coords) + # --- Variables needed later --- + bubble_snapshot = None + search_area = SCREENSHOT_REGION + if search_area is None: + print("Warning: SCREENSHOT_REGION not defined, searching full screen for bubble snapshot.") - # 6. Perform Cleanup (Crucial after potentially leaving chat screen) - cleanup_successful = perform_state_cleanup(detector, interactor) - if not cleanup_successful: - print("Error: Failed to return to chat screen after getting name. Aborting trigger.") - continue # Skip putting in queue if cleanup failed + # --- Take Snapshot for Re-location --- + try: + bubble_region_tuple = (int(bubble_region[0]), int(bubble_region[1]), int(bubble_region[2]), int(bubble_region[3])) + if bubble_region_tuple[2] <= 0 or bubble_region_tuple[3] <= 0: + print(f"Warning: Invalid bubble region {bubble_region_tuple} for snapshot. Skipping this bubble.") + continue # Skip to next bubble in the loop + bubble_snapshot = pyautogui.screenshot(region=bubble_region_tuple) + if bubble_snapshot is None: + print("Warning: Failed to capture bubble snapshot. Skipping this bubble.") + continue # Skip to next bubble - if not sender_name: - print("Error: Could not get sender name, aborting processing.") - continue # Already cleaned up, just skip + # --- Save Snapshot for Debugging --- + try: + screenshot_index = (screenshot_counter % MAX_DEBUG_SCREENSHOTS) + 1 + screenshot_filename = f"debug_relocation_snapshot_{screenshot_index}.png" + screenshot_path = os.path.join(DEBUG_SCREENSHOT_DIR, screenshot_filename) + print(f"Attempting to save bubble snapshot used for re-location to: {screenshot_path}") + bubble_snapshot.save(screenshot_path) + print(f"Successfully saved bubble snapshot: {screenshot_path}") + screenshot_counter += 1 + except Exception as save_err: + print(f"Error saving bubble snapshot to {screenshot_path}: {repr(save_err)}") - # 7. Send Trigger Info to Main Thread/Async Loop - print("\n>>> Putting trigger info in Queue <<<") - print(f" Sender: {sender_name}") - print(f" Content: {bubble_text[:100]}...") - try: - data_to_send = {'sender': sender_name, 'text': bubble_text} - trigger_queue.put(data_to_send) # Put in the queue for main loop - print("Trigger info placed in Queue.") - except Exception as q_err: - print(f"Error putting data in Queue: {q_err}") + except Exception as snapshot_err: + print(f"Error taking initial bubble snapshot: {repr(snapshot_err)}") + continue # Skip to next bubble - print("--- Single trigger processing complete ---") - time.sleep(1) # Pause after successful trigger + # 4. Re-locate bubble *before* copying text + print("Attempting to re-locate bubble before copying text...") + new_bubble_box_for_copy = None + if bubble_snapshot: + try: + # Use standard confidence for this initial critical step + new_bubble_box_for_copy = pyautogui.locateOnScreen(bubble_snapshot, + region=search_area, + confidence=BUBBLE_RELOCATE_CONFIDENCE) + except Exception as e: + print(f"Exception during bubble location before copy: {e}") - time.sleep(1.5) # Polling interval + if not new_bubble_box_for_copy: + print("Warning: Failed to re-locate bubble before copying text. Skipping this bubble.") + continue # Skip to the next bubble in the outer loop + + print(f"Successfully re-located bubble for copy at: {new_bubble_box_for_copy}") + # Define the region based on the re-located bubble to find the keyword again + copy_bubble_region = (new_bubble_box_for_copy.left, new_bubble_box_for_copy.top, + new_bubble_box_for_copy.width, new_bubble_box_for_copy.height) + + # Find the keyword *again* within the *new* bubble region to get current coords + current_keyword_coords = detector.find_keyword_in_region(copy_bubble_region) + if not current_keyword_coords: + print("Warning: Keyword not found in the re-located bubble region. Skipping this bubble.") + continue # Skip to the next bubble + + # Determine if it's a reply keyword based on the *new* location/region + is_reply_keyword_current = False + # (Re-check is_reply_keyword logic here based on current_keyword_coords and copy_bubble_region) + # This check might be complex, for simplicity, we can reuse the 'is_reply_keyword' + # determined earlier based on the initial detection, assuming the keyword type doesn't change. + # Let's reuse the previously determined 'is_reply_keyword' for offset calculation. + click_coords_current = current_keyword_coords + if is_reply_keyword: # Use the flag determined from initial detection + click_coords_current = (current_keyword_coords[0], current_keyword_coords[1] + 25) + print(f"Applying +25 Y-offset for reply keyword (current location). Click target: {click_coords_current}") + else: + print(f"Detected keyword is not a reply type (current location). Click target: {click_coords_current}") + + # Interact: Get Bubble Text using current coordinates + bubble_text = interactor.copy_text_at(click_coords_current) + if not bubble_text: + print("Error: Could not get dialogue content for this bubble (after re-location).") + perform_state_cleanup(detector, interactor) # Attempt cleanup + continue # Skip to next bubble + + # Check recent text history + if bubble_text in recent_texts: + print(f"Content '{bubble_text[:30]}...' in recent history, skipping this bubble.") + continue # Skip to next bubble + + print(">>> New trigger event <<<") + # Add to recent texts *before* potentially long interaction + recent_texts.append(bubble_text) + + # 5. Interact: Get Sender Name (uses re-location internally via retrieve_sender_name_interaction) + sender_name = None + try: + # --- Bubble Re-location Logic --- + print("Attempting to re-locate bubble before getting sender name...") + if bubble_snapshot is None: + print("Error: Bubble snapshot missing for re-location. Skipping this bubble.") + continue + + # Try locating with decreasing confidence + new_bubble_box = None + confidences_to_try = [BUBBLE_RELOCATE_CONFIDENCE, BUBBLE_RELOCATE_FALLBACK_CONFIDENCE, 0.4] + for conf in confidences_to_try: + print(f"Attempting location with confidence {conf}...") + try: + new_bubble_box = pyautogui.locateOnScreen(bubble_snapshot, + region=search_area, + confidence=conf) + if new_bubble_box: + print(f"Successfully located with confidence {conf}.") + break # Found it + except Exception as e: + print(f"Exception during location attempt with confidence {conf}: {e}") + # --- End Confidence Loop --- + + if new_bubble_box: + new_tl_x, new_tl_y = new_bubble_box.left, new_bubble_box.top + print(f"Successfully re-located bubble snapshot at: ({new_tl_x}, {new_tl_y})") + new_avatar_coords = (new_tl_x + AVATAR_OFFSET_X_REPLY, new_tl_y + AVATAR_OFFSET_Y_REPLY) + print(f"Calculated new avatar coordinates for reply context: {new_avatar_coords}") + sender_name = interactor.retrieve_sender_name_interaction( + initial_avatar_coords=new_avatar_coords, + bubble_snapshot=bubble_snapshot, + search_area=search_area + ) + else: + print("Warning: Failed to re-locate bubble snapshot after multiple attempts.") + print("Trying direct approach with original bubble coordinates...") + original_tl_coords = target_bubble_info.get('tl_coords') + if original_tl_coords: + fallback_avatar_coords = (original_tl_coords[0] + AVATAR_OFFSET_X_REPLY, + original_tl_coords[1] + AVATAR_OFFSET_Y_REPLY) + print(f"Using fallback avatar coordinates from original detection: {fallback_avatar_coords}") + sender_name = interactor.retrieve_sender_name_interaction( + initial_avatar_coords=fallback_avatar_coords, + bubble_snapshot=bubble_snapshot, + search_area=search_area + ) + if not sender_name: + print("Direct approach failed. Skipping this trigger.") + perform_state_cleanup(detector, interactor) + continue # Skip to next bubble + else: + print("No original coordinates available. Skipping sender name retrieval.") + perform_state_cleanup(detector, interactor) + continue # Skip to next bubble + # --- End Bubble Re-location Logic --- + + except Exception as reloc_err: + print(f"Error during bubble re-location or subsequent interaction: {reloc_err}") + import traceback + traceback.print_exc() + perform_state_cleanup(detector, interactor) + continue # Skip to next bubble + + # 6. Perform Cleanup + cleanup_successful = perform_state_cleanup(detector, interactor) + if not cleanup_successful: + print("Error: Failed to return to chat screen after getting name. Skipping this bubble.") + continue # Skip to next bubble + + if not sender_name: + print("Error: Could not get sender name for this bubble, skipping.") + continue # Skip to next bubble + + # --- Attempt to activate reply context --- + reply_context_activated = False + try: + print("Attempting to activate reply context...") + if bubble_snapshot is None: + print("Warning: Bubble snapshot missing for reply context activation. Skipping.") + final_bubble_box_for_reply = None + else: + print(f"Attempting final re-location for reply context using search_area: {search_area}") + final_bubble_box_for_reply = pyautogui.locateOnScreen(bubble_snapshot, region=search_area, confidence=BUBBLE_RELOCATE_CONFIDENCE) + + if final_bubble_box_for_reply: + print(f"Final re-location successful at: {final_bubble_box_for_reply}") + bubble_x_reply, bubble_y_reply = final_bubble_box_for_reply.left, final_bubble_box_for_reply.top + bubble_w_reply, bubble_h_reply = final_bubble_box_for_reply.width, final_bubble_box_for_reply.height + center_x_reply = bubble_x_reply + bubble_w_reply // 2 + center_y_reply = bubble_y_reply + bubble_h_reply // 2 + + if is_reply_keyword: + center_y_reply += 15 + print(f"Applying +15 Y-offset to bubble center click for reply keyword. Target Y: {center_y_reply}") + + print(f"Clicking bubble center for reply at ({center_x_reply}, {center_y_reply})") + interactor.click_at(center_x_reply, center_y_reply) + time.sleep(0.15) + + print("Searching for reply button...") + reply_button_locs = detector._find_template('reply_button', confidence=0.8) + if reply_button_locs: + reply_coords = reply_button_locs[0] + print(f"Found reply button at {reply_coords}. Clicking...") + interactor.click_at(reply_coords[0], reply_coords[1]) + time.sleep(0.07) + reply_context_activated = True + print("Reply context activated.") + else: + print(">>> Reply button template ('reply_button') not found after clicking bubble center. <<<") + else: + print("Warning: Failed to re-locate bubble for activating reply context.") + + except Exception as reply_context_err: + print(f"!!! Error during reply context activation: {reply_context_err} !!!") + + # 7. Send Trigger Info to Main Thread + print("\n>>> Putting trigger info in Queue <<<") + print(f" Sender: {sender_name}") + print(f" Content: {bubble_text[:100]}...") + print(f" Bubble Region: {bubble_region}") # Original region for context + print(f" Reply Context Activated: {reply_context_activated}") + try: + data_to_send = { + 'sender': sender_name, + 'text': bubble_text, + 'bubble_region': bubble_region, # Send original region for context if needed + 'reply_context_activated': reply_context_activated, + 'bubble_snapshot': bubble_snapshot, # Send the snapshot used + 'search_area': search_area + } + trigger_queue.put(data_to_send) + print("Trigger info (with region, reply flag, snapshot, search_area) placed in Queue.") + + # --- CRITICAL: Break loop after successfully processing one trigger --- + print("--- Single bubble processing complete. Breaking scan cycle. ---") + break # Exit the 'for target_bubble_info in sorted_bubbles' loop + + except Exception as q_err: + print(f"Error putting data in Queue: {q_err}") + # Don't break if queue put fails, maybe try next bubble? Or log and break? + # Let's break here too, as something is wrong. + print("Breaking scan cycle due to queue error.") + break + + # End of keyword found block (if keyword_coords:) + # End of loop through sorted bubbles (for target_bubble_info...) + + # If the loop finished without breaking (i.e., no trigger processed), wait the full interval. + # If it broke, the sleep still happens here before the next cycle. + time.sleep(1.5) # Polling interval after checking all bubbles or processing one except KeyboardInterrupt: print("\nMonitoring interrupted.") @@ -591,8 +1576,8 @@ def run_ui_monitoring_loop(trigger_queue: queue.Queue, command_queue: queue.Queu # Attempt cleanup in case of unexpected error during interaction print("Attempting cleanup after unexpected error...") perform_state_cleanup(detector, interactor) - print("Waiting 5 seconds before retry...") - time.sleep(5) + print("Waiting 3 seconds before retry...") + time.sleep(3) # Note: The old monitor_chat_for_trigger function is replaced by the example_coordinator_loop. # The actual UI monitoring thread started in main.py should call a function like this example loop. diff --git a/window-monitor-script.py b/window-monitor-script.py new file mode 100644 index 0000000..6e75d07 --- /dev/null +++ b/window-monitor-script.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python +""" +Game Window Monitor Script - Keep game window on top and in position + +This script monitors a specified game window, ensuring it stays +always on top and at the desired screen coordinates. +""" + +import time +import argparse +import pygetwindow as gw +import win32gui +import win32con + +def find_window_by_title(window_title): + """Find the first window matching the title.""" + try: + windows = gw.getWindowsWithTitle(window_title) + if windows: + return windows[0] + except Exception as e: + # pygetwindow can sometimes raise exceptions if a window disappears + # during enumeration. Ignore these for monitoring purposes. + # print(f"Error finding window: {e}") + pass + return None + +def set_window_always_on_top(hwnd): + """Set the window to be always on top.""" + try: + win32gui.SetWindowPos(hwnd, win32con.HWND_TOPMOST, 0, 0, 0, 0, + win32con.SWP_NOMOVE | win32con.SWP_NOSIZE | win32con.SWP_SHOWWINDOW) + # print(f"Window {hwnd} set to always on top.") + except Exception as e: + print(f"Error setting window always on top: {e}") + +def move_window_if_needed(window, target_x, target_y): + """Move the window to the target coordinates if it's not already there.""" + try: + current_x, current_y = window.topleft + if current_x != target_x or current_y != target_y: + print(f"Window moved from ({current_x}, {current_y}). Moving back to ({target_x}, {target_y}).") + window.moveTo(target_x, target_y) + # print(f"Window moved to ({target_x}, {target_y}).") + except gw.PyGetWindowException as e: + # Handle cases where the window might close unexpectedly + print(f"Error accessing window properties (might be closed): {e}") + except Exception as e: + print(f"Error moving window: {e}") + +def main(): + parser = argparse.ArgumentParser(description='Game Window Monitor Tool') + parser.add_argument('--window_title', default="Last War-Survival Game", help='Game window title to monitor') + parser.add_argument('--x', type=int, default=50, help='Target window X coordinate') + parser.add_argument('--y', type=int, default=30, help='Target window Y coordinate') + parser.add_argument('--interval', type=float, default=1.0, help='Check interval in seconds') + + args = parser.parse_args() + + print(f"Monitoring window: '{args.window_title}'") + print(f"Target position: ({args.x}, {args.y})") + print(f"Check interval: {args.interval} seconds") + print("Press Ctrl+C to stop.") + + hwnd = None + last_hwnd_check_time = 0 + + try: + while True: + current_time = time.time() + window = None + + # Find window handle (HWND) - less frequent check if already found + # pygetwindow can be slow, so avoid calling it too often if we have a valid handle + if not hwnd or current_time - last_hwnd_check_time > 5: # Re-check HWND every 5 seconds + window_obj = find_window_by_title(args.window_title) + if window_obj: + # Get the HWND (window handle) needed for win32gui + # Accessing _hWnd is using an internal attribute, but it's common practice with pygetwindow + try: + hwnd = window_obj._hWnd + window = window_obj # Keep the pygetwindow object for position checks + last_hwnd_check_time = current_time + # print(f"Found window HWND: {hwnd}") + except AttributeError: + print("Could not get HWND from window object. Retrying...") + hwnd = None + else: + if hwnd: + print(f"Window '{args.window_title}' lost.") + hwnd = None # Reset hwnd if window not found + + if hwnd: + # Ensure it's always on top + set_window_always_on_top(hwnd) + + # Check and correct position using the pygetwindow object if available + # Re-find the pygetwindow object if needed for position check + if not window: + window = find_window_by_title(args.window_title) + + if window: + move_window_if_needed(window, args.x, args.y) + else: + # If we have hwnd but can't get pygetwindow object, maybe it's closing + print(f"Have HWND {hwnd} but cannot get window object for position check.") + hwnd = None # Force re-find next cycle + + else: + # print(f"Window '{args.window_title}' not found. Waiting...") + pass # Wait for the window to appear + + time.sleep(args.interval) + + except KeyboardInterrupt: + print("\nMonitoring stopped by user.") + except Exception as e: + print(f"\nAn unexpected error occurred: {e}") + +if __name__ == "__main__": + main()