Mirroring 传输层 深度解析
Mirroring 传输层 这页聚焦 窗口发现、截图回退与 HID 事件。关键证据包括 src/phone_harness/helpers.py:45、src/phone_harness/mirror.py:91、src/phone_harness/ocr.py:19、src/phone_harness/admin.py:12,所有判断都回到真实源码行号。
导入链与契约
窗口发现不是靠 Accessibility tree,而是枚举 CGWindow 列表。这里的关键证据是 src/phone_harness/mirror.py:24、src/phone_harness/mirror.py:29、src/phone_harness/mirror.py:33、src/phone_harness/mirror.py:38:它只接受 iPhone Mirroring 主窗口,并把全局屏幕点作为后续输入坐标。
24def find_window(): 25 """{x, y, w, h, id} of the mirroring window in screen points, or None.""" 26 wins = Quartz.CGWindowListCopyWindowInfo( 27 Quartz.kCGWindowListOptionOnScreenOnly, Quartz.kCGNullWindowID) or [] 28 for w in wins: 29 if w.get("kCGWindowOwnerName") == APP_NAME and w.get("kCGWindowLayer", 1) == 0: 30 b = w["kCGWindowBounds"] 31 if b["Width"] < 100: # ignore panels/toolbars 32 continue 33 return {"x": b["X"], "y": b["Y"], "w": b["Width"], "h": b["Height"], 34 "id": int(w["kCGWindowNumber"])} 35 return None
核心实现,逐段拆
▸截图回退
capture() 先用 window id 截图,失败后激活窗口并按区域截取。src/phone_harness/mirror.py:91、src/phone_harness/mirror.py:105、src/phone_harness/mirror.py:112、src/phone_harness/mirror.py:118 这组行号解释了为什么视频窗口被遮挡或未 composited 时仍有第二条路径。
91def capture(path=None, retries=2): 92 """Capture the mirroring window as a PNG. Returns (path, window_bounds). 93 94 `screencapture -l <id>` grabs only the window (no shadow, even if covered) 95 but fails with "could not create image from window" when the window is not 96 frontmost/composited. So we activate first, and fall back to a region 97 capture of the window rect, which succeeds regardless of window backing 98 (the app is frontmost by then, so nothing occludes the rect). 99 """ 100 path = str(path or TMP / "window.png") 101 last = None 102 for attempt in range(retries + 1): 103 win = find_window() or ensure_window() 104 # -l: window-only, ideal when it works 105 ok, last = _run_capture( 106 ["screencapture", "-x", "-o", "-l", str(win["id"]), path], path) 107 if ok: 108 return path, win 109 # not backed — bring it frontmost and retry the region instead 110 activate() 111 win = find_window() or win 112 region = f"{int(win['x'])},{int(win['y'])},{int(win['w'])},{int(win['h'])}" 113 ok, last = _run_capture( 114 ["screencapture", "-x", "-R", region, path], path) 115 if ok: 116 return path, win 117 time.sleep(0.3) 118 raise RuntimeError(f"window capture failed after {retries + 1} tries: {last}")
▸HID 输入
tap、drag、scroll_wheel 都通过 Quartz CGEvent 发到 HID tap。src/phone_harness/mirror.py:123、src/phone_harness/mirror.py:133、src/phone_harness/mirror.py:151、src/phone_harness/mirror.py:165、src/phone_harness/mirror.py:171 说明它绕开无语义的视频窗口,把鼠标与滚轮变成 iOS 触摸。
123def _post_mouse(etype, x, y): 124 ev = Quartz.CGEventCreateMouseEvent( 125 None, etype, Quartz.CGPointMake(x, y), Quartz.kCGMouseButtonLeft) 126 Quartz.CGEventPost(Quartz.kCGHIDEventTap, ev) 127 128 129def _focus(): 130 activate() 131 132 133def tap(x, y): 134 _focus() 135 _post_mouse(Quartz.kCGEventMouseMoved, x, y) 136 time.sleep(0.1) 137 _post_mouse(Quartz.kCGEventLeftMouseDown, x, y) 138 time.sleep(0.06) 139 _post_mouse(Quartz.kCGEventLeftMouseUp, x, y) 140 141 142def long_press(x, y, duration=0.8): 143 _focus() 144 _post_mouse(Quartz.kCGEventMouseMoved, x, y) 145 time.sleep(0.1) 146 _post_mouse(Quartz.kCGEventLeftMouseDown, x, y) 147 time.sleep(duration) 148 _post_mouse(Quartz.kCGEventLeftMouseUp, x, y) 149 150 151def drag(x1, y1, x2, y2, duration=0.35, steps=14): 152 """Touch-drag (what iOS sees as a swipe).""" 153 _focus() 154 _post_mouse(Quartz.kCGEventMouseMoved, x1, y1) 155 time.sleep(0.1) 156 _post_mouse(Quartz.kCGEventLeftMouseDown, x1, y1) 157 for i in range(1, steps + 1): 158 t = i / steps 159 _post_mouse(Quartz.kCGEventLeftMouseDragged, 160 x1 + (x2 - x1) * t, y1 + (y2 - y1) * t) 161 time.sleep(duration / steps) 162 _post_mouse(Quartz.kCGEventLeftMouseUp, x2, y2) 163 164 165def scroll_wheel(dy, x, y, steps=6): 166 """Scroll-gesture at (x, y). Positive dy scrolls content up (finger down).""" 167 _focus() 168 _post_mouse(Quartz.kCGEventMouseMoved, x, y) 169 time.sleep(0.1) 170 for _ in range(steps): 171 ev = Quartz.CGEventCreateScrollWheelEvent( 172 None, Quartz.kCGScrollEventUnitPixel, 1, int(dy / steps)) 173 Quartz.CGEventSetLocation(ev, Quartz.CGPointMake(x, y)) 174 Quartz.CGEventPost(Quartz.kCGHIDEventTap, ev) 175 time.sleep(0.03)
关键流程
这个流程的关键是每一步都有可观察证据:src/phone_harness/mirror.py:84、src/phone_harness/ocr.py:48、src/phone_harness/helpers.py:73、src/phone_harness/helpers.py:316、src/phone_harness/admin.py:65 分别覆盖截图、OCR 坐标、screen_info、稳定等待和 doctor 结果。它不像浏览器自动化那样有 DOM,所以截图就是最终断言。
边界 / 失败模式
src/phone_harness/helpers.py:67遇到 blocked interstitial 时要求用户处理,防止 agent 越权。src/phone_harness/mirror.py:251对无法输入字符抛错,避免用户以为文本已发送。src/phone_harness/helpers.py:280需要连续不移动才结束列表,避免 lazy-load 误判。src/phone_harness/admin.py:55用截图字节数识别空 capture,提示重启终端让录屏权限生效。