phone-harnessHarness
§ · Nested detail page

Mirroring 传输层 深度解析

Mirroring 传输层 这页聚焦 窗口发现、截图回退与 HID 事件。关键证据包括 src/phone_harness/helpers.py:45、src/phone_harness/mirror.py:91、src/phone_harness/ocr.py:19、src/phone_harness/admin.py:12,所有判断都回到真实源码行号。

§ 00 · IMPORT CHAIN

导入链与契约

窗口发现不是靠 Accessibility tree,而是枚举 CGWindow 列表。这里的关键证据是 src/phone_harness/mirror.py:24、src/phone_harness/mirror.py:29、src/phone_harness/mirror.py:33、src/phone_harness/mirror.py:38:它只接受 iPhone Mirroring 主窗口,并把全局屏幕点作为后续输入坐标。

src/phone_harness/mirror.py · L24-35Python
24def find_window():
25    """{x, y, w, h, id} of the mirroring window in screen points, or None."""
26    wins = Quartz.CGWindowListCopyWindowInfo(
27        Quartz.kCGWindowListOptionOnScreenOnly, Quartz.kCGNullWindowID) or []
28    for w in wins:
29        if w.get("kCGWindowOwnerName") == APP_NAME and w.get("kCGWindowLayer", 1) == 0:
30            b = w["kCGWindowBounds"]
31            if b["Width"] < 100:  # ignore panels/toolbars
32                continue
33            return {"x": b["X"], "y": b["Y"], "w": b["Width"], "h": b["Height"],
34                    "id": int(w["kCGWindowNumber"])}
35    return None
§ 01 · CORE IMPL

核心实现,逐段拆

▸截图回退

capture() 先用 window id 截图,失败后激活窗口并按区域截取。src/phone_harness/mirror.py:91、src/phone_harness/mirror.py:105、src/phone_harness/mirror.py:112、src/phone_harness/mirror.py:118 这组行号解释了为什么视频窗口被遮挡或未 composited 时仍有第二条路径。

src/phone_harness/mirror.py · L91-118Python
91def capture(path=None, retries=2):
92    """Capture the mirroring window as a PNG. Returns (path, window_bounds).
93
94    `screencapture -l <id>` grabs only the window (no shadow, even if covered)
95    but fails with "could not create image from window" when the window is not
96    frontmost/composited. So we activate first, and fall back to a region
97    capture of the window rect, which succeeds regardless of window backing
98    (the app is frontmost by then, so nothing occludes the rect).
99    """
100    path = str(path or TMP / "window.png")
101    last = None
102    for attempt in range(retries + 1):
103        win = find_window() or ensure_window()
104        # -l: window-only, ideal when it works
105        ok, last = _run_capture(
106            ["screencapture", "-x", "-o", "-l", str(win["id"]), path], path)
107        if ok:
108            return path, win
109        # not backed — bring it frontmost and retry the region instead
110        activate()
111        win = find_window() or win
112        region = f"{int(win['x'])},{int(win['y'])},{int(win['w'])},{int(win['h'])}"
113        ok, last = _run_capture(
114            ["screencapture", "-x", "-R", region, path], path)
115        if ok:
116            return path, win
117        time.sleep(0.3)
118    raise RuntimeError(f"window capture failed after {retries + 1} tries: {last}")

▸HID 输入

tap、drag、scroll_wheel 都通过 Quartz CGEvent 发到 HID tap。src/phone_harness/mirror.py:123、src/phone_harness/mirror.py:133、src/phone_harness/mirror.py:151、src/phone_harness/mirror.py:165、src/phone_harness/mirror.py:171 说明它绕开无语义的视频窗口,把鼠标与滚轮变成 iOS 触摸。

src/phone_harness/mirror.py · L123-175Python
123def _post_mouse(etype, x, y):
124    ev = Quartz.CGEventCreateMouseEvent(
125        None, etype, Quartz.CGPointMake(x, y), Quartz.kCGMouseButtonLeft)
126    Quartz.CGEventPost(Quartz.kCGHIDEventTap, ev)
127
128
129def _focus():
130    activate()
131
132
133def tap(x, y):
134    _focus()
135    _post_mouse(Quartz.kCGEventMouseMoved, x, y)
136    time.sleep(0.1)
137    _post_mouse(Quartz.kCGEventLeftMouseDown, x, y)
138    time.sleep(0.06)
139    _post_mouse(Quartz.kCGEventLeftMouseUp, x, y)
140
141
142def long_press(x, y, duration=0.8):
143    _focus()
144    _post_mouse(Quartz.kCGEventMouseMoved, x, y)
145    time.sleep(0.1)
146    _post_mouse(Quartz.kCGEventLeftMouseDown, x, y)
147    time.sleep(duration)
148    _post_mouse(Quartz.kCGEventLeftMouseUp, x, y)
149
150
151def drag(x1, y1, x2, y2, duration=0.35, steps=14):
152    """Touch-drag (what iOS sees as a swipe)."""
153    _focus()
154    _post_mouse(Quartz.kCGEventMouseMoved, x1, y1)
155    time.sleep(0.1)
156    _post_mouse(Quartz.kCGEventLeftMouseDown, x1, y1)
157    for i in range(1, steps + 1):
158        t = i / steps
159        _post_mouse(Quartz.kCGEventLeftMouseDragged,
160                    x1 + (x2 - x1) * t, y1 + (y2 - y1) * t)
161        time.sleep(duration / steps)
162    _post_mouse(Quartz.kCGEventLeftMouseUp, x2, y2)
163
164
165def scroll_wheel(dy, x, y, steps=6):
166    """Scroll-gesture at (x, y). Positive dy scrolls content up (finger down)."""
167    _focus()
168    _post_mouse(Quartz.kCGEventMouseMoved, x, y)
169    time.sleep(0.1)
170    for _ in range(steps):
171        ev = Quartz.CGEventCreateScrollWheelEvent(
172            None, Quartz.kCGScrollEventUnitPixel, 1, int(dy / steps))
173        Quartz.CGEventSetLocation(ev, Quartz.CGPointMake(x, y))
174        Quartz.CGEventPost(Quartz.kCGHIDEventTap, ev)
175        time.sleep(0.03)
§ 02 · KEY FLOW

关键流程

capture / OCR / helper → compute screen-point evidence → post Quartz event or raise user-action error → capture again and compare visible state

这个流程的关键是每一步都有可观察证据:src/phone_harness/mirror.py:84、src/phone_harness/ocr.py:48、src/phone_harness/helpers.py:73、src/phone_harness/helpers.py:316、src/phone_harness/admin.py:65 分别覆盖截图、OCR 坐标、screen_info、稳定等待和 doctor 结果。它不像浏览器自动化那样有 DOM,所以截图就是最终断言。

§ 03 · EDGE CASES

边界 / 失败模式

  • src/phone_harness/helpers.py:67 遇到 blocked interstitial 时要求用户处理,防止 agent 越权。
  • src/phone_harness/mirror.py:251 对无法输入字符抛错,避免用户以为文本已发送。
  • src/phone_harness/helpers.py:280 需要连续不移动才结束列表,避免 lazy-load 误判。
  • src/phone_harness/admin.py:55 用截图字节数识别空 capture,提示重启终端让录屏权限生效。
Sub-page · 2026-08-10 Source · ShawnPana/phone-harness ↑ phone-harness 主页面