mac-auto-bridge Cloud Screen-UI OCR: Service Selection and Architecture Research

Which OCR + UI-grounding stack gives pixel-accurate, clickable coordinates for macOS screen automation? A service-selection and architecture study — PP-OCRv6, GUI-Owl-1.5, Holo3, and Google Gemini 2.5 Computer Use — updated through June 2026.

OCRGUI AutomationComputer Usemac-auto-bridgePaddleOCRUI GroundingmacOSVLM

Covers technology through June 2026. The supplement (2026 updates) has been integrated throughout; superseded recommendations are noted inline.

TL;DR — Core Conclusions

  • No single off-the-shelf service satisfies all eight requirements. The most pragmatic architecture is: a self-hosted PP-OCRv6 (medium tier) — or its ONNX-wrapped sibling RapidOCR as a lightweight fallback — as the coordinate-accurate "text + bounding-box + confidence" primary engine, packaged as your own cloud REST/gRPC API; a UI grounding model (GUI-Owl-1.5 for accuracy, or Holo3.1 quantized small models for latency) as the "clickable-element / icon" detection layer; and Google Gemini 2.5 Computer Use API (updated recommendation; see below) as the cloud fallback for hard samples.

  • For your top priority — coordinate accuracy — dedicated two-stage OCR (PaddleOCR/RapidOCR/Surya) outperforms general VLMs. PP-OCRv6 returns four-vertex polygons (dt_polys) plus per-box confidence (dt_scores); localization is precise and hallucination-free. General VLMs (GPT-4o / Claude / Qwen) frequently produce offset coordinates, require normalization conversion, and can hallucinate. Given your core pain point — "a misclicked point is worse than a misread character" — dedicated OCR must be the ground truth for coordinates; VLM/grounding models serve only as semantic and icon supplements.

  • Region/window-cropped OCR is the single most effective lever for simultaneously satisfying latency, cost, and accuracy requirements. Crop a local image first using the macOS Accessibility (AX) tree or window bounds, then feed it to OCR. With GPU + TensorRT the local-window path can reach 100–300 ms; full-screen 2K–4K images run the GPU self-hosted path at 500 ms–1 s.

  • Three-layer stack update (June 2026): The original recommended stack (self-hosted OCR base + grounding enhancement + cloud fallback) is structurally sound and requires no fundamental redesign. All three layers should be upgraded: base → PP-OCRv6; grounding → GUI-Owl-1.5 (accuracy) / Holo3.1 quantized (latency); fallback → Gemini 2.5 Computer Use.

Key Findings

1. Three Candidate Classes, Each with Trade-offs

  • Cloud OCR APIs (Google Vision, Azure Read, AWS Textract, Baidu/Tencent/Alibaba/Huawei): ready to use out of the box, return bounding boxes + confidence scores, but coordinates are whole-image pixel coordinates; a single network round-trip typically costs 200 ms–1 s+; billed per call; no awareness of "window semantics." Cloud OCR in 2026 remains document-centric — no dedicated low-latency UI-grounding endpoints have appeared.

  • Self-hosted open-source OCR (PaddleOCR/PP-OCRv6, RapidOCR, Surya, Tesseract, EasyOCR): deployable on your own GPU; coordinate-accurate; supports region OCR and batch processing; low unit cost; the best foundation for building your own cloud service.

  • UI grounding / GUI agent models (OmniParser v2, GUI-Owl-1.5, UI-TARS, Qwen3-VL, Holo3/3.1, UGround, OS-Atlas, GTA1, UI-Venus, etc.): output coordinates of "clickable elements" directly, more closely aligned with your actual goal (click target), but higher latency (0.6–2 s/frame); require GPU; coordinates must be normalized and converted.

2. Coordinate Accuracy Comparison (Top Priority)

  • PP-OCRv6 (supersedes PP-OCRv5 as the recommended base): classic two-stage detection + recognition pipeline; outputs dt_polys (four-vertex polygon per box) + dt_scores (per-box confidence); text + per-line confidence on recognition. Deterministic, replayable, save_to_json-stable. PP-OCRv6 medium achieves 83.2% recognition accuracy and 86.2% detection Hmean, outperforming PP-OCRv5_server by +5.1% and +4.6% respectively, while reporting 93.2% vs Qwen3-VL-235B's 80.6% on hallucination rate — with ~6 800× fewer parameters (vendor self-evaluation; verify on your own dataset). Paper: arXiv:2606.13108, released 2026-06-11 with PaddleOCR 3.7.0.

  • Google Cloud Vision returns boundingPoly.vertices (four-vertex polygon, whole-image pixel coordinates); Page→Block→Paragraph→Word→Symbol hierarchy with bounding boxes at each level. Note: when a coordinate value is 0, that field is omitted in the response.

  • Baidu General Text Recognition (with position) returns location: {top, left, width, height} (rectangle, pixel coordinates); supports CJK-Latin mixed text.

  • VLM/grounding models mostly output normalized coordinates: Qwen2.5-VL uses absolute coordinates as JSON; Qwen3-VL switched to [0, 1000] normalized; UI-TARS is based on Qwen2-VL and uses absolute coordinates requiring smart_resize conversion. Coordinate offset is a known issue (the Qwen2.5-VL-7B community has reported y-axis offsets). GUI-Owl-1.5 is based on Qwen3-VL; its exact coordinate convention is not fully confirmed from primary sources — verify in its cookbook before integrating. Holo3/3.1 explicitly uses [0, 1000] normalized coordinates, integer values, top-left origin.

3. Latency Benchmarks

CandidateConditionLatency
PP-OCRv6_mediumA100 GPU0.13 s
PP-OCRv6_tinyApple M40.96 s (6.1× faster than PP-OCRv5_server 5.82 s)
PP-OCRv6_mediumIntel Xeon CPU + OpenVINO1.40 s (5.2× faster than PP-OCRv5_server 7.30 s)
PP-OCRv5_server (detection only)V100 GPU, 2048-px input~253 ms
PP-OCRv5_serverCPU ONNX~3 035 ms (GPU mandatory)
RapidOCR (ONNX)CPU~0.2 s inference; ~80 MB install
TurboOCR (PP-OCRv5 + TensorRT FP16, RTX 5090)FUNSD A4 form270 img/s; sparse images 1 200+ img/s
OmniParser v2A1000.6 s/frame (60% improvement over V1)
OmniParser v2RTX 40900.8 s/frame
UI-TARS-2 W4A8 quantized—2.5 s/round-trip (4.0 s unquantized)
Cloud APIs (Azure/Google)network includedtypically 1–2 s

Speed figures for PP-OCRv6 are from the PaddleOCR official documentation (paddleocr.ai). TurboOCR figures demonstrate that a self-hosted GPU pipeline can comfortably meet your latency targets.

4. Latest Developments, 2025–2026

This field is evolving extremely fast.

OCR side:

  • PaddleOCR 3.0 / PP-OCRv5 (2025) → PP-OCRv6 (2026-06-11, arXiv:2606.13108): 1.5M/7.7M/34.5M parameters (tiny/small/medium); LCNetV4 unified backbone; RepLKFPN detection neck; EncoderWithLightSVTR (CTC + NRTR multi-head) recognition neck. Single model covers 50 languages. Explicitly improves on digital displays, dot-matrix characters, industrial text, and small point sizes — directly relevant to terminal/menu-bar/code-editor content.
  • PaddleOCR-VL-1.6 (2026-05-28): 0.9B VLM; OmniDocBench v1.6 96.3%; document-parsing oriented.
  • GLM-OCR (Zhipu Z.ai, 2026-03-11, arXiv:2603.10910): 0.9B, MIT license; OmniDocBench V1.5 94.62. Key limitation: the base model does not natively output bounding boxes; structured boxes require the vendor's GitHub SDK. Not ideal for UI coordinate scenarios.
  • HunyuanOCR (Tencent, open-sourced 2025-11-25, stable demo 2026-01-13): 1B end-to-end (Hunyuan ViT 0.4B + MLP + 0.5B LM); explicitly supports line-level "text content + coordinates" output; covers screenshots and games — directly relevant. OCRBench 860 (SOTA for <3B). Custom community license with regional/use restrictions; requires legal review for commercial use.
  • FireRed-OCR (Xiaohongshu, 2026-02-28, arXiv:2603.01840): 2B, OmniDocBench v1.5 92.94; Format-Constrained GRPO. Document-structure oriented.
  • DeepSeek-OCR-2 / dots.ocr-1.5 / MiniMax-M3-VL (all 2026): VLM-OCR iterations; document parsing primary; latency too high for real-time UI OCR.

GUI grounding side — ScreenSpot-Pro single-shot accuracy (higher is better):

ScreenSpot-Pro (arXiv:2504.07981, Li et al.) reported that the best existing model scored only 18.9% (OS-Atlas-7B) at time of publication. The field has since advanced rapidly:

ModelScreenSpot-Pro (single-shot)Notes
OS-Atlas-7B18.9%Baseline at ScreenSpot-Pro paper publication
UGround-72B34.5%
OmniParser v2 + GPT-4o39.6%vs. GPT-4o alone at 0.8%
GUI-Owl-7B / GUI-Owl-32B~54.9% / ~58.0%Earlier GUI-Owl; independently verified
GTA1-7B / GTA1-32B50.1% / 63.6%arXiv:2507.05791; 7B beats UGround-72B
UI-Venus-7B / UI-Venus-72B50.8% / 61.9%
Holo1.5-7B57.94%
Holo2-30B-A3B66.1%
Qwen3-VL-235B~61.8%
UI-TARS-1.5-7B61.6% (ByteDance blog) / ~42–49% (third-party)See Caveats for regime difference
GUI-Owl-1.5 / Mobile-Agent-v3.572.9% (single-shot) / 80.3% (two-stage crop)Self-reported, arXiv:2602.16855; verify before adopting
MAI-UI67.9% / 73.5% (zoom-in)
Holo3-35B-A3BSee vendor reportApache-2.0; OSWorld-Verified 77.8% (vendor)

Qwen3-VL family (2025 Q4–2026 Q1 flash snapshots): open-source 2B/4B/8B/32B dense + 30B-A3B/235B-A22B MoE; native GUI grounding (bbox + point output); coordinates switched to [0, 1000] normalized (resolution-independent). Note: vLLM ≥0.11.1 has a coordinate accuracy regression bug for Qwen3-VL grounding (GitHub vllm #29595). Lock versions and validate before deploying. Raw ScreenSpot-Pro for 4B/8B/30B-A3B ≈ 52.7–55.5%; can be boosted with crop/training methods (e.g., VISTA, arXiv:2606.14579, raises 30B-A3B to 67.0; GUI-CURSOR multi-step visual feedback raises 7B from 26.8→56.5).

UI-TARS-1.5-7B: open weights, Apache-2.0; outputs absolute pixel coordinates; designed for GUI click/drag grounding; can be self-hosted via vLLM. UI-TARS-2 (2025-09-04, arXiv:2509.02544): Online-Mind2Web 88.2%, OSWorld 47.5%, WindowsAgentArena 50.6%; W4A8 quantization reduces per-round latency 4.0→2.5 s, throughput 29.6→47 tok/s, OSWorld 47.5→44.4 (acceptable quality-cost trade-off). Confirmed: there is no UI-TARS-3 as of June 2026. The most recent ByteDance release remains UI-TARS-2; the ui-tars-desktop repository contains only tooling/application-layer updates (Agent TARS CLI, etc.), not a new model generation.

New GUI grounding models (2026):

  • GUI-Owl-1.5 / Mobile-Agent-v3.5 (Alibaba Tongyi Lab, 2026-02-15, arXiv:2602.16855): based on Qwen3-VL; 2B/4B/8B/32B/235B (Instruct + Thinking variants); MIT license; weights fully open on HuggingFace (mPLUG org); GitHub: X-PLUG/MobileAgent; trained with MRPO (multi-platform environment RL). vLLM deployment validated on A100 96 GB. Self-reports ScreenSpot-Pro 72.9% (single-shot) / 80.3% (two-stage crop-refine) and OSWorld-Verified 56.5% (vs. UI-TARS-2 53.1%, OpenCUA-72B 45.0%). Must self-verify before adopting — see Caveats.
  • Holo3 (H Company, 2026-03): Holo3-35B-A3B (Apache-2.0, Qwen3.5 MoE, 35B total / 3B active, vendor-reported OSWorld-Verified 77.8%) and Holo3-122B-A10B (vendor-reported OSWorld-Verified 78.85%, research license). Explicit coordinate format: [0, 1000] normalized, integer, top-left origin.
  • Holo3.1 (H Company, 2026-06): adds 0.8B/4B/9B/35B-A3B; first release of quantized checkpoints (FP8 / Q4 GGUF / NVFP4 for NVIDIA Blackwell); adds native function-calling (in addition to existing structured JSON output); AndroidWorld 35B improved from 67%→79.3%. Small quantized variants (4B/9B) are promising for the "local-crop 100–300 ms" target.
  • Step-GUI (Stepfun, arXiv:2512.15431) and GUI-Actor (Microsoft, coordinate-free attention action head; 7B exceeds UI-TARS-72B): research-oriented.

New benchmarks (2025–2026): OSWorld-Verified (official upgraded version), OSWorld-G/Jedi (NeurIPS 2025 spotlight, 564 samples / 32 element classes / includes refusal), VenusBench-GD (2025-12), MMBench-GUI-L2, ScreenSpot-Pro.

Apple WWDC26 (2026-06-08): Foundation Models framework now supports multimodal prompts; the Vision framework's OCR and barcode tools can be called directly by the on-device model, enabling hybrid on-device pre-filtering + cloud grounding architectures.

Details — Candidate-by-Candidate Technical Specifics

PP-OCRv6 (Primary Base — Supersedes PP-OCRv5)

Paper: arXiv:2606.13108, "From 1.5M to 34.5M Parameters, Surpassing Billion-Scale VLMs on OCR Tasks"; released 2026-06-11 with PaddleOCR 3.7.0.

Architecture: LCNetV4 (MetaFormer-style, structural reparameterization) unified backbone; RepLKFPN detection neck; EncoderWithLightSVTR (CTC + NRTR multi-head) recognition neck. Three tiers: tiny (1.5M) / small (7.7M) / medium (34.5M).

Why it is the right base for coordinate-accurate UI OCR:

  • Retains the classic two-stage detection + recognition pipeline; outputs text boxes + recognition confidence — exactly what the coordinate-precision scenario requires. Unlike VLM generative coordinates, no hallucination risk.
  • PP-OCRv6_medium: 83.2% recognition accuracy, 86.2% detection Hmean (vendor multi-scenario benchmark), outperforming PP-OCRv5_server by +5.1% / +4.6%. Hallucination rate: 93.2% vs Qwen3-VL-235B's 80.6%, with ~6 800× fewer parameters. (These are vendor self-evaluation benchmarks; validate on your own screen sample set.)
  • Speed: medium at 0.13 s on A100 GPU (satisfies the full-screen <1 s target); tiny at 0.96 s on Apple M4 (6.1× speedup over PP-OCRv5_server). Local-crop inference with the medium model is expected to satisfy the 100–300 ms target on a GPU host.
  • Explicitly addresses digital displays, dot-matrix characters, industrial text, and small point sizes — directly applicable to terminal, menu bar, and code editor content.
  • Single model covers 50 languages: Simplified/Traditional Chinese, Chinese pinyin, English, Japanese, + 46 Latin-script languages; strong CJK-Latin mixed text — matches your CJK + symbols + file-path + command-line text requirements.

Output: dt_polys (four-vertex polygon per detected box) + dt_scores (per-box confidence); recognized text + per-line confidence. save_to_json-stable, replayable, usable as evidence records.

Region OCR: supports slice/tiling with configurable stride; engine auto-merges. Can also infer directly on a cropped image to obtain local-origin coordinates.

Deployment: Apache-2.0; CUDA / ONNX Runtime / OpenVINO. RapidOCR has an open tracking issue (#686) for v6 support; RapidOCR remains the recommended ONNX lightweight fallback in the interim.

Sorting stability: Implement explicit top-to-bottom, left-to-right stable sorting yourself (PaddleOCR provides reading-order / XY-cut utilities; TurboOCR provides class-aware XY-cut).

PP-OCRv5 (Reference — Previous Generation)

Still valid context for understanding the baseline. PP-OCRv5_server produces four-vertex polygons + per-box confidence, strong CJK-Latin mixed text, supports 100+ languages. V5's larger character dictionary makes it slower than V4 on the same hardware; GPU + TensorRT is required. V6 is a direct drop-in upgrade and is uniformly superior; migrate as soon as practical.

RapidOCR (ONNX Lightweight Fallback)

Converts PaddleOCR models to ONNX and removes the PaddlePaddle dependency. Approximately 0.2 s inference; ~80 MB install size; multi-language bindings (Python, C++, Java, C#); cross-platform including macOS. Four-vertex box output. Best suited for CPU fallback or lightweight deployments. The .NET variant RapidOcrNet uses PP-OCRv5 models (pending v6 support). No native Rust binding, but callable via REST.

Surya

Supports 90+ languages; line-level detection; built-in layout / table analysis; good reading-order output. GPU recommended; 1.5–2 s per page; ~200 MB. GPL license — commercial use restrictions apply.

Tesseract / EasyOCR

Tesseract is lightweight and fast but weak on small fonts and complex UIs. EasyOCR has systematic misreads for currency symbols and similar characters — high risk for financial / command-line text. Neither is recommended as the primary engine for screen UI.

New-generation VLM-OCR (PaddleOCR-VL-1.6, GLM-OCR, HunyuanOCR, FireRed-OCR, DeepSeek-OCR-2)

Can return text + bbox in a single pass, but these are document-OCR oriented, require GPU, and have higher latency than the two-stage pipeline. Suitable as a fallback sub-channel for complex documents, tables, or difficult samples:

  • HunyuanOCR is the most relevant for UI because it explicitly covers screenshots and games, and supports line-level coordinate output. Check the custom community license before commercial use.
  • GLM-OCR does not natively output bounding boxes from the base model; structured boxes require the vendor SDK — not ideal for UI coordinate workflows.
  • FireRed-OCR: document structure oriented; OmniDocBench v1.5 92.94.
  • PaddleOCR-VL-1.6: OmniDocBench v1.6 96.3%; complex-layout fallback only.

Google Cloud Vision (TEXT_DETECTION / DOCUMENT_TEXT_DETECTION)

  • Output: four-vertex boundingPoly (whole-image pixel coordinates); DOCUMENT mode provides paragraph-level hierarchy. Among third-party benchmarks, frequently achieves top accuracy (ROUGE FUNSD 75.0% / STROIE 87.8%).
  • Pricing (Google Cloud Vision official pricing page): first 1,000 units/feature/month free; 1,001–5,000,000 at $1.50/1,000; above 5,000,000 at $0.60/1,000 (TEXT_DETECTION and DOCUMENT_TEXT_DETECTION priced the same; billed as feature × image units).
  • Integration: REST + gRPC; Rust can call directly via REST. Enable enable_text_detection_confidence_score for confidence values.
  • 2026 note: Document AI requires migration to pretrained-ocr-v2.1-2024-08-07 before 2026-06-30. No new low-latency UI-specific endpoints added in 2026.

Azure AI Vision Read API (now "Azure Vision in Foundry Tools")

  • Output: structured JSON with bbox coordinates + 0–1 confidence; supports mixed-language and print + handwritten text.
  • Azure Vision 4.0 with synchronous Read OCR (Image Analysis 4.0) is optimized for non-document images and supports a low-latency synchronous channel suitable for real-time UI.
  • 164 languages GA for the Read API.
  • Migration notice: Computer Vision API versions 1.0–3.1 retire 2026-09-13; migrate to 3.2 / 4.0.
  • Supports containerized local/edge deployment (data sovereignty / lower latency).

Baidu / Tencent / Alibaba / Huawei (Strong for CJK Scenarios)

  • Baidu General Text Recognition (with position): location: {top, left, width, height} rectangular pixel coordinates; standard / high-precision × with-position = 4 tiers; CJK + 20+ language mixed support; public cloud SLA 99.9%.
  • Tencent Cloud General Printed Text: returns coordinates; recognition time typically 200 ms–1 s (varies by image size / character count / network); printed text accuracy 95%+; general tier ¥0.8/thousand calls; high-precision ¥2.4/thousand.
  • Alibaba Cloud ReadSense (读光) OCR: DAMO Academy; pay-as-you-go.
  • These services generally outperform Western vendors for CJK mixed-script and small-font Chinese UIs, but coordinates remain whole-image pixels and network round-trip latency is still present.

AWS Textract / Rekognition

Textract is among the faster services in latency benchmarks but is form/document oriented. No significant new 2026 models noted.

Mathpix

include_line_data returns line_data (type, four-vertex cnt polygon, confidence, is_printed/is_handwritten, image_width/image_height) — rich diagnostic data for code editor / terminal / math / file-path / wide-character scenarios. v3/text from $0.002/call; vendor claims 30% latency improvement. Useful as a dedicated "code editor / terminal" sub-channel.

Google Gemini 2.5 Computer Use API (Updated Cloud Fallback — Replaces Generic Cloud OCR)

Model ID: gemini-2.5-computer-use-preview-10-2025; built on Gemini 2.5 Pro; public preview via Gemini API (AI Studio / Vertex AI).

Why it is the new preferred cloud fallback: it is the only major-vendor service that exposes a clean, reusable coordinate interface. The computer_use tool returns function_call results in a loop: click_at(x, y), type_text_at, etc. Per Google AI for Developers official documentation (ai.google.dev, updated 2026-06-24): "The model outputs normalized coordinates (0–999) regardless of the input image dimensions... The recommended screen size is (1440, 900)"; click_at uses "x and y values based on a 1000×1000 grid and are scaled to the screen dimensions." This is a clean grounding/coordinate interface that can directly serve as an OCR/grounding backend fallback. Each step includes a safety_decision (ALLOWED / REQUIRES_CONFIRMATION). Project Mariner has been merged into this model.

Comparison to other computer-use products:

  • OpenAI Computer Use: Responses API computer tool (computer-use-preview, trained on GPT-5.4 per secondary sources). Recommend detail:"original" (retains up to 10.24 M pixels for click accuracy); after downsampling, coordinates must be remapped back to the original image space; recommended 1440×900 / 1600×900. Returns actions (click/type) rather than a reusable grounding coordinate — less suited as a pure coordinate backend. OSWorld historical baseline 38.1%.
  • Anthropic Claude Computer Use: tool-use paradigm (observe screen → move cursor → click/type); returns actions; latest models are Opus 4.x series. Not useful as a coordinate API; better suited when you need a full agent action loop.

C. UI Grounding / GUI Agent Models (Recommended as Clickable-Element Enhancement Layer)

GUI-Owl-1.5 / Mobile-Agent-v3.5 (Priority for Accuracy)

Paper: arXiv:2602.16855, "Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents"; 2026-02-15. Alibaba Tongyi Lab.

  • Based on Qwen3-VL; 2B/4B/8B/32B/235B variants (Instruct + Thinking); MIT license; weights fully open on HuggingFace (mPLUG organization); GitHub: X-PLUG/MobileAgent.
  • Trained with MRPO (multi-platform environment RL); vLLM deployment validated on A100 96 GB.
  • Self-reported ScreenSpot-Pro: 72.9% single-shot (claims to exceed Gemini-3-Pro's 72.7%), 80.3% two-stage crop-refine. Self-reported OSWorld-Verified: 56.5% (vs. UI-TARS-2 53.1%, OpenCUA-72B 45.0%).
  • Reliability note: These numbers come from the model's own paper (arXiv:2602.16855). Independent verification could only confirm the earlier GUI-Owl version scores (~54.9% for GUI-Owl-7B, ~58.0% for GUI-Owl-32B). Must self-verify before adopting. Use the official ScreenSpot-Pro dataset.
  • Coordinate output format: based on Qwen3-VL (0–1000 normalized), but GUI-Owl-1.5's exact convention is not confirmed from primary sources. Check the cookbook before integrating.

Holo3 / Holo3.1 (Priority for Low-Latency Self-Hosting)

H Company. Coordinate format is explicitly documented: [0, 1000] normalized, integer values, top-left origin — straightforward to map to Mac Retina logical points.

  • Holo3 (2026-03): Holo3-35B-A3B (Apache-2.0, Qwen3.5 MoE, 35B total / 3B active; vendor-reported OSWorld-Verified 77.8%); Holo3-122B-A10B (vendor-reported OSWorld-Verified 78.85%; paid API / research license — both figures exceed the human baseline of 72.36%).
  • Holo3.1 (2026-06): adds 0.8B/4B/9B/35B-A3B tiers; first release of quantized checkpoints (FP8 / Q4 GGUF / NVFP4 for NVIDIA Blackwell); native function-calling (in addition to existing structured JSON output); AndroidWorld 35B improved from 67%→79.3%. Small quantized models (4B/9B) target the "local-crop 100–300 ms" latency goal.

OmniParser v2 (Microsoft)

YOLOv8 (icon_detect; outputs [x1, y1, x2, y2] + confidence + interactability flag) + Florence-2 (icon_caption); integrates PaddleOCR for text boxes. Per Microsoft model card: OmniParser V2 + GPT-4o achieves 39.6% average accuracy on ScreenSpot-Pro (vs. GPT-4o alone at 0.8%); A100 0.6 s/frame. License warning: icon_detect inherits YOLOv8's AGPL-3.0 — closed-source commercial use is restricted; icon_caption is MIT. Avoid or replace the AGPL component for commercial products.

Qwen2.5-VL / Qwen3-VL

Native grounding; JSON output of bbox + point. Qwen2.5-VL uses absolute coordinates; Qwen3-VL switched to [0, 1000] normalized; open-source multi-tier; self-hostable via vLLM (note grounding accuracy regression bug in vLLM ≥0.11.1, GitHub #29595 — lock version and validate). Raw ScreenSpot-Pro for Qwen3-VL 4B/8B/30B-A3B ≈ 52.7–55.5%; boostable with crop/training methods.

UI-TARS (ByteDance)

  • UI-TARS-1.5-7B: open weights (Apache-2.0); absolute pixel coordinate output; trained specifically for GUI click/drag grounding; action space covers click/drag/hotkey/type/scroll; self-hostable via vLLM. ScreenSpot-Pro: ByteDance blog reports 61.6% (ScreenSpot-V2 94.2%, OSWorld 42.5%); third-party reproductions typically land at ~42–49% (regime difference; see Caveats).
  • UI-TARS-2 (2025-09-04, arXiv:2509.02544): W4A8 quantization throughput 29.6→47 tok/s; per-round latency 4.0→2.5 s; OSWorld 47.5→44.4. Open-source status of weights: not confirmed as of June 2026. Only UI-TARS-1.5-7B weights are confirmed open (Apache-2.0).

Other Grounding Models (GPU Self-Hosting Required)

GTA1-7B/32B, UI-Venus-7B/72B, Holo1.5-7B, Holo2-30B-A3B, MAI-UI, CogAgent, SeeClick, Ferret-UI, ShowUI, Aria-UI, UGround, OS-Atlas — all grounding-specialist models on the ScreenSpot-Pro leaderboard, all requiring GPU self-hosting. The ScreenSpot-Pro scores in the table in Key Findings provide the latest rankings.

D. macOS AX Tree Fusion + Retina Coordinate Mapping

Retina / Coordinate Mapping

macOS screenshots are rendered at Retina resolution: a selection of 100×100 logical points produces a 200×200 pixel PNG, with the scale factor provided by NSScreen.backingScaleFactor / NSWindow.backingScaleFactor (typically 2.0). OCR returns screenshot pixel coordinates; clicks require logical points. The service must therefore return: original image width/height (pixels), scale_factor, and the local image's origin offset relative to the full screen or window, so the client can compute logical_point = origin + pixel_coordinate / scale_factor.

macOS Vision API on-device: VNRecognizeTextRequest returns VNRecognizedTextObservation.boundingBox in normalized, bottom-left-origin coordinates. These must be Y-flipped and multiplied by the image dimensions. With WWDC26, the Foundation Models framework can now call Vision's OCR tools directly on-device — useful for a client-side quick pre-filter on pure text regions (menu bar, path bar, file names) before committing to a cloud round-trip.

AX Tree Fusion

The macOS Accessibility (AXUIElement) API provides screen logical points with a top-left origin. Normalize both sources to "window-top-left-origin logical points," then merge: AX provides control semantics (role / enabled state / value); OCR provides the actually rendered text. Rules:

  • When an AX element has no text or is a custom-drawn control (terminal, code editor, Canvas), trust OCR coordinates.
  • When an AX element has explicit bounds, trust AX.
  • Deduplicate using IoU threshold before merging.

macOS 27 UI Style Warning

macOS 27 (previewed 2026-05-19) is reported to adopt visionOS-style translucent / frosted-glass windows with material effects that shift with cursor position. Higher translucency and frosted-glass backgrounds significantly increase OCR and grounding difficulty (semantic transparency, low-contrast text). Cloud services should apply contrast enhancement / de-transparency preprocessing for this UI style. Note: some macOS 27 UI-change specifics come from non-official media sources; treat as expected but not fully confirmed.

Recommendations — Phased Rollout

Phase 1 (Immediate, Low Risk, Highest Return)

  1. Upgrade the base OCR engine from PP-OCRv5/RapidOCR to PP-OCRv6 medium (A100 0.13 s; satisfies full-screen <1 s; local-crop can reach 100–300 ms). Retain the two-stage detection + recognition pipeline to obtain stable text boxes + confidence — aligned with the "coordinate accuracy first" goal. The RapidOCR channel can continue as ONNX lightweight backup pending its official v6 support.

  2. Deploy on a GPU host (A100 / RTX 4090 / or TensorRT-optimized consumer card); wrap with Rust (axum / actix) or equivalent as REST + gRPC.

  3. API contract: accept whole image / window-cropped image / rectangular region; carry origin (offset of the region within the full screen or window) and scale_factor. Return a stable JSON envelope:

    {
  "image_w": <int>, "image_h": <int>,
  "scale_factor": <float>,
  "origin": {"x": <float>, "y": <float>},
  "elapsed_ms": <int>,
  "model_version": "<string>",
  "lang": "<string>",
  "lines": [
    {
      "text": "<string>",
      "box": {"x": <float>, "y": <float>, "w": <float>, "h": <float>},
      "poly": [[x1,y1],[x2,y2],[x3,y3],[x4,y4]],
      "confidence": <float>
    }
  ],
  "words": [...],
  "low_confidence": <bool>,
  "empty": <bool>
}

    Sort lines stably top-to-bottom, left-to-right.

  4. Add a client-side Apple Vision VNRecognizeTextRequest pre-filter for pure text regions (menu bar, path bar, file names): obtain box + confidence on-device, skip the cloud round-trip entirely, reduce latency and cost.

  5. Latency threshold: local window image (<1024 px side) target <300 ms; full-screen 2K–4K image target <1 s. If not met with GPU, enable TensorRT FP16 or reduce the detection input resolution.

Phase 2 — Grounding Enhancement Layer + AX Fusion

  1. Introduce GUI-Owl-1.5 (MIT, open-source) as the primary grounding model for icons and complex controls, replacing UI-TARS-1.5-7B. For high-DPI Mac screens with small targets, use a two-stage "coarse localization then crop-and-refine" approach. Before integrating: (a) self-verify its 72.9/80.3 self-reported ScreenSpot-Pro scores on the official dataset; (b) confirm its coordinate convention (absolute pixel vs. 0–1000 normalized).

  2. For latency-sensitive paths, use Holo3.1 4B/9B + FP8/NVFP4 quantization (coordinates explicitly 0–1000 normalized, native function-calling/JSON, straightforward Retina mapping).

  3. On the client side (Rust / Swift), implement OCR × AX fusion: unify coordinate systems → IoU deduplication → AX provides semantics, OCR provides text.

Phase 3 — Cloud Fallback, Cache, and Observability

  1. Replace generic cloud OCR with Gemini 2.5 Computer Use API as the cloud fallback: the only major-vendor service exposing a clean coordinate interface (1000×1000 grid, 0–999 normalized, per-step safety decisions). Reserve OpenAI / Anthropic computer use only when a full agent action loop is required.

  2. For remaining hard samples that need text extraction specifically (low confidence, empty results, unusual languages), fall back to Google Cloud Vision or Baidu High-Precision with Position — these return whole-image pixel coordinates that require origin + scale metadata to localize.

  3. Add caching: cache OCR results keyed by window content hash; run incremental OCR only over changed regions.

  4. Establish a benchmark regression suite: use ScreenSpot-Pro (high-DPI small targets) + OSWorld-G/Jedi (32 element classes, includes refusal) + a self-built set of CJK-Latin mixed / dark-mode / translucent / small-font / terminal / code-editor samples. Gate model swaps on coordinate hit-rate thresholds.

Signals That Should Trigger Reassessment

  • If a self-hosted open-source model stably exceeds 75% ScreenSpot-Pro single-shot (no crop) on your own test set, it should replace the current grounding primary.
  • If PP-OCRv6 recognition rate on your CJK/small-font set falls below a 1B-class VLM-OCR (e.g., HunyuanOCR line-level coordinates), route that hard-sample class to the VLM-OCR sub-channel.
  • If macOS 27 translucent UI drops coordinate hit rate by more than 5 percentage points, add a contrast-enhancement / de-transparency preprocessing stage.
  • If local-window OCR p95 > 300 ms after TensorRT optimization → upgrade GPU or reduce input resolution.
  • If CJK small-font accuracy is insufficient → switch to Baidu High-Precision with Position, or fine-tune PP-OCRv6.
  • If "icon / custom-drawn control click" is the primary failure mode → promote the grounding model to the primary path.

Caveats — Important Uncertainties

Benchmark regime differences: ScreenSpot-Pro single-shot grounding versus zoom-in / agentic pipeline can differ by 5–15 percentage points. This is the root cause of the gap between UI-TARS-1.5 ByteDance-reported 61.6% and third-party reproductions of ~42–49%. Always note the evaluation regime when citing numbers. Third-party aggregator sites (e.g., llm-stats listing Claude Opus at 87.9%) have opaque methodologies — cite with caution.

Vendor-self-reported benchmarks: PP-OCRv6 accuracy/speed figures, GUI-Owl-1.5 ScreenSpot-Pro 72.9/80.3, Holo3 OSWorld-Verified 77.8/78.85, and Gemini 2.5 Computer Use coordinate API behavior are all taken from first-party sources (arXiv papers, official GitHub/HuggingFace model cards, vendor documentation). PP-OCRv6 speed/accuracy and Gemini 0–999 coordinates and UI-TARS-2 scores have been independently cross-checked. GUI-Owl-1.5 ScreenSpot-Pro scores come solely from its own paper; independent verification could only confirm earlier GUI-Owl version scores (lower). Validate all vendor-self-reported numbers on your own screen sample set before committing.

Qwen3-VL grounding accuracy regression bug in vLLM ≥0.11.1 (GitHub vllm #29595); llama.cpp also has coordinate errors for non-square images. Lock versions and validate when self-hosting.

UI-TARS-2 open weights not confirmed as of June 2026. Only UI-TARS-1.5-7B weights are confirmed open (Apache-2.0). There is no UI-TARS-3.

OmniParser icon_detect is AGPL-3.0 — closed-source commercial use is restricted. Surya is GPL. Replace or segregate these components for commercial products.

Cloud API latency includes network round-trips and cannot reliably reach 100–300 ms. All cloud APIs return whole-image pixel coordinates; you must supply origin / scale metadata yourself.

VLM coordinate offset and hallucination are real risks. For your constraint — "misclicked coordinate is worse than a misread character" — dedicated OCR must remain the coordinate source of truth; VLMs and grounding models are semantic / icon supplements only.

Rapidly evolving landscape: PP-OCRv6, GUI-Owl-1.5, GLM-OCR, FireRed-OCR, Holo3/3.1, Qwen3-VL, and Gemini 2.5 Computer Use are all recent releases (2025 H2 – 2026 Q2). Run your own screen-sample benchmark suite — real mac windows, menu bar, terminal, dark mode, translucent/frosted-glass windows, CJK-Latin mixed small-font — before any model swap.

Leaderboard aggregators (coasty.ai, llm-stats, benchlm, codesota, instavar, ofox, decodethefuture, etc.) appear to contain AI-generated SEO content with contradictory numbers (e.g., OSWorld first place ranging 79%–87.9%). These have been excluded or flagged as suspect; do not cite them as authoritative.

ScreenSpot-Pro and OSWorld-Verified official leaderboard tables (gui-agent.github.io, os-world.github.io, updated 2026-06-22) use JavaScript dynamic rendering; current rankings cannot be directly extracted from page HTML. Model scores in this document are taken from each model's first-party paper or model card and may differ slightly from real-time leaderboard rankings.

Frontier model version numbers (GPT-5.4/5.5, Claude "Opus 4.8/Fable 5", Gemini "3.1") come largely from secondary sources and have not been confirmed by first-party announcements; treat as unverified.

macOS 27 UI translucency change details come partly from non-official media; treat as expected but not fully confirmed until official release.

This document merges the original research report (2025 baseline) with the 2026 supplement update. Where the supplement supersedes an earlier recommendation — PP-OCRv5 → PP-OCRv6, generic cloud OCR → Gemini 2.5 Computer Use, UI-TARS-1.5 → GUI-Owl-1.5 / Holo3.1 — the current recommendation is presented as the primary, with the prior noted for context.

Helper Disconnected