Agent文件任务输出协议MAIGroundingAgentsrc/mai_grounding_agent.pyUI 元素定位单步grounding_think./grounding_think{coordinate:[x,y]}坐标基于 SCALE_FACT0R999 归一化MAIUINavigationAgentsrc/mai_navigation_agent.py多步移动端GUI导航支持ask_user与mcp_call.tool_call{json}/tool_call多轮带历史截图0x01 工程实现特色MAI-UI 工程实现的三个特色如下。1.1 特色1特色1三套系统提示词对应三种Agent形态grounding / 纯导航 / ask_user MCP 增强导航src/prompt.py同时维护MAI_MOBILE_SYS_PROMPT_GROUNDING 一 单步元素定位MAI_MOBILE_SYS_PROMPT 一 标准多步导航MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP 一 在导航动作集里叠加两个特殊工具ask_userquestion模型主动反问用户、把任务“打回去mcp_calltoolargs调外部MCP工具如高德导航补全设备端做不到的能力意义这是Agent-User Interaction MCP Augmentation 范式在代码层面的真实落点一不是新API、就是同一个模型在不同system prompt下解锁不同动作集。新增交互类工具的正确姿势就是改prompt.pyparse_tagged_text的 schema而不是另起一个Agent类。1.2 特色2特色2是归一化坐标空间SCALE_FACTOR 999 XML标签输出协议而非function-calling。src/mai_grounding_agent.py 与 src/mai_naivigation_agent.py 都硬编码为 SCALE_FACTOR999模型永远输出[0999]区间整数由客户端按当前截图WH反归一化。输出不是OpenAI function-calling而是裸文本里的 XML 标签Grounding:grounding_think.../grounding_think{coordinate:[x,y]}Navigation:.tool_call{json}/tool_call兼容 thinking 模型的解析器parse_grounding_response、parse_tagged_text错误统一抛 ValueError。意义跨分辨率泛化同一个模型同一个权重无缝服务任意手机分辨率不需要在 prompt里写屏幕尺寸协议无关于推理后端一VLLM0.11.0、HFtransformers 本地推理、DashScope都能用因为只解析纯文本不依赖任何后端的tool-call结构代价解析鲁棒性必须由客户端自己保证所以两个parser都做了容错显式异常1.3 特色 3特色 3无状态服务端 客户端自管TrajMemory每步把历史截图重塞回 messagesBaseAgent 持有 traj_memoryTrajMemory每个 TrajStep 同时存 screenshot: Image 和 screenshot_bytesbytes渲染vs序列化双用MAIUINaivigationAgent._build_messages() 按 runtime_conf[history_n] 把最近 N 步的“截图模型回复“重组成多轮user/assistant对话再发给vLLM一一一vLLM 端零会话状态。save_traj()/load_traj()走bytes可被序列化/回放/做评测离线分析。stept的请求体每步独立、无状态如下意义可回放、可评测、可断点续跑—save_traj出dict、load_traj直接灌回离线replay不需要真机/模拟器横向扩展友好一VLLM可以集群水平扩因为没有会话粘性这正契合 scaling parallel environments up to 512的训l练形态在推理侧的对应做法代价每步N张图都要重传带宽与 prefill 成本随 history_n线性增长调小 history_n是常见的省 token 技巧。1.4 小结MAI-UI的工程独到之处不是模型本身而是这套客户端契约分辨率无关的999坐标空间 XML标签协议与后端解耦 无状态多轮重放与历史长度解耦 三档 prompt解锁的grounding/导航/ask_userMCP三种形态一一一后续任何二次开发都沿着这四条线走而不是去改模型契约。0x02 提示词2.1 提示词代码以下是提示词代码。MAI_MOBILE_SYS_PROMPTMAI_MOBILE_SYS_PROMPT You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task. ## Output Format For each function call, return the thinking process in thinking /thinking tags, and a json object with function name and arguments within tool_call/tool_call XML tags: thinking ... /thinking tool_call {name: mobile_use, arguments: args-json-object} /tool_call ## Action Space {action: click, coordinate: [x, y]} {action: long_press, coordinate: [x, y]} {action: type, text: } {action: swipe, direction: up or down or left or right, coordinate: [x, y]} # coordinate is optional. Use the coordinate if you want to swipe a specific UI element. {action: open, text: app_name} {action: drag, start_coordinate: [x1, y1], end_coordinate: [x2, y2]} {action: system_button, button: button_name} # Options: back, home, menu, enter {action: wait} {action: terminate, status: success or fail} {action: answer, text: xxx} # Use escape characters \\, \\, and \\n in text part to ensure we can parse the text in normal python string format. ## Note - Write a small plan and finally summarize your next action (with its target element) in one sentence in thinking/thinking part. - Available Apps: [Camera,Chrome,Clock,Contacts,Dialer,Files,Settings,Markor,Tasks,Simple Draw Pro,Simple Gallery Pro,Simple SMS Messenger,Audio Recorder,Pro Expense,Broccoli APP,OSMand,VLC,Joplin,Retro Music,OpenTracks,Simple Calendar Pro]. You should use the open action to open the app as possible as you can, because it is the fast way to open the app. - You must follow the Action Space strictly, and return the correct json object within thinking /thinking and tool_call/tool_call XML tags. .strip()MAI_MOBILE_SYS_PROMPT_NO_THINKINGMAI_MOBILE_SYS_PROMPT_NO_THINKING You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task. ## Output Format For each function call, return a json object with function name and arguments within tool_call/tool_call XML tags: tool_call {name: mobile_use, arguments: args-json-object} /tool_call ## Action Space {action: click, coordinate: [x, y]} {action: long_press, coordinate: [x, y]} {action: type, text: } {action: swipe, direction: up or down or left or right, coordinate: [x, y]} # coordinate is optional. Use the coordinate if you want to swipe a specific UI element. {action: open, text: app_name} {action: drag, start_coordinate: [x1, y1], end_coordinate: [x2, y2]} {action: system_button, button: button_name} # Options: back, home, menu, enter {action: wait} {action: terminate, status: success or fail} {action: answer, text: xxx} # Use escape characters \\, \\, and \\n in text part to ensure we can parse the text in normal python string format. ## Note - Available Apps: [Camera,Chrome,Clock,Contacts,Dialer,Files,Settings,Markor,Tasks,Simple Draw Pro,Simple Gallery Pro,Simple SMS Messenger,Audio Recorder,Pro Expense,Broccoli APP,OSMand,VLC,Joplin,Retro Music,OpenTracks,Simple Calendar Pro]. You should use the open action to open the app as possible as you can, because it is the fast way to open the app. - You must follow the Action Space strictly, and return the correct json object within thinking /thinking and tool_call/tool_call XML tags. .strip()MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP# Placeholder prompts for future features MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP Template( You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task. ## Output Format For each function call, return the thinking process in thinking /thinking tags, and a json object with function name and arguments within tool_call/tool_call XML tags: thinking ... /thinking tool_call {name: mobile_use, arguments: args-json-object} /tool_call ## Action Space {action: click, coordinate: [x, y]} {action: long_press, coordinate: [x, y]} {action: type, text: } {action: swipe, direction: up or down or left or right, coordinate: [x, y]} # coordinate is optional. Use the coordinate if you want to swipe a specific UI element. {action: open, text: app_name} {action: drag, start_coordinate: [x1, y1], end_coordinate: [x2, y2]} {action: system_button, button: button_name} # Options: back, home, menu, enter {action: wait} {action: terminate, status: success or fail} {action: answer, text: xxx} # Use escape characters \\, \\, and \\n in text part to ensure we can parse the text in normal python string format. {action: ask_user, text: xxx} # you can ask user for more information to complete the task. {action: double_click, coordinate: [x, y]} {% if tools -%} ## MCP Tools You are also provided with MCP tools, you can use them to complete the task. {{ tools }} If you want to use MCP tools, you must output as the following format: thinking ... /thinking tool_call {name: function-name, arguments: args-json-object} /tool_call {% endif -%} ## Note - Available Apps: [Contacts, Settings, Clock, Maps, Chrome, Calendar, files, Gallery, Taodian, Mattermost, Mastodon, Mail, SMS, Camera]. - Write a small plan and finally summarize your next action (with its target element) in one sentence in thinking/thinking part. .strip() )MAI_MOBILE_SYS_PROMPT_GROUNDINGMAI_MOBILE_SYS_PROMPT_GROUNDING You are a GUI grounding agent. ## Task Given a screenshot and the users grounding instruction. Your task is to accurately locate a UI element based on the users instructions. First, you should carefully examine the screenshot and analyze the users instructions, translate the users instruction into a effective reasoning process, and then provide the final coordinate. ## Output Format Return a json object with a reasoning process in grounding_think/grounding_think tags, a [x,y] format coordinate within answer/answer XML tags: grounding_think.../grounding_think answer {coordinate: [x,y]} /answer .strip()2.2 移动系统提示词差异一览只有MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP模板支持 MCP 工具集成且通过 Jinja2 条件语法实现动态插入其余提示词版本均不包含 MCP 功能。提示词 ID核心用途思考标签操作空间特殊功能MAI_MOBILE_SYS_PROMPT标准 GUI 代理 必须点击/长按/输入/滑动等全功能无MAI_MOBILE_SYS_PROMPT_NO_THINKING快速响应无思考标签同上省略思考直接返回 JSONMAI_MOBILE_SYS_PROMPT_ASK_USER_MCP模板化用户询问可选同上ask_user、double_click、Jinja2 模板、MCP 工具集成MAI_MOBILE_SYS_PROMPT_GROUNDING纯定位专用仅元素识别输出 [x,y] 坐标无操作命令2.3 工具集成差异MCP 功能只在MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP模板层集成其余版本需外部桥接。集成位置仅MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP内置 MCP 工具调用入口通过 Jinja2 模板动态注入。其余版本无 MCP 工具入口需外部调用。提示词层差异标准版无 MCP 占位符纯 JSON 输出。MCP 版模板内预留{{mcp_tools}}变量运行时注入具体工具描述。运行时差异标准版LLM 输出传统动作 JSON由外部框架手动转发至 MCP。MCP 版渲染后提示词包含完整 MCP 工具 JSONLLM 可直接调用。条件性集成仅 MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP使用 Jinja2 模板语法{%if tools -%}...{%endif -%}实现动态集成独立## MCP Tools区域存放 MCP 工具描述通过{{tools}}变量动态插入可用工具信息输出格式与标准移动操作不同 内直接嵌入 MCP 函数调用0x03 输出3.1 输出格式区别非 MCP 版本MAI_MOBILE_SYS_PROMPT统一格式所有操作通过mobile_use函数调用固定结构GUI 操作封装在arguments字段示例thinking.../thinking tool_call {name:mobile_use,arguments:args-json-object} /tool_callMCP 版本MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP双重格式支持标准 GUI 操作和 MCP 工具调用工具特定格式MCP 工具调用使用实际函数名作为name示例thinking.../thinking tool_call {name:function-name,arguments:args-json-object} /tool_call下面代码把LLM的输出转换为结构化输出def parse_action_to_structure_output(text: str) - Dict[str, Any]: Parse model output text into structured action format. Args: text: Raw model output containing thinking and tool_call tags. Returns: Dictionary with keys: - thinking: The models reasoning process - action_json: Parsed action with normalized coordinates Note: Coordinates are normalized to [0, 1] range by dividing by SCALE_FACTOR. text text.strip() results parse_tagged_text(text) thinking results[thinking] tool_call results[tool_call] action tool_call[arguments] # Normalize coordinates from SCALE_FACTOR range to [0, 1] if coordinate in action: coordinates action[coordinate] if len(coordinates) 2: point_x, point_y coordinates elif len(coordinates) 4: x1, y1, x2, y2 coordinates point_x (x1 x2) / 2 point_y (y1 y2) / 2 else: raise ValueError( fInvalid coordinate format: expected 2 or 4 values, got {len(coordinates)} ) point_x point_x / SCALE_FACTOR point_y point_y / SCALE_FACTOR action[coordinate] [point_x, point_y] return { thinking: thinking, action_json: action, }3.2 功能范围区别非 MCP 版本有限操作集仅预定义 GUI 操作点击、滑动、输入等移动设备专属专注触摸屏界面交互固定动作空间无法扩展新操作类型MCP 版本扩展操作集除 GUI 操作外支持 MCP 工具系统级功能可通过 MCP 工具执行复杂系统操作动态功能依据配置工具动态扩展功能范围3.3 实际应用场景标准 GUI 操作MCP 版本中标准 GUI 操作仍使用 mobile_use 函数与非 MCP 版本行为基本相同MCP 工具调用需执行 MCP 工具时使用工具名称作为函数名可执行复杂任务系统配置、数据处理等代码实现中的处理在 MAIUIMobileAgent 类中若 self.tools 非空使用 MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP 模板通过 render(toolstools_str) 将工具列表注入提示词未配置工具时回退到标准 MAI_MOBILE_SYS_PROMPT。代码如下property def system_prompt(self) - str: Generate the system prompt based on available MCP tools. Returns: System prompt string, with MCP tools section if tools are configured. if self.tools: tools_str \n.join( [json.dumps(tool, ensure_asciiFalse) for tool in self.tools] ) return MAI_MOBILE_SYS_PROMPT_ASK_USER_MCP.render(toolstools_str) return MAI_MOBILE_SYS_PROMPTMCP 版本提供更灵活的操作能力允许智能体在标准 GUI 操作与 MCP 工具间切换从而执行更复杂任务非 MCP 版本则专注纯粹移动界面操作。0x04 MAIUINaivigationAgentMAIUINaivigationAgent移动端 GUI 导航智能体是整个 MAI-GUI 智能体的 “底座模块”—— 它封装了 LLM 初始化、历史界面上下文管理、多模态消息构建等核心能力专门为移动端 GUI 自动化场景设计能基于任务指令和多步历史界面截图构建标准化的多模态消息发送给 LLM为后续动作生成提供统一的输入基础。4.1 核心特色MAIUINaivigationAgent 的核心逻辑如下初始化配置 LLM 客户端→ 图片预处理历史 当前截图统一格式→ 消息构建按固定结构拼接多模态内容全流程为 LLM 提供标准化、结构化的输入。特色维度具体说明历史上下文智能管理支持配置history_n参数默认 3自动截取最近 N 步的界面截图作为历史上下文既保留关键操作轨迹又避免上下文过长导致 LLM 推理效率下降仅加载history_n-1条历史截图 当前截图精准控制上下文长度多格式图片兼容处理_prepare_images方法支持字节流、PIL Image 等多种图片输入格式自动转换为 RGB 格式的 PIL Image解决不同来源截图的格式兼容问题适配移动端截图的多样化场景MCP 工具集成能力初始化时支持传入 MCP 工具列表为后续 LLM 调用 MCP 工具如执行设备操作预留扩展接口兼容 MCP 协议生态标准化多模态消息构建_build_messages方法按 “系统提示词→用户指令→历史截图 历史响应→当前截图” 的固定逻辑构建消息严格对齐 LLM 多模态输入格式确保不同历史长度下消息结构统一高度可配置化支持自定义温度、top_k、top_p 等 LLM 推理参数以及历史上下文长度history_n可根据不同移动端任务如简单点击 / 复杂表单填写调整配置4.2 定义MAIUINaivigationAgent 的定义如下。class MAIUINaivigationAgent(BaseAgent): Mobile automation agent using vision-language models. This agent processes screenshots and natural language instructions to generate GUI actions for mobile device automation. Attributes: llm_base_url: Base URL for the LLM API endpoint. model_name: Name of the model to use for predictions. runtime_conf: Configuration dictionary for runtime parameters. history_n: Number of history steps to include in context. def __init__( self, llm_base_url: str, model_name: str, runtime_conf: Optional[Dict[str, Any]] None, tools: Optional[List[Dict[str, Any]]] None, ): Initialize the MAIMobileAgent. Args: llm_base_url: Base URL for the LLM API endpoint. model_name: Name of the model to use. runtime_conf: Optional configuration dictionary with keys: - history_n: Number of history images to include (default: 3) - max_pixels: Maximum pixels for image processing - min_pixels: Minimum pixels for image processing - temperature: Sampling temperature (default: 0.0) - top_k: Top-k sampling parameter (default: -1) - top_p: Top-p sampling parameter (default: 1.0) - max_tokens: Maximum tokens in response (default: 2048) tools: Optional list of MCP tool definitions. Each tool should be a dict with name, description, and parameters keys. super().__init__() # Store MCP tools self.tools tools or [] # Set default configuration default_conf { history_n: 3, temperature: 0.0, top_k: -1, top_p: 1.0, max_tokens: 2048, } self.runtime_conf {**default_conf, **(runtime_conf or {})} self.llm_base_url llm_base_url self.model_name model_name self.llm OpenAI( base_urlself.llm_base_url, api_keyempty, ) # Extract frequently used config values self.temperature self.runtime_conf[temperature] self.top_k self.runtime_conf[top_k] self.top_p self.runtime_conf[top_p] self.max_tokens self.runtime_conf[max_tokens] self.history_n self.runtime_conf[history_n]4.3 构建图像_prepare_images 函数被用来构建图像。def _prepare_images(self, screenshot_bytes: bytes) - List[Image.Image]: Prepare image list including history and current screenshot. Args: screenshot_bytes: Current screenshot as bytes. Returns: List of PIL Images (history current). # Calculate how many history images to include if len(self.history_images) 0: max_history min(len(self.history_images), self.history_n - 1) recent_history self.history_images[-max_history:] if max_history 0 else [] else: recent_history [] # Add current image bytes recent_history.append(screenshot_bytes) # Normalize input type if isinstance(recent_history, bytes): recent_history [recent_history] elif isinstance(recent_history, np.ndarray): recent_history list(recent_history) elif not isinstance(recent_history, list): raise TypeError(fUnidentified images type: {type(recent_history)}) # Convert all images to PIL format images [] for image in recent_history: if isinstance(image, bytes): image Image.open(BytesIO(image)) elif isinstance(image, Image.Image): pass else: raise TypeError(fExpected bytes or PIL Image, got {type(image)}) if image.mode ! RGB: image image.convert(RGB) images.append(image) return images4.4 构建文字def _build_messages( self, instruction: str, images: List[Image.Image], ) - List[Dict[str, Any]]: Build the message list for the LLM API call. Args: instruction: Task instruction from user. images: List of prepared images. Returns: List of message dictionaries for the API. messages [ { role: system, content: [{type: text, text: self.system_prompt}], }, { role: user, content: [{type: text, text: instruction}], }, ] image_num 0 history_responses self.history_responses if len(history_responses) 0: for history_idx, history_response in enumerate(history_responses): # Only include images for recent history (last history_n responses) if history_idx self.history_n len(history_responses): # Add image before the assistant response if image_num len(images) - 1: cur_image images[image_num] encoded_string pil_to_base64(cur_image) messages.append({ role: user, content: [{ type: image_url, image_url: {url: fdata:image/png;base64,{encoded_string}}, }], }) image_num 1 messages.append({ role: assistant, content: [{type: text, text: history_response}], }) # Add current image (last one in images list) if image_num len(images): cur_image images[image_num] encoded_string pil_to_base64(cur_image) messages.append({ role: user, content: [{ type: image_url, image_url: {url: fdata:image/png;base64,{encoded_string}}, }], }) else: # No history, just add the current image cur_image images[0] encoded_string pil_to_base64(cur_image) messages.append({ role: user, content: [{ type: image_url, image_url: {url: fdata:image/png;base64,{encoded_string}}, }], }) return messages4.5 流程MAIUINaivigationAgent 多步循环流程图如下特殊动作ask_user(question) →暂停把问题返还给用户设备-云协同里的用户交互mcp_call(toolargs) →调用外部MCP工具如高德地图导航finish() 任务结束也参见如下4.6 推理核心作用predict 是 MAI-GUI 智能体的核心决策与动作生成模块是 GUI Agent 的 “决策大脑”核心解决 “根据任务指令和当前界面状态生成下一步具体 GUI 动作” 的问题区别于单纯的元素定位模块。predict 的核心功能是接收任务指令如 “完成 APP 登录”和当前界面观测信息截图 可选的无障碍树通过调用大语言模型生成并解析出下一步要执行的结构化 GUI 动作如点击、滑动、输入等同时记录完整的任务轨迹Trajectory是 GUI Agent 实现 “根据界面状态决策操作” 的核心环节。predict 的流程闭环是输入处理→消息构建→LLM 调用→响应解析→轨迹记录→结果输出全流程覆盖异常处理确保动作生成的稳定性。核心特色特色维度具体说明任务轨迹全链路记录内置traj_memory轨迹记忆模块每一步操作都会存储截图、模型响应、解析后的动作、推理过程等全量信息支持任务溯源、调试和复盘多维度界面观测输入同时接收截图视觉信息和无障碍树结构化 UI 信息相比纯视觉输入更精准理解界面结构适配复杂 GUI 场景鲁棒的 LLM 调用与解析① 内置 3 次 API 重试机制捕获并打印异常栈信息提升调用稳定性② 标准化解析模型响应为thinking推理过程action_json结构化动作确保输出格式统一任务目标持久化首次调用时将任务指令存入轨迹记忆作为持久化目标避免后续步骤丢失核心任务方向日志可视化友好对包含图片的消息做脱敏打印mask_image_urls_for_logging既保留日志完整性又避免 Base64 编码刷屏便于调试流程predict 的流程如下时序图时序图用户 ⇔ Agent ⇔ vLLM (Navigation 场景)如下要点每步都把历史 history_n张截图重新塞进 messages无服务端会话状态vLLM是无状态的 chat completionsask_user/mcp_call 是模型直接吐出的tool_call调度由外层环境完成agent 本身不做副作用日志路径上的 base64 图片一定经过 mask_image_urls_for_logging 替换为 [IMAGE_DATA]。代码def predict( self, instruction: str, obs: Dict[str, Any], **kwargs: Any, ) - Tuple[str, Dict[str, Any]]: Predict the next action based on the current observation. Args: instruction: Task instruction/goal. obs: Current observation containing: - screenshot: PIL Image or bytes of current screen - accessibility_tree: Optional accessibility tree data **kwargs: Additional arguments including: - extra_info: Optional extra context string Returns: Tuple of (prediction_text, action_dict) where: - prediction_text: Raw model response or error message - action_dict: Parsed action dictionary # Set task goal if not already set if not self.traj_memory.task_goal: self.traj_memory.task_goal instruction # Process screenshot screenshot_pil obs[screenshot] screenshot_bytes safe_pil_to_bytes(screenshot_pil) # Prepare images images self._prepare_images(screenshot_bytes) # Build messages messages self._build_messages(instruction, images) # Make API call with retry logic max_retries 3 prediction None action_json None for attempt in range(max_retries): try: messages_print mask_image_urls_for_logging(messages) print(fMessages (attempt {attempt 1}):\n{messages_print}) response self.llm.chat.completions.create( modelself.model_name, messagesmessages, max_tokensself.max_tokens, temperatureself.temperature, top_pself.top_p, frequency_penalty0.0, presence_penalty0.0, extra_body{repetition_penalty: 1.0, top_k: self.top_k}, seed42, ) prediction response.choices[0].message.content.strip() print(fRaw response:\n{prediction}) # Parse response parsed_response parse_action_to_structure_output(prediction) thinking parsed_response[thinking] action_json parsed_response[action_json] print(fParsed response:\n{parsed_response}) break except Exception as e: print(fError on attempt {attempt 1}: {e}) traceback.print_exc() prediction None action_json None # Return error if all retries failed if prediction is None or action_json is None: print(Max retry attempts reached, returning error flag.) return llm client error, {action: None} # Create and store trajectory step traj_step TrajStep( screenshotscreenshot_pil, accessibility_treeobs.get(accessibility_tree), predictionprediction, actionaction_json, conclusion, thoughtthinking, step_indexlen(self.traj_memory.steps), agent_typeMAIMobileAgent, model_nameself.model_name, screenshot_bytesscreenshot_bytes, structured_action{action_json: action_json}, ) self.traj_memory.steps.append(traj_step) return prediction, action_json4.7 轨迹TrajMemory / TrajStep 数据结构图