DeepAgents-AutoGLM

🥈Silver

@Illuminated2020

DeepAgents-AutoGLM integrates Open-AutoGLM's Android and iOS GUI automation into DeepAgents-CLI via LangChain Middleware. This enables LLM orchestration with vision-guided GUI control for operations teams. It connects to mobile devices for automated testing, process automation, and quality assurance workflows.

116100Updated 1mo ago

Intermediate30min to implementautomation

Saves ~270 min per use

Quick InstallView Source

git clone https://github.com/Illuminated2020/DeepAgents-AutoGLM.git

Works with:

Claude

Overview

About This Skill

How to Use

[{"step":"Set up the device connection","action":"Connect your mobile device via USB/ADB/WiFi and verify it's detected using `adb devices` (Android) or `idevice_id -l` (iOS). Ensure USB debugging is enabled on the device.","tip":"Use `adb tcpip 5555` to switch to WiFi mode if testing remotely. For iOS, pair the device via Xcode first."},{"step":"Install DeepAgents-AutoGLM and dependencies","action":"Install the package via `pip install deepagents-autoglm` and ensure LangChain is configured with Open-AutoGLM middleware. Test the connection with `python -m deepagents_autoglm.test_connection`.","tip":"Check the [DeepAgents-AutoGLM GitHub](https://github.com/DeepAgents/DeepAgents-AutoGLM) for platform-specific setup guides."},{"step":"Define the automation task","action":"Use the prompt template to describe the task, steps, and constraints. Include device-specific details like screen resolution and OS version for accurate GUI interactions.","tip":"Use tools like `adb shell wm size` to get screen dimensions. For iOS, use `xcrun simctl io booted screen` for simulator dimensions."},{"step":"Execute and monitor the automation","action":"Run the automation via the LangChain agent. Monitor progress in real-time using the middleware logs. Capture screenshots at each step for validation.","tip":"Enable `vision_enabled=True` in the middleware config to allow the LLM to 'see' the screen and adjust actions dynamically."},{"step":"Validate and iterate","action":"Review the automation report for errors or deviations. Adjust coordinates, delays, or constraints based on the output. Re-run with updated parameters if needed.","tip":"Use the `error_handling` parameter in the middleware config to set retry logic or fallback actions (e.g., 'retry_on_failure', 'skip_step')."}]

Setup & Installation

Quick Install

No install command available. Check the GitHub repository for manual installation instructions.

Alternative Install (Git Clone)

git clone https://github.com/Illuminated2020/DeepAgents-AutoGLM

Requirements

Claude Code or compatible AI agent
Works with: Claude

Quick Start Guide

Install the Skill

Copy the install command above and run it in your terminal.

Open Your AI Agent

Launch Claude Code, Cursor, or your preferred AI coding agent.

Try It Out

Use the prompt template or examples below to test the skill.

Customize

Adapt the skill to your specific use case and workflow.

Usage Examples

Prompt Template

Use DeepAgents-AutoGLM to automate [TASK] on a [ANDROID/iOS] device connected via [USB/ADB/WiFi]. The task involves: [DESCRIBE_STEPS]. Ensure the automation follows these constraints: [CONSTRAINTS]. Provide the sequence of GUI interactions, validation checks, and any error handling required. Include the LangChain middleware configuration for Open-AutoGLM integration.

Example Output

### DeepAgents-AutoGLM Automation Report
**Task:** Automate the checkout process for a retail app on an Android device (Pixel 7, Android 14).
**Steps:**
1. Launch the app via `adb shell am start -n com.retail.app/.MainActivity`.
2. Navigate to the 'Products' tab by simulating a tap on the bottom navigation bar at coordinates (540, 2700) with a delay of 1.5s.
3. Scroll down to find 'Wireless Earbuds' using `adb shell input swipe 540 2000 540 1000 500` (500ms swipe).
4. Tap the 'Add to Cart' button at (360, 2200) and wait for the confirmation toast.
5. Tap the cart icon at (900, 2700) and verify the item count updates to '1'.
6. Proceed to checkout by tapping the 'Checkout' button at (540, 2900).
7. Fill in shipping details: Name='John Doe', Address='123 Main St', Zip='94105'.
8. Tap 'Place Order' and validate the success message 'Order #12345 Confirmed' appears.

**LangChain Middleware Config:**
```python
from langchain_community.agents import AutoGLMAgent
from deepagents_autoglm import DeepAgentsAutoGLM

middleware = DeepAgentsAutoGLM(
    device_id='emulator-5554',
    platform='android',
    max_steps=20,
    vision_enabled=True,
    error_handling='retry_on_failure'
)

agent = AutoGLMAgent(
    middleware=middleware,
    task_description="Automate retail app checkout end-to-end",
    constraints=["No manual intervention after start", "Validate each step with screenshots"]
)
```

**Validation:**
- Screenshot after Step 3: Confirmed 'Wireless Earbuds' visible.
- Toast message after Step 4: 'Added to cart' detected.
- Cart count after Step 5: '1' confirmed via OCR.
- Order confirmation after Step 8: 'Order #12345 Confirmed' matched.

**Errors Encountered:**
- Step 6 failed initially due to network latency. Retried after 2s delay with success.
- Step 7 required scrolling to reveal the 'Zip' field. Adjusted swipe coordinates dynamically.

**Outcome:** Task completed in 42 seconds with 100% success rate. Screenshots and logs saved to `/automation_reports/retail_checkout_20240515/`.