Midscene.js

English | 简体中文

Your AI Operator for Web, Android, Automation & Testing

Showcases

Instruction	Video
Use JS code to drive task orchestration, collect information about Jay Chou's concert, and write it into Google Docs (By UI-TARS model)	google-doc-1080p.mp4
Control Maps App on Android (By Qwen-2.5-VL model)	control-maps-app-on-android.mp4
Using midscene mcp to browse the page (https://www.saucedemo.com/), perform login, add products, place orders, and finally generate test cases based on mcp execution steps and playwright example	showcase-3-mcp.mp4

💡 Features

Write Automation with Natural Language

• Describe your goals and steps, and Midscene will plan and operate the user interface for you.
• Use Javascript SDK or YAML to write your automation script.

Web & Mobile App

• Web Automation 🖥️: Either integrate with Puppeteer, Playwright or use Bridge Mode to control your desktop browser.
• Android Automation 📱: Use Javascript SDK with adb to control your local Android device.

Tools

• Visual Reports for Debugging 🎞️: Through our test reports and Playground, you can easily understand, replay and debug the entire process.
• Caching for Efficiency 🔄: Replay your script with cache and get the result faster.
• MCP 🔗: Allows other MCP Clients to directly use Midscene's capabilities.

Three kinds of APIs

• Interaction API 🔗: interact with the user interface.
• Data Extraction API 🔗: extract data from the user interface and dom.
• Utility API 🔗: utility functions like aiAssert(), aiLocate(), aiWaitFor().

👉 Zero-code Quick Experience

• Chrome Extension: Start in-browser experience immediately through the Chrome Extension, without writing any code.
• Android Playground: There is also a built-in Android playground to control your local Android device.

✨ Model Choices

Midscene.js supports both multimodal LLMs like gpt-4o, and visual-language models like Qwen2.5-VL, Doubao-1.5-thinking-vision-pro, gemini-2.5-pro and UI-TARS.

Visual-language models are recommended for UI automation.

Read more about Choose a model

💡 Two Styles of Automation

Auto Planning

Midscene will automatically plan the steps and execute them. It may be slower and heavily rely on the quality of the AI model.

await aiAction('click all the records one by one. If one record contains the text "completed", skip it');

Workflow Style

Split complex logic into multiple steps to improve the stability of the automation code.

const recordList = await agent.aiQuery('string[], the record list')
for (const record of recordList) {
  const hasCompleted = await agent.aiBoolean(`check if the record contains the text "completed"`)
  if (!hasCompleted) {
    await agent.aiTap(record)
  }
}

For more details about the workflow style, please refer to Blog - Use JavaScript to Optimize the AI Automation Code

👀 Comparing to other projects

There are so many UI automation tools out there, and each one seems to be all-powerful. What's special about Midscene.js?

• Debugging Experience: You will soon realize that debugging and maintaining automation scripts is the real challenge. No matter how magical the demo looks, ensuring stability over time requires careful debugging. Midscene.js offers a visualized report file, a built-in playground, and a Chrome Extension to simplify the debugging process. These are the tools most developers truly need, and we're continually working to improve the debugging experience.

• Open Source, Free, Deploy as you want: Midscene.js is an open-source project. It's decoupled from any cloud service and model provider, you can choose either public or private deployment. There is always a suitable plan for your business.

• Integrate with Javascript: You can always bet on Javascript 😎

📄 Resources

• Home Page and Documentation: https://midscenejs.com
• API Reference: https://midscenejs.com/api.html
• GitHub: https://github.com/web-infra-dev/midscene

🤝 Community

• Discord
• Follow us on X
• Lark Group(飞书交流群)

📝 Credits

We would like to thank the following projects:

• Rsbuild for the build tool.
• UI-TARS for the open-source agent model UI-TARS.
• Qwen2.5-VL for the open-source VL model Qwen2.5-VL.
• scrcpy and yume-chan allow us to control Android devices with browser.
• appium-adb for the javascript bridge of adb.
• YADB for the yadb tool which improves the performance of text input.
• Puppeteer for browser automation and control.
• Playwright for browser automation and control and testing.

📝 Citation

If you use Midscene.js in your research or project, please cite:

@software{Midscene.js,
  author = {Xiao Zhou, Tao Yu, YiBing Lin},
  title = {Midscene.js: Your AI Operator for Web, Android, Automation & Testing.},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/web-infra-dev/midscene}
}

📝 License

Midscene.js is MIT licensed.

---

If this project helps you or inspires you, please give us a ⭐️