Linux Desktop GUI Automation

Automate any X11 Linux desktop: capture screens, find and click elements, type, use hotkeys, manage windows.

Preferred screenshot interpretation path: capture with capture.sh and interpret the image directly in your OpenClaw chat (existing image-capable model connection).

Prerequisites

• X11 session running (XFCE, GNOME on X11, KDE on X11, i3, openbox, etc.)
• DISPLAY environment variable set (usually :0)
• Run bash install.sh once to install dependencies
• No extra key needed for screenshot interpretation when using OpenClaw's image-capable chat path

Quick Reference

Task	Command
Take screenshot	bash capture.sh
Screenshot of window	bash capture.sh --window "Firefox"
List windows	bash inspect.sh
Active window info	bash inspect.sh --active
Find window by name	bash inspect.sh --window "Firefox"
Click at coordinates	bash click.sh --x 500 --y 300
Right-click	bash click.sh --x 500 --y 300 --button right
Double-click	bash click.sh --x 500 --y 300 --double
Click relative to window	bash click.sh --window "Firefox" --x 200 --y 150
Type text	bash type.sh "hello world"
Type into window	bash type.sh --window "Terminal" "ls -la"
Send hotkey	bash hotkey.sh "ctrl+c"
Send Enter	bash hotkey.sh "Return"
Scroll down	bash scroll.sh --direction down --amount 3
Scroll up at position	bash scroll.sh --x 500 --y 300 --direction up --amount 3
Focus window	bash window.sh --action focus --window "Firefox"
Minimize window	bash window.sh --action minimize --window "Firefox"
Maximize window	bash window.sh --action maximize --window "Firefox"
Close window	bash window.sh --action close --window "Firefox"
Move window	bash window.sh --action move --window "Firefox" --x 100 --y 50
Resize window	bash window.sh --action resize --window "Firefox" --width 1280 --height 800

Typical Automation Workflow

For most GUI automation tasks, follow this pattern:

1. Capture a screenshot with capture.sh — note the file path printed
2. Look at the screenshot yourself to understand what's on screen
3. Find the target element by examining the screenshot and estimating its pixel coordinates
4. Act using the coordinates: click.sh --x X --y Y
5. Verify by capturing another screenshot and checking the result

Example: Click the Save button in a dialog

# Step 1: Capture the screen
SCREENSHOT=$(bash capture.sh | tail -1)

# Step 2: Look at the screenshot (read the image file with your vision)
# Examine the image and identify the Save button's position

# Step 3: Click at the coordinates you identified
bash click.sh --x 450 --y 320

Example: Type into a specific application

# Focus the terminal window and type a command
bash type.sh --window "Terminal" "ls -la"
bash hotkey.sh "Return"

Example: Window management

# Maximize Firefox, then focus a terminal
bash window.sh --action maximize --window "Firefox"
bash window.sh --action focus --window "Terminal"

JSON Output

All tools support a --json flag for machine-readable output:

{"success": true, "output": "...", "error": null}

On failure:

{"success": false, "output": null, "error": "Error description"}

Environment Setup

If DISPLAY is not set (e.g., running over SSH), set it before calling any tool:

export DISPLAY=:0

For headless servers with a virtual display:

Xvfb :0 -screen 0 1920x1080x24 &
export DISPLAY=:0

Hotkey Reference

Key names follow X11 conventions:

Key	Name
Enter	Return
Tab	Tab
Escape	Escape
Backspace	BackSpace
Delete	Delete
Home	Home
End	End
Page Up	Page_Up
Page Down	Page_Down
F1-F12	F1 through F12
Super/Win	super
Ctrl	ctrl
Alt	alt
Shift	shift

Combine with +: ctrl+c, ctrl+shift+t, alt+F4, super+d

Limitations

• X11 only — does not work on Wayland sessions
• Cannot interact with Wayland-native apps in a Wayland session
• Some apps with custom rendering (games, Electron apps with security flags) may resist automation
• Screenshot quality depends on compositor; disable compositing if captures look wrong