Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Python PyAutoGUI Mouse, Keyboard, and Screenshot Automation

Automate mouse movements, keyboard input, and screenshot capture with PyAutoGUI in Python. Covers setup, practical API usage, and reliability tips.

~6 min readPython
#PyAutoGUI#UI Automation#Screenshot Capture#Desktop Automation
Illustration of a Python script controlling a mouse cursor and capturing a screenshot on a desktop screen.

PyAutoGUI is a cross-platform Python library for programmatically controlling the mouse and keyboard and capturing screenshots. It is commonly used for automating repetitive desktop tasks, testing GUI applications, and building simple bots. This article covers practical mouse control, keyboard input, screenshot capture, reliability considerations, and the main limitations to keep in mind.

Core Capabilities of PyAutoGUI

PyAutoGUI provides a single API for moving the mouse, typing text, pressing keys, and taking screenshots. It works on Windows, macOS, and Linux, though some functions behave differently across platforms. The library uses platform-specific backends: Windows uses the Win32 API, macOS uses Quartz, and Linux uses X11 APIs. Under a Wayland session on Linux, some input and screen-capture functions may not work the same way.

The main operations are:

  • Moving the cursor to absolute or relative screen coordinates.
  • Clicking, double-clicking, right-clicking, and dragging.
  • Scrolling the mouse wheel.
  • Typing strings and pressing individual keys or key combinations.
  • Capturing the entire screen or a region as an image.
  • Locating an image on the screen to find UI elements.

Because PyAutoGUI works at the operating system level, it does not require any hooks into the target application. This makes it useful for automating legacy software, remote desktop sessions, and games, but it also means it is sensitive to screen resolution, scaling, and the current state of the desktop.

Installation and Initial Setup

Install PyAutoGUI with pip:

bash
pip install pyautogui

On Linux, you may need to install additional system dependencies for screenshot support. For example, on Debian-based systems, python3-tk and scrot are often required. On macOS, you may need to grant the terminal or IDE accessibility permissions under System Settings > Privacy & Security > Accessibility.

After installation, verify that the library can access the screen and that coordinates are reported as expected:

python
import pyautogui # Get the screen size width, height = pyautogui.size() print(f"Screen size: {width}x{height}") # Get the current mouse position x, y = pyautogui.position() print(f"Mouse position: ({x}, {y})")

A common mistake is assuming that coordinates are in physical pixels. On displays with scaling (e.g., a 4K monitor at 150% scaling), PyAutoGUI uses logical coordinates that match the operating system's reported resolution. This can cause clicks to land in the wrong place if you mix physical and logical coordinates.

Mouse Automation with PyAutoGUI

Mouse control is the most straightforward part of PyAutoGUI. The moveTo() function moves the cursor to an absolute position, while moveRel() moves it relative to the current position. Both accept a duration parameter to control the speed of the movement, which is useful for simulating human-like motion.

python
import pyautogui # Move to absolute coordinates (100, 200) over 0.5 seconds pyautogui.moveTo(100, 200, duration=0.5) # Move 50 pixels right and 30 pixels down pyautogui.moveRel(50, 30, duration=0.2)

Clicking is done with click(), doubleClick(), rightClick(), and middleClick(). You can specify the button and the number of clicks, and you can pass coordinates to move first:

python
pyautogui.click(400, 300, button='left') pyautogui.doubleClick(400, 300) pyautogui.rightClick(400, 300)

Dragging is performed with dragTo() and dragRel(), which behave like mouse movement but keep the button pressed. This is useful for selecting text or moving files:

python
# Drag from current position to (500, 500) pyautogui.dragTo(500, 500, duration=0.3, button='left')

Scrolling is done with scroll(), where a positive number scrolls up and a negative number scrolls down. The unit is roughly a line or a notch, depending on the platform.

python
pyautogui.scroll(5) # scroll up 5 notches pyautogui.scroll(-3) # scroll down 3 notches

One important behavior is that PyAutoGUI functions are blocking. moveTo() will not return until the movement is complete. This is fine for most scripts, but if you need to perform other work during the movement, you would need to run it in a separate thread.

Keyboard Automation with PyAutoGUI

The keyboard API lets you type strings and press individual keys. The write() function (with typewrite() as an older alias) types a string character by character. It handles letters and digits reliably, but the mapping of special characters such as @ or # can depend on the active keyboard layout, so it is not reliable for arbitrary symbols on all systems.

python
pyautogui.write('Hello, world!', interval=0.05)

The press() function presses a key, and hotkey() presses a combination of keys in sequence. For example, to copy selected text with Ctrl+C:

python
pyautogui.hotkey('ctrl', 'c')

Key names are strings like 'enter', 'esc', 'tab', 'space', and arrow keys. The full list is available in the PyAutoGUI documentation, but the most common ones are intuitive.

You can also hold a key down for a specific duration using keyDown() and keyUp():

python
pyautogui.keyDown('shift') pyautogui.press('a') # types uppercase A pyautogui.keyUp('shift')

A common pitfall is that write() does not handle the @ symbol correctly on some non-US keyboard layouts. If you need to type special characters, consider using pyperclip to copy the text to the clipboard and then paste it with Ctrl+V. This is more reliable across layouts.

Capturing Screenshots and Locating Images

PyAutoGUI can capture the entire screen or a specific region using screenshot(). The function returns a PIL Image object, which you can save or manipulate.

python
import pyautogui # Full screen img = pyautogui.screenshot() img.save('full.png') # Region: left, top, width, height region = (100, 100, 500, 400) img = pyautogui.screenshot(region=region) img.save('region.png')

Screenshots are essential for verifying the state of the UI and for locating elements. The locateOnScreen() function searches for an image within the screen and returns the location of the first match. This is useful when you need to click a button that is not at a fixed position.

python
button_img = 'button.png' # a small image of the button location = pyautogui.locateOnScreen(button_img, confidence=0.8) if location: center = pyautogui.center(location) pyautogui.click(center) else: print("Button not found")

The confidence parameter is only available on OpenCV-enabled installations. Without it, the image matching is pixel-perfect, which can be fragile if the UI changes slightly. When using confidence, you must install opencv-python separately.

Screenshot-based image matching is slower than native UI automation because it compares screen pixels against the source image. You can limit the search area with the region parameter to improve speed and avoid false positives.

Timing, Reliability, and Failure Modes

PyAutoGUI automation is inherently timing-sensitive. The library does not wait for the UI to respond, so you must add explicit delays when the application takes time to render or process input. A common pattern is to use pyautogui.sleep() or Python's time.sleep() after triggering an action.

python
pyautogui.click(100, 200) pyautogui.sleep(1) # wait for the UI to update

Another reliability issue is that a user can move the mouse while the script is running, which causes clicks to land in unexpected places. PyAutoGUI's fail-safe is enabled by default in current releases: moving the cursor to a designated corner raises FailSafeException and stops the script. Keep it enabled for unattended automation so you have a way to stop a runaway script. It does not prevent a user from moving the cursor, however.

Failure modes include:

  • Screen resolution changes: If the display resolution changes mid-script, coordinates become invalid.
  • Scaling factors: On Windows, changing display scaling changes logical coordinates.
  • Image matching failures: If the UI theme changes, locateOnScreen may not find the target image.
  • Permissions: On macOS and Linux, missing accessibility permissions can cause input events to be silently ignored.

To make automation more robust, you can add retry loops around operations that depend on UI state. For example, wait for an image to appear before clicking it:

python
import time for _ in range(10): loc = pyautogui.locateOnScreen('submit.png', confidence=0.8) if loc: pyautogui.click(pyautogui.center(loc)) break time.sleep(0.5)

When PyAutoGUI Is the Right Choice

PyAutoGUI is a good fit for quick scripts that need to interact with desktop applications, especially when you do not have access to the application's internal automation interfaces. It is also useful for cross-platform GUI testing where you want to simulate real user input.

However, it is not suitable for scenarios that require high precision or high throughput. Because it relies on screen coordinates and image matching, it is slower and more brittle than using native accessibility APIs like Windows UI Automation or macOS Accessibility. For production-grade test automation, consider using tools like Selenium for web apps or pywinauto for Windows desktop apps, which provide more stable element locators.

For one-off tasks, PyAutoGUI's simplicity is a major advantage. You can write a script in a few minutes without needing to understand the target application's internals. The tradeoff is that the script will break if the UI layout changes. Keep that in mind when deciding whether to invest in a more robust solution.

A final consideration is that PyAutoGUI runs on the same machine as the UI. It cannot automate remote systems without a remote desktop session, and it will interfere with any interactive user. For headless automation, you would need a virtual display like Xvfb on Linux, but that adds complexity and may not work for all applications.

Try matching browser tools

All tools

Small utilities that fit Python work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Python guides

More in Python

All Python guides

The guides filed next to this one in Python, in library order.