macos-accessibility

macOS Accessibility APIs enable cross-application automation including window control, text selection reading, keyboard monitoring, and UI element inspection. Script-kit-gpui uses these APIs extensively for text expansion, window tiling, and getting selected text from other applications.

Crate Dependencies

toml

# Cargo.toml
get-selected-text = "0.1"      # Hybrid AX + clipboard fallback for reading selected text
macos-accessibility-client = "0.0.1"  # Permission checking for accessibility APIs

Permission Requirements

Why Accessibility Permission Is Required

Accessibility permission enables:

Text expansion / snippets - Global keyboard monitoring
Window control (move, resize, tile) - Cross-process window manipulation
Get selected text from other apps - AXSelectedText attribute access
Global keyboard shortcuts - System-wide hotkey capture

Checking Permission

rust

use macos_accessibility_client::accessibility;

/// Check if accessibility permissions are granted
pub fn has_accessibility_permission() -> bool {
    accessibility::application_is_trusted()
}

Requesting Permission (Shows System Dialog)

rust

use macos_accessibility_client::accessibility;

/// Request accessibility permission - opens System Preferences with prompt
pub fn request_accessibility_permission() -> bool {
    accessibility::application_is_trusted_with_prompt()
}

Opening Settings Directly

rust

use std::process::Command;

/// Open System Preferences to Accessibility pane (no prompt)
pub fn open_accessibility_settings() -> std::io::Result<()> {
    Command::new("open")
        .arg("x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility")
        .spawn()?;
    Ok(())
}

Permission Flow in script-kit-gpui

See src/permissions_wizard.rs for the complete permission management system:

PermissionStatus - Overall permission state
PermissionInfo - Per-permission details with UI-ready descriptions
check_all_permissions() - Main entry point

Reading Selected Text

The get-selected-text crate provides a hybrid approach with automatic fallbacks:

rust

use get_selected_text::get_selected_text as get_selected_text_impl;

pub fn get_selected_text() -> Result<String> {
    // Check permissions first
    if !has_accessibility_permission() {
        bail!("Accessibility permission required");
    }
    
    // The crate handles:
    // 1. AXSelectedText attribute (fastest, most reliable)
    // 2. AXSelectedTextRange + AXStringForRange (fallback)
    // 3. Clipboard simulation with Cmd+C (last resort)
    match get_selected_text_impl() {
        Ok(text) => Ok(text),
        Err(e) => bail!("Failed to get selected text: {}", e),
    }
}

Selection Reading Strategies (in order)

AXSelectedText - Direct attribute read, fastest
AXSelectedTextRange + AXStringForRange - Range-based fallback
Clipboard simulation - Saves clipboard, sends Cmd+C, restores

The crate caches per-app behavior with an LRU cache for efficiency.

Setting Selected Text (Replace Selection)

rust

use arboard::Clipboard;

pub fn set_selected_text(text: &str) -> Result<()> {
    if !has_accessibility_permission() {
        bail!("Accessibility permission required");
    }
    
    let mut clipboard = Clipboard::new()?;
    
    // Save original clipboard
    let original = clipboard.get_text().ok();
    
    // Set new text
    clipboard.set_text(text)?;
    thread::sleep(Duration::from_millis(10));
    
    // Simulate Cmd+V
    simulate_paste_with_cg()?;
    thread::sleep(Duration::from_millis(50));
    
    // Restore original clipboard
    if let Some(original_text) = original {
        thread::sleep(Duration::from_millis(100));
        clipboard.set_text(&original_text)?;
    }
    
    Ok(())
}

Simulating Paste with Core Graphics

rust

use core_graphics::event::{CGEvent, CGEventFlags, CGEventTapLocation, CGKeyCode};
use core_graphics::event_source::{CGEventSource, CGEventSourceStateID};

pub fn simulate_paste_with_cg() -> Result<()> {
    const KEY_V: CGKeyCode = 9;  // 'v' keycode on macOS
    
    let source = CGEventSource::new(CGEventSourceStateID::HIDSystemState)
        .ok().context("Failed to create CGEventSource")?;
    
    // Key down with Cmd
    let key_down = CGEvent::new_keyboard_event(source.clone(), KEY_V, true)
        .ok().context("Failed to create key down event")?;
    key_down.set_flags(CGEventFlags::CGEventFlagCommand);
    
    // Key up with Cmd
    let key_up = CGEvent::new_keyboard_event(source, KEY_V, false)
        .ok().context("Failed to create key up event")?;
    key_up.set_flags(CGEventFlags::CGEventFlagCommand);
    
    // Post events
    key_down.post(CGEventTapLocation::HID);
    thread::sleep(Duration::from_millis(5));
    key_up.post(CGEventTapLocation::HID);
    
    Ok(())
}

AXUIElement API

FFI Declarations

rust

#![allow(non_upper_case_globals)]

use std::ffi::c_void;

type AXUIElementRef = *const c_void;
type CFTypeRef = *const c_void;
type CFStringRef = *const c_void;
type CFArrayRef = *const c_void;

#[link(name = "ApplicationServices", kind = "framework")]
extern "C" {
    fn AXUIElementCreateSystemWide() -> AXUIElementRef;
    fn AXUIElementCreateApplication(pid: i32) -> AXUIElementRef;
    fn AXUIElementCopyAttributeValue(
        element: AXUIElementRef,
        attribute: CFStringRef,
        value: *mut CFTypeRef,
    ) -> i32;
    fn AXUIElementSetAttributeValue(
        element: AXUIElementRef,
        attribute: CFStringRef,
        value: CFTypeRef,
    ) -> i32;
    fn AXUIElementPerformAction(element: AXUIElementRef, action: CFStringRef) -> i32;
    fn AXUIElementIsAttributeSettable(
        element: AXUIElementRef,
        attribute: CFStringRef,
        settable: *mut bool,
    ) -> i32;
}

// AXError codes
const kAXErrorSuccess: i32 = 0;
const kAXErrorAPIDisabled: i32 = -25211;
const kAXErrorNoValue: i32 = -25212;

Getting Attribute Values

rust

fn get_ax_attribute(element: AXUIElementRef, attribute: &str) -> Result<CFTypeRef> {
    let attr_str = create_cf_string(attribute);
    let mut value: CFTypeRef = std::ptr::null();
    
    let result = unsafe {
        AXUIElementCopyAttributeValue(element, attr_str, &mut value as *mut CFTypeRef)
    };
    
    cf_release(attr_str);
    
    match result {
        kAXErrorSuccess => Ok(value),
        kAXErrorAPIDisabled => bail!("Accessibility API is disabled"),
        kAXErrorNoValue => bail!("No value for attribute: {}", attribute),
        _ => bail!("Failed to get attribute {}: error {}", attribute, result),
    }
}

Setting Attribute Values

rust

fn set_ax_attribute(element: AXUIElementRef, attribute: &str, value: CFTypeRef) -> Result<()> {
    let attr_str = create_cf_string(attribute);
    let result = unsafe { AXUIElementSetAttributeValue(element, attr_str, value) };
    cf_release(attr_str);
    
    match result {
        kAXErrorSuccess => Ok(()),
        kAXErrorAPIDisabled => bail!("Accessibility API is disabled"),
        _ => bail!("Failed to set attribute {}: error {}", attribute, result),
    }
}

Checking Attribute Settability

rust

pub fn is_attribute_settable(element: AXUIElementRef, attribute: &str) -> bool {
    if element.is_null() {
        return false;
    }
    
    let attr_str = create_cf_string(attribute);
    let mut settable = false;
    
    let result = unsafe {
        AXUIElementIsAttributeSettable(element, attr_str, &mut settable as *mut bool)
    };
    
    cf_release(attr_str);
    result == kAXErrorSuccess && settable
}

Common AX Attributes

Attribute	Type	Description
`AXPosition`	AXValue (CGPoint)	Window position
`AXSize`	AXValue (CGSize)	Window dimensions
`AXTitle`	CFString	Window title
`AXWindows`	CFArray	Application's windows
`AXFocusedWindow`	AXUIElement	Currently focused window
`AXMainWindow`	AXUIElement	Application's main window
`AXMinimized`	CFBoolean	Minimization state
`AXSelectedText`	CFString	Currently selected text
`AXSelectedTextRange`	AXValue	Selection range
`AXCloseButton`	AXUIElement	Close button element
`AXMinimizeButton`	AXUIElement	Minimize button element
`AXFullScreenButton`	AXUIElement	Fullscreen button element

Common AX Actions

Action	Description
`AXRaise`	Bring window to front
`AXPress`	Press a button element

Window Control Pattern

See src/window_control.rs for complete implementation.

Listing Windows

rust

pub fn list_windows() -> Result<Vec<WindowInfo>> {
    if !has_accessibility_permission() {
        bail!("Accessibility permission required");
    }
    
    let mut windows = Vec::new();
    
    // Iterate running applications via NSWorkspace
    unsafe {
        use objc::{msg_send, sel, sel_impl};
        use objc::runtime::{Class, Object};
        
        let workspace_class = Class::get("NSWorkspace")?;
        let workspace: *mut Object = msg_send![workspace_class, sharedWorkspace];
        let running_apps: *mut Object = msg_send![workspace, runningApplications];
        
        // For each app...
        let ax_app = AXUIElementCreateApplication(pid);
        if let Ok(windows_value) = get_ax_attribute(ax_app, "AXWindows") {
            // Iterate windows...
        }
        cf_release(ax_app);
    }
    
    Ok(windows)
}

Getting Focused Window of Previous App

For LSUIElement (accessory) apps like Script Kit that don't take menu bar ownership:

rust

pub fn get_frontmost_window_of_previous_app() -> Result<Option<WindowInfo>> {
    // Menu bar owner is the previously active app
    let target_pid = get_menu_bar_owner_pid()?;
    
    let ax_app = unsafe { AXUIElementCreateApplication(target_pid) };
    
    // Strategy 1: AXFocusedWindow (most accurate)
    // Strategy 2: AXMainWindow (fallback)
    // Strategy 3: First window in AXWindows array
    
    // ...
}

pub fn get_menu_bar_owner_pid() -> Result<i32> {
    unsafe {
        let workspace: *mut Object = msg_send![workspace_class, sharedWorkspace];
        let menu_owner: *mut Object = msg_send![workspace, menuBarOwningApplication];
        let pid: i32 = msg_send![menu_owner, processIdentifier];
        Ok(pid)
    }
}

Window Capability Detection

See src/window_control_enhanced/capabilities.rs:

rust

pub fn detect_window_capabilities(ax_element: *const c_void) -> WindowCapabilities {
    WindowCapabilities {
        can_move: is_attribute_settable(ax_element, "AXPosition"),
        can_resize: is_attribute_settable(ax_element, "AXSize"),
        can_minimize: has_attribute(ax_element, "AXMinimizeButton"),
        can_close: has_attribute(ax_element, "AXCloseButton"),
        can_fullscreen: has_attribute(ax_element, "AXFullScreenButton"),
        supports_space_move: false,
    }
}

CoreFoundation Memory Management

Critical: AX functions follow CoreFoundation naming conventions:

AXUIElementCreate* - Returns owned object, caller must release
AXUIElementCopy* - Returns owned copy, caller must release
CFArrayGetValueAtIndex - Returns borrowed reference, retain if keeping

rust

fn cf_release(cf: CFTypeRef) {
    if !cf.is_null() {
        unsafe { CFRelease(cf); }
    }
}

fn cf_retain(cf: CFTypeRef) -> CFTypeRef {
    if !cf.is_null() {
        unsafe { CFRetain(cf) }
    } else {
        cf
    }
}

Retain Pattern for Array Elements

rust

// CFArrayGetValueAtIndex returns borrowed - must retain for storage
let ax_window = CFArrayGetValueAtIndex(windows_value as CFArrayRef, j);
let retained_window = cf_retain(ax_window);  // Now we own it
cache_window(window_id, retained_window as AXUIElementRef);

// Release the array when done
cf_release(windows_value);

Privacy Considerations

What Triggers Permission Dialogs

accessibility::application_is_trusted_with_prompt() - Shows system dialog
First AX API call without permission - May show dialog

What Does NOT Trigger Dialogs

accessibility::application_is_trusted() - Silent check
Opening settings URL directly

User Flow

Check permission silently at startup
If missing, show custom UI explaining why it's needed
Provide button that calls request_accessibility_permission()
Optionally show "Open Settings" button for manual enablement

Fallback Strategies

When AX API Fails

No permission - Guide user through permission flow
App doesn't support AX - Fall back to clipboard simulation
Element not accessible - Try parent element or alternate attribute
Operation fails - Check AXUIElementIsAttributeSettable first

Clipboard Fallback for Text Operations

rust

// Always save/restore clipboard
let original = clipboard.get_text().ok();
// ... do operation ...
if let Some(orig) = original {
    clipboard.set_text(&orig)?;
}

Anti-Patterns

Memory Leaks

rust

// BAD: Leaks CFString
let attr = create_cf_string("AXPosition");
// ... use attr but never release ...

// GOOD: Always release
let attr = create_cf_string("AXPosition");
// ... use attr ...
cf_release(attr);

Dangling References

rust

// BAD: Using borrowed reference after array released
let window = CFArrayGetValueAtIndex(array, 0);
cf_release(array);  // window is now invalid!
do_something(window);  // CRASH

// GOOD: Retain before releasing array
let window = cf_retain(CFArrayGetValueAtIndex(array, 0));
cf_release(array);
do_something(window);  // Safe
cf_release(window);  // Clean up our retained copy

Missing Permission Checks

rust

// BAD: Will fail cryptically
pub fn get_windows() -> Vec<Window> {
    let ax_app = AXUIElementCreateApplication(pid);
    // ...
}

// GOOD: Fail fast with clear error
pub fn get_windows() -> Result<Vec<Window>> {
    if !has_accessibility_permission() {
        bail!("Accessibility permission required");
    }
    // ...
}

Blocking on Permission Request

rust

// BAD: Blocks UI thread waiting for user
let granted = request_accessibility_permission();
if !granted {
    panic!("Need permission!");
}

// GOOD: Non-blocking flow
if !has_accessibility_permission() {
    show_permission_ui();
    return; // Let user grant permission in their own time
}

Key Files in script-kit-gpui

File	Purpose
`src/selected_text.rs`	Get/set selected text operations
`src/window_control.rs`	Window listing, moving, resizing, tiling
`src/window_control_enhanced/`	Enhanced window ops with capability detection
`src/permissions_wizard.rs`	Permission checking and UI data
`src/expand_manager.rs`	Text expansion using keyboard monitoring
`src/executor/selected_text.rs`	Message handlers for selected text

Testing Accessibility Code

rust

#[cfg(all(test, feature = "system-tests"))]
mod system_tests {
    #[test]
    fn test_permission_check_does_not_panic() {
        let _ = has_accessibility_permission();
    }
    
    #[test]
    #[ignore] // Requires manual setup
    fn test_get_selected_text() {
        // 1. Open TextEdit, type and select text
        // 2. Run: cargo test --features system-tests test_get_selected_text -- --ignored
        let text = get_selected_text().expect("Should get selected text");
        assert!(!text.is_empty());
    }
}

Search AI Tools

Install this agent skill to your Project

SKILL.md

macos-accessibility

Crate Dependencies

Permission Requirements

Why Accessibility Permission Is Required

Checking Permission

Requesting Permission (Shows System Dialog)

Opening Settings Directly

Permission Flow in script-kit-gpui

Reading Selected Text

Selection Reading Strategies (in order)

Setting Selected Text (Replace Selection)

Simulating Paste with Core Graphics

AXUIElement API

FFI Declarations

Getting Attribute Values

Setting Attribute Values

Checking Attribute Settability

Common AX Attributes

Common AX Actions

Window Control Pattern

Listing Windows

Getting Focused Window of Previous App

Window Capability Detection

CoreFoundation Memory Management

Retain Pattern for Array Elements

Privacy Considerations

What Triggers Permission Dialogs

What Does NOT Trigger Dialogs

User Flow

Fallback Strategies

When AX API Fails

Clipboard Fallback for Text Operations

Anti-Patterns

Memory Leaks

Dangling References

Missing Permission Checks

Blocking on Permission Request

Key Files in script-kit-gpui

Testing Accessibility Code