aai - sigma-rt

This module provides a high-level interface for controlling Android device UI through the AAI (Advanced Accessibility Interface),
allowing script-level automation of UI elements like buttons, text fields, checkboxes, sliders, and more.

It is designed to work with Total Control’s FindNode engine and integrates seamlessly with the UI Explorer tool.
Developers can locate and interact with UI components via selectors, perform input and gesture operations,
monitor screen states, and manage multi-device batch actions.

All functions are accessible via UiElement, UiElementArray, UiSelector, or directly from Device.sendAai(...).

Key Features

UI Query & Selection: Use UiSelector or FindNode queries to locate UI elements on screen.
Element Interaction: Click, long-click, input text, get/set properties, take screenshots.
Batch Control: UiElementArray enables simultaneous actions across devices or multiple UI nodes.
State Monitoring: Register listeners with addQueryListener() to watch UI changes in real-time.
FindNode Integration: Supports advanced queries like "T:Settings&&R:.settings" with rich filter support.
Custom Automation: Compatible with TCThreadVirtual, DeviceArray, and async scripts for full test automation.

Supported Query Modes

Selector strings: T:Text, D:Description, R:Resource ID, C:Class Name
Chained selectors: via UiSelector (e.g., .text("OK").idEndsWith("btn"))
Raw Node IDs: 153b8 (from UI Explorer)
Bounds-based: coordinates-based queries using BI:[x,y]

Example

var { Device } = require("sigma/device");
var { UiElement, UiElementArray, UiSelector, addQueryListener } = require("sigma/aai");
var sigmaDevice = Device.getMain();

// Find and click a button by text
var el = UiElement.findObject(sigmaDevice, "T:Settings");
el.clickSync();

// Input text into multiple fields across devices
var el1 = UiElement.findObject(Device.getMain(), "T:Search");
var el2 = UiElement.findObject(Device.getMain(), "T:Input");
var arr = new UiElementArray();
arr.add(el1);
arr.add(el2);
arr.inputTextSync(0, "Hello AAI");

// Monitor screen for appearance of "Allow" button
addQueryListener("watchAllow", sigmaDevice, "T:Allow", function(name, dev, pkg, node) {
    node.clickSync();
});

Notes

All AAI operations are executed using device.sendAai() behind the scenes.
For full selector reference, see the UI Explorer or FindNode documentation.
Most actions require accessibility permissions granted to Total Control on the device.
inputTextSync() requires Sigma input method to be active on device.

Classes

UiElement
UiElementArray
UiSelector

Methods

(static) addQueryListener(name, device, query, callback) → {string}

Registers a UI query listener on the specified device that continuously monitors for UI elements matching the given query condition.
When a match is found, the provided callback function is triggered.

Example

// Example: Add a listener to detect when the text "Total Control" appears
var { Device } = require("sigma/device");
var { addQueryListener } = require("sigma/aai");
var sigmaDevice = Device.getMain();

function callback(name, device, packageName, node) {
    print("Trigger callback");
    print(name);
    print(device.getName());
    print(packageName);
    print(node.getText());
}

var name = "test";
var id = addQueryListener(name, sigmaDevice, "T:Total Control", callback);
print(id); // Outputs: listener ID

// Operation Result:
// If it executes successfully:
- Triggers the callback when the query condition is matched.
- Prints the assigned listener ID.

Parameters:

Name	Type	Description
`name`	string	A unique identifier for this listener (can be used later for removal or management).
`device`	Device	The target device object, typically obtained by `Device.getMain()`.
`query`	string	A string that specifies the UI query condition (e.g. `"T:Total Control"`, `"D:Settings"`). Supports prefix types: `T` (text), `D` (description), `R` (resource-id), `C` (class name).
`callback`	function	A function executed when the query condition is matched. It receives four parameters: `name` {string} - The name of the listener. `device` {Device} - The device object. `packageName` {string} - The package name of the foreground app. `node` {Object} - The matched UI node, from which you can retrieve properties like `.getText()`, `.clickSync()`, etc.

Returns:

The ID of the registered listener. Can be used for later removal via removeQueryListener(id).

Type: string

(static) buildAction(fn, …args) → {string}

Serialize a command-like call into a single string: fn(arg1, arg2, ...).

Mapping rules:

If an arg is a "raw fragment" (i.e. has __raw), emit it as-is (unquoted).
If an arg is a number or boolean, emit it as-is (unquoted).
Otherwise, treat it as a string and emit a single double-quoted literal via quoteArg(...).

IMPORTANT:

Objects/arrays passed directly will NOT become JSON text automatically;
they'll be coerced to "[object Object]" / "x,y" as a single quoted string.
If you need real JSON text, do JSON.stringify(...) yourself and pass that string,
OR, if your command language supports object/array literals, wrap them with raw(...).

Example

// 1) Unquoted enum/identifier
buildAction("setMode", raw("HTML"));          // => setMode(HTML)
buildAction("useProfile", raw("DEFAULT"));    // => useProfile(DEFAULT)

// 2) Mixed raw + quoted strings
buildAction("setClipData", raw("html"), "Hello", "<p style='color:red'>Hi</p>");
// => setClipData(html, "Hello", "<p style=\\'color:red\\'>Hi</p>")
// (Here `html` is an identifier, not the string "html".)

// 3) Numbers/booleans stay unquoted automatically
buildAction("configure", raw("FAST"), 3000, true);
// => configure(FAST, 3000, true)

// 4) Raw function calls / expressions
buildAction("setTime", raw("NOW()"));         // => setTime(NOW())
buildAction("scale", raw("width*0.5"));       // => scale(width*0.5)

// 5A) Combining with objects/arrays as ONE quoted JSON string
const obj = { a: 1 };
const arr = ["x", "y"];
buildAction("sendData", JSON.stringify(obj), JSON.stringify(arr));
// => sendData("{\\\"a\\\":1}", "[\\\"x\\\",\\\"y\\\"]")

// 5B) If your command language supports raw object/array literals, wrap them:
buildAction("sendData", raw("{a:1}"), raw("['x','y']"));
// => sendData({a:1}, ['x','y'])

Parameters:

Name	Type	Attributes	Description
`fn`	string		Function/command name.
`args`	*	<repeatable>	Positional arguments.

Returns:

Serialized call, e.g. "do("hi", 42, ENUM.A)".

Type: string

(static) clickSync(text) → {boolean|null}

Click the buttons on the current device screen.

Example

var { Device } = require('sigma/device');
var aai = require("sigma/aai");
// Click the "Settings" button on the screen of the current device
var sigmaDevice = Device.getMain();
var ret = sigmaDevice.clickSync("Settings");
if (ret === true) {
    print("Click successfully.");
} else {
    print("Failed to click: " + lastError());
}

// Operation Result:
// If it executes successfully, it will return:
Click successfully.

Parameters:

Name	Type	Description
`text`	string	The visible text of the UI element to click, Total Control will first find the component with the text (pick "text" first then "description"). (e.g., `"Settings"`).

Returns:

true if the click is performed successfully.
false if the element is found but the click fails.
null if the element cannot be found or an error occurs (check lastError()).

Type: boolean | null

(static) deleteQueryListener(name) → {Boolean}

Delete the specified monitoring event for the current device

Parameters:

Name	Type	Description
`name`	string	Defined AAI event key

Returns:

Returns true if successful, false if it fails, and the specific error message is obtained by the lastError() function.

Type: Boolean

(static) findObject(device, query) → {UiElement|null}

Use UI Explorer to take a screenshot of the current device and generate multiple nodes.

Each node is tied to a element (and bounds) on the screen:

For example, text fields, buttons, items in lists, pictures, layouts, groups, each of which is represented by a node.Furthermore, each node will provide rectangular "bounds" that consists of the coordinate and size of the UI element.

By using this API,Total Control will send the query to device, find the matching node and create an object for it.The return object will provide you with a series of method to deal with the nodes.

Note: UI Explorer instructions can refer to this chapter document (UI Explorer for Total Control )

Currently Total Control provides two methods to obtain UiElement object.

Method 1: Obtain the UiElement object manually by calling sendAai()

sendAai code template：
var cm = sendAai({"elements":["nodeID"], "action":"getUniqQuery"});
query = cm.query
You can get the "nodeID" value through UIExplorer.

Method 2: Obtain the UiElement object through UiExplorer "Code" button.

The "Code" button will provide a Query generator to display a query that will match the selected node. You can cut-n-paste to the JavaScript code to create UiElement object.
The object of the node "scan" on the screen is automatically generated.

Example

// Example 1: Obtain the "scan" object in Total Control APP, and call the method clickSync()
var { Device } = require("sigma/device");
var { UiElement } = require("sigma/aai");
var sigmaDevice = Device.getMain();
var cm = sendAai({"elements":["153b8"], "action":"getUniqQuery"});
var query = cm.query;
var obj = UiElement.findObject(sigmaDevice, query);
var ret = obj.clickSync();
if (ret == true) {
    print("Click successfully.");
} else {
    print("Failed to click: " + lastError());
}

// Expected Output:
Click successfully


/ Example 2: Get text field element information
var { Device } = require("sigma/device");
var { UiElement } = require("sigma/aai");
var sigmaDevice = Device.getMain();
var obj = UiElement.findObject(sigmaDevice, "T:Text Message");
print(obj.getText());
print(obj.getId());
print(obj.getDescription());
print(obj.getPackageName());
print(obj.getBounds());
print(obj.getResourceId());
print(obj.getClassName());

// Expected Output:
Text Message
e32b
undefined
com.android.mms
l:190, t:1803, r:836, b:1884
com.android.mms:id/embedded_text_editor
android.widget.EditText

// Example 3: Screenshot the "Camera" element and save to file
var { Device } = require("sigma/device");
var { UiElement } = require("sigma/aai");
var sigmaDevice = Device.getMain();
var obj = UiElement.findObject(sigmaDevice, "T:Camera");
var ret = obj.screenshot("E:/test/camera.bmp", 1);
if (ret == 0) {
    print("Screenshot successfully.");
} else {
    print("Failed to screenshot: " + lastError());
}

// Expected Output:
Screenshot successfully.

Parameters:

Name	Type	Description
`device`	Device	The target device.
`query`	string \| Object	A selector string (e.g. `"T:Text Message"`) or a query object (e.g., from `sendAai()`).

Returns:

The matched UI element object, or null if not found.

Type: UiElement | null

(static) findObjects(deviceOrDevices, query) → {UiElementArray}

This interface is used for multiple or single devices to obtain UiElmentArray. By calling a series of methods of UiElmentArray, multiple devices can perform operations such as click/inputText in batches.

Reference document: UiElementArray UiElement.findObject

Note: The interface judges whether there is a component that meets the conditions in the interface through input parameters, if there is, it will return the result of UiElementArray object.

If the parameter is device, it returns all components that meet the conditions in a single device;
If the parameter is devices, it returns the first component that satisfies the condition among multiple devices

Example

// Example 1: Multi-device search and click the "Settings" icon
var { Device } = require("sigma/device");
var { UiElement } = require("sigma/aai");
var sigmaDevices = Device.searchObject(tcConst.devAll); // Search all connected devices
var uiElmentArray = UiElement.findObjects(sigmaDevices, "T:Settings");
uiElmentArray.clickSync();

// Expected Output:
UiElement: 3fd06, UiElement: 3ee02 //Multi-device

// Example 2: Single-device search and click the "Settings" icon
var { Device } = require("sigma/device");
var { UiElement } = require("sigma/aai");
var sigmaDevice = Device.getMain(); // Get the main device
var uiElmentArray = UiElement.findObjects(sigmaDevice, "T:Settings");
uiElmentArray.clickSync();

// Expected Output:
UiElement: 3fd06 //Single-device

### Note:
- The query string `"T:Settings"` searches for elements with text "Settings".
- The `clickSync()` method will attempt to click each matched element.

Parameters:

Name	Type	Description
`deviceOrDevices`	Device \| Array.<Device>	A single `Device` instance or an array of devices.
`query`	string \| Object	A selector string (e.g., `"T:Settings"`) or a complex query object.

Returns:

An array-like object containing matched UI elements.

Type: UiElementArray

(static) inputTextSync(position, text) → {boolean}

input text content in the specified text field on the current device screen. After entering the new text content, the previous content of the text field will be overwritten by the new content.

Example

var { Device } = require('sigma/device');
var aai = require("sigma/aai");
// input the text "Total Control" in the second text field on the current device screen
var sigmaDevice = Device.getMain();
var ret = sigmaDevice.inputTextSync(1, "Total Control");
if (ret === false) {
    print("Failed to input text: " + lastError());
} else {
    print("Input text successfully.");
}

// Operation Result:
// If it executes successfully, it will return:
Input text successfully.
// Otherwise the output is:
Failed to input text: "Error message".

Parameters:

Name Type Description

Name	Type	Description
`position`	number	Text field number: 0 represents the first text field on the current device screen; 1 represents the second text field, recursively... If this parameter is omitted, 0 is used as the default parameter. New in UP30: Input according to the label name (see example)
`text`	string	The text string to input into the specified field.

position

number

Text field number:

0 represents the first text field on the current device screen;
1 represents the second text field, recursively...
If this parameter is omitted, 0 is used as the default parameter. New in UP30: Input according to the label name (see example)

text

string

The text string to input into the specified field.

Returns:

true if the input action is completed successfully.
false if the input action fails (check lastError() for the error details).

Type: boolean

(static) listQueryListener(device) → {Array}

Return all monitoring events registered for the current device

Parameters:

Name	Type	Description
`device`	string	Singal device object

Returns:

Return the collection of all monitoring events registered for the current device in Array type

Type: Array

(static) restartAppSync(packageName, queryopt) → {boolean}

Restart the App that exists in the current device by the App package name.

Example

// Example 1: Restart Baidu Map app
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
var ret = sigmaDevice.restartAppSync("com.baidu.BaiduMap");
if (ret === true) {
    print("Restart App successfully.");
} else {
    print("Failed to restart App: " + lastError());
}


// Example 2: Restart using a query object
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
var cm = sigmaDevice.sendAai({ action: "getUniqQuery", elements: ["17426"] });
var query = cm.query;
var ret = sigmaDevice.restartAppSync("com.baidu.BaiduMap", query);
if (ret === true) {
    print("Restart App successfully.");
} else {
    print("Failed to restart App: " + lastError());
}

// Operation Result:
// If it executes successfully, it will return:
Restart App successfully.
// Otherwise:
Failed to restart App: "Error message".

Parameters:

Name	Type	Attributes	Description
`packageName`	string		The package name of the app to restart (e.g., `"com.baidu.BaiduMap"`).
`query`	object	<optional>	The parameter is the query content (query can be obtained through sendAai())

Returns:

true if the app was restarted successfully.
false if the restart failed (check lastError() for details).

Type: boolean

(static) runAppSync(packageName, queryopt) → {boolean}

Start the App that exists in the current device by the App package name.

Example

// Example 1: Launch the Total Control app
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
var ret = sigmaDevice.runAppSync("com.sigma_rt.totalcontrol");
if (ret === true) {
    print("Run App successfully.");
} else {
    print("Failed to run App: " + lastError());
}



// Example 2: Launch an app using a query object
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
var cm = sigmaDevice.sendAai({ action: "getUniqQuery", elements: ["17426"] });
var query = cm.query;
var ret = sigmaDevice.runAppSync("com.sigma_rt.totalcontrol", query);
if (ret === true) {
    print("Run App successfully.");
} else {
    print("Failed to run App: " + lastError());
}

// Operation Result:
// If it executes successfully, it will return:
Run App successfully.
// Otherwise:
Failed to run App: "Error message".

Parameters:

Name	Type	Attributes	Description
`packageName`	string		The package name of the app to run (e.g., `"com.sigma_rt.totalcontrol"`).
`query`	object	<optional>	The parameter is the query content (query can be obtained through sendAai())

Returns:

true if the app was launched successfully.
false if the launch failed (check lastError() for details).

Type: boolean

(static) sendAai(params) → {object|null}

TC provide "device.sendAai()" to communicate in FindNode way.For more information about the use of sendAai and FindNode syntax, please refer to the document FindNode

Example

// Example 1: Get bounds of specific layout components
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
var ret = sigmaDevice.sendAai('{"childCount":0,"className":"!/Group|Layout/", "action":"getBounds"}');
if (ret == null) {
    print("Failed to get result: " + lastError());
} else {
    print(JSON.stringify(ret));
}

// The execution is successful, the output is:
{bounds: [[36,72,288,366],[288,72,540,366],[576,102,756,282],[540,282,792,349],[792,72,1044,366],[36,366,288,660],[324,396,504,576],[288,576,540,643],[540,366,792,660],
[792,366,1044,660],[36,660,288,954],[288,660,540,954],[112,1547,967,1589],[36,1626,288,1902],[288,1626,540,1902],[540,1626,792,1902],[792,1626,1044,1902]], count: 17,
ids: ['e08e','e44f','181f9','185ba','ebd1','ef92','1897b','18d3c','f714','fad5','fe96','10257','957a','b382','b743','bb04','bec5']}
// Otherwise the output is:
Failed to get result: "Error message".


// Example 2: Query next element vertically after 'Bluetooth'
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
var ret = sigmaDevice.sendAai({
    query: "T:Bluetooth||OY:1",
    action: "getNodes",
    fields: "T"
});
print(JSON.stringify(ret));

// Operation Result:
// If the execution is successful:
{ count: 1, list: [{ id: '1739b', text: 'Mobile network' }] }
// Otherwise:
Failed to get result: "Error message"


// Example 3: Wait for 'Bluetooth' and then click
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
sigmaDevice.sendAai({
    actions: ["newQuery('T:Bluetooth',5000)", "click"]
});


// Example 4: Query by field types and perform clicks
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
sigmaDevice.sendAai({ query: "T:Scan now", action: "click" });      // Match text
sigmaDevice.sendAai({ query: "D:Settings", action: "click" });      // Match description
sigmaDevice.sendAai({ query: "C:.Button", action: "click" });       // Match class ID
sigmaDevice.sendAai({ query: "R:.settings", action: "click" });     // Match resource ID


// Example 4.1: Combine query conditions
var { Device } = require('sigma/device');
var aai = require("sigma/aai");
var sigmaDevice = Device.getMain();
sigmaDevice.sendAai({ query: "C:.Button&&R:.settings",action: "click" });     // Match class + resource ID
sigmaDevice.sendAai({ query: "D:Settings&&R:.settings", action: "click" });    // Match description + resource ID


// Example 5: Click a partial text using wildcard
var aai = require("sigma/aai");
device.sendAai({
    query: "T:Allow to connect to the intern*&&OX:1",
    action: "click"
});

Parameters:

Name Type Description

Name	Type	Description
`params`	object \| string	A JSON object or string describing the UI interaction to perform. Common fields: `query` `{string}` – Node query string (e.g., `"T:Bluetooth"` for text match). `action` `{string}` – Action to perform after the query (e.g., `"click"`, `"getBounds"`). `actions` `{string[]}` – Composite AAI action list (e.g., `["newQuery('T:Bluetooth',5000)", "click"]`). `fields` `{string}` – Result field filters (e.g., `"T"` to return text only).

params

object | string

A JSON object or string describing the UI interaction to perform.
Common fields:

query {string} – Node query string (e.g., "T:Bluetooth" for text match).
action {string} – Action to perform after the query (e.g., "click", "getBounds").
actions {string[]} – Composite AAI action list (e.g., ["newQuery('T:Bluetooth',5000)", "click"]).
fields {string} – Result field filters (e.g., "T" to return text only).

Returns:

If the execution succeeds, returns a result object; otherwise null and sets lastError() with the error message.

Type: object | null