FindNode Examples

FindNode Examples

Please refer to FindNode User Guide for more information.


To get started with FindNode programming, you’ll need three components:

1. UI Explorer

(Main pane → Scripts → Tools → UI Explorer)
UI Explorer lets you inspect the properties of individual UI elements. You can also experiment with FindNode query syntax here by locating and verifying the nodes you’re interested in.

2. Terminal

(Main pane → Scripts → Terminal)
There are three Terminal windows available. The Terminal is used to try out FindNode programming interactively using its only command: sendAai(). This is a convenient way to test your queries directly against the connected device.

3. JavaScript (Rhino)

Use UI Explorer to generate your FindNode queries, use the Terminal to test them with sendAai(), and then use JavaScript (Rhino) to put everything together into reusable automation scripts.

SigmaTestApp

We have created an app called SigmaTestApp to help you try out our query syntax, sendAai(), and basic scripting.

If you have an Android device connected, you can launch it directly from Total Control:

… → Open SigmaTestApp

If SigmaTestApp is not installed on your device, Total Control will automatically install it and then launch the app.

Example nodes in UI Explorer

FindNode is node-based: most UI elements on the screen can be searched and actions can be performed on them.
Each UI element on the screen is represented as a node. A node:

  • is identified by a unique ID (unique on that screen), and
  • has a set of properties that you can use in your queries.

In the screenshot below, the EDITTEXT button in SigmaTestApp is selected. On the right side of UI Explorer, you can see its node information:

  • ID: a002
  • Package Name: com.sigma_rt.sigmatestapp
  • Class Name: .Button
  • Resource ID: .button5
  • Bounds: [53,714][543,837]
  • Text: EDITTEXT
  • Properties: for example clickable = true, enabled = true, checkable = false, etc.

These fields are the building blocks of your FindNode queries. In later examples, we will show how to locate this node (the EDITTEXT button) using its ID, text, class name, or other properties, and then perform actions on it with sendAai() and JavaScript.



Query

Basic Query

Basic queries apply to the properties of individual UI elements. The interactions with FindNode is thru "sendAai()"

Most basic query is an empty object:

var output = device.sendAai({});

The default template is "more" (unless you change it with setConfig), and the default action is "getIds". So the following is equivalent to the basic query above:

var output = device.sendAai({ query: "TP:more", action: "getIds" });

This returns most of the nodes on the screen. The default return of action: "getIds" is a list of node IDs, for example:

{ count: 89, ids: ["80006cbe","7440","7bc2","7f83","8344","8705","8ac6","8e87","1f6e7", …] }

Examples

Filtering by node properties

{ query: "CC:!=0" }          	// nodes whose child count is not zero
{ query: "CC:>0" }           	// same as above, child count > 0
{ query: "IT:>10000" }       	// nodes whose (custom) inputType value > 10000
{ query: "CC:!=0&&IT:>10000" }
// nodes with non-zero child count AND inputType greater than 10,000

Entering text

// enter "Hello" into the last editable text field
{ query: "BP:editable&&IX:-1", action: "setText(Hello)" }
// find an empty text field whose *initial* text is "Input text here" 
// (hint text shown as normal text) and replace it with "Hello"
{ query: "T:Input text here", action: "setText(Hello)" }
// find a field whose *hint text* is "Input text here" and enter "Hello".
// Unlike T, HT refers to hinted text only; setText does not overwrite 
// the hint itself. Better solution.
{ query: "HT:Input text here", action: "setText(Hello)" }

Matching multi-line text

device.sendAai({
    query: "T:*REAL, \nSANTA CLARA*",
    action: "getText"
});
// Example return:
{ retval: "9999 EL CAMINO REAL,\nSANTA CLARA, CA 95555" }

Scrolling and clicking

// Scroll until a match is found, then click the first matching node:
device.sendAai({
    actions: ["scrollIntoView(T:ci:john, down, 2);click"]
});

This example scrolls down up to 2 pages, searches for a node whose text contains john (case-insensitive), and clicks the first match once it is visible.


Extended Query

Extended queries operate across multiple nodes on the screen, allowing you to search patterns or relationships that span more than one UI element.


Example 1: Finding icons and labels on the bottom navigation bar, you can use UI Explorer, click screenshot, enter "TP:appNavBar" in query and press enter (TP:appNavBar is introduced in version 20).

>> device.sendAai({query:"TP:line,bottom,-1", action:"getText"})
{count: 5, retval: ['Alarms','World Clock','Timers','Stopwatch','Bedtime']}
>> device.sendAai({query:"TP:appNavBar,labels", action:"getText"})
{count: 5, retval: ['Alarms','World Clock','Timers','Stopwatch','Bedtime']

By using offset (OX/OY), index (IX) or query such as "T:", you can locate almost every node on the above examples, below 4 examples on click icon on top of the "Stopwatch":

>> device.sendAai({query:"TP:line,bottom,-1&&IX:3&&OY:-1", action:"getIds;click"})
{count: 2, list: [{count: 1, retval: ['3ee03']},{retval: true}]}
>> device.sendAai({query:"TP:appNavBar,icons&&IX:3", action:"getIds;click"})
{count: 2, list: [{count: 1, retval: ['3ee03']},{retval: true}]}

To find out what has been selected, use "BP:selected", since we click on icon, icon has been selected, verify the ID:

>> device.sendAai({query:"TP:appNavBar,icons&&BP:selected", action:"getIds;getText(OY:1)"})
{count: 2, list: [{count: 1, retval: ['3ee03']},{retval: 'Stopwatch'}]}


Example 2: To find out the Android release from About Phone, one second screen, enter "T:Android version&&OY:1", it will show 16:

>> device.sendAai({action:`openAndroidSetting(DEVICE_INFO_SETTINGS);
    click(T:Software information);
    refreshQuery;
    getText('T:Android version&&OY:1')`})
{count: 4, list: [{retval: true},{retval: true},{retval: true},{retval: '16'}]}
>>device.sendAai({action:`openAndroidSetting(DEVICE_INFO_SETTINGS);
    clickForNewWindow(T:Software information);
    getText('T:Android version&&OY:1')`})
{count: 3, list: [{retval: true},{retval: true},{retval: '16'}]}

To get the version:

>> var x = device.sendAai(...);
>> x.list.at(-1).retval
16


Example 3: Enter text in Teams text box, can discover from UI Explorer before and after entering text (may need 2 screenshots).

Before entering text:

After text is entered, need to click the button to send (2 offsets away: 0X:2):

This can be done by using in one query command, alternatively, can replace "TP:textInput" with "BP:editable".

>> var input = "Hello";
// Use backtick
>> device.sendAai({query:"TP:textInput", actions:[`setText('${input}')`, 'click(OX:2)']});
{count: 2, list: [{retval: true},{retval: true}]}

Another way is using variable substitution along with ";" as separator for multiple actions:

>> device.sendAai({query:"TP:textInput", actions:`setText('${input}');click(OX:2)`})
{count: 2, list: [{retval: true},{retval: true}]}

The following function run in Chat (or Contact) screen to find the person's name, click, send a message and return back to main page. In this case, "waitQuery" is required to ensure the text is sent before clicking Back key. 

>> function sendTeams(device, name, text) {
    var retval = device.sendAai({actions:[
        `scrollIntoView(T:${name})`, 
        "click", 
        "newQuery(TP:textInput)", 
        `setText('${text}')`, 
        "click(OX:2)",
        "waitQuery(HT:Type a message)",
        "sendKey(Back)"
    ]});
    if (retval == null) {
        print("Error: " + lastError());
        return false;
    }
    print(retval);
    return true;
}

To call the function:

>> sendTeams(device, "John", "Hello, how are you?")
{count: 7, list: [{retval: true},{retval: true},{retval: 1},{retval: true},{retval: true},{retval: true},{retval: true}]}

The above example can be done using "function" command in FindNode:

>> device.sendAai({action:"function(sendTeams)", sendTeams:[
    `scrollIntoView(T:%1)`, 
    "click", 
    "newQuery(TP:textInput)", 
    `setText('%2')`, 
    "click(OX:2)",
    "waitQuery(T:Type a message)",
    "sendKey(Back)"]})
{retval: true}

To call the function:

>> device.sendAai({action:"sendTeams(John, 'Hello, how are you?')"})
{retval: true}

To put this into script: restartApp(Teams) or openApp(Teams) and call sendSkype() to send message.

>> device.sendAai({action:"openApp(Teams);sendTeams(John, 'Hello, how are you?')"})
{count: 2, list: [{retval: true},{retval: true}]}


Example 4: This opens up Samsung calculator, obtain the node IDs of the buttons in calculator, create an element array from a formula and use "forEach" to click thru the element array to obtain the result.

var {getDevice} = require('sigma/device');
var idList = {};

function sendAai(obj) {
    var retval = getDevice().sendAai(obj);
    if (retval == null) {
        throw "Error on: " + JSON.stringify(obj) + ":" + lastError();
    }
    print("$ " + JSON.stringify(obj) + "\n" + JSON.stringify(retval));
    return retval;
}

function init() {
    sendAai({action:"openApp(com.sec.android.app.popupcalculator)"});
    var list = sendAai({query:"T:/^[0-9+=-]$/", action:"getNodes(T)"}).retval;
    list.forEach((p) => idList[p.text+""] = p.id);
}

function calc(formula) {
    var fList = formula.split("");
    var elemList = [];
    fList.forEach((p) => elemList.push(idList[p]));
    print(elemList);
elemList.push(idList['=']);
// nsClick does not wait for sync, sleep to ensure the keypresses are processed
    sendAai({elements:elemList, action:"forEach(nsClick);sleep(500)"});
    var result = sendAai({actions:"refresh(*R:.calc_edt_formula);getText"}).list[1].retval;
    return parseInt(result);
}

init();
print("Result is: " + calc("123+456"));

Can save the above code into a file (e.g."/tmp/excalc.js"), to run it

>> var exitCode = sigmaLoad('/tmp/excalc.js')
$ {"action":"openApp(com.sec.android.app.popupcalculator)"}
{"retval":true}
$ {"query":"T:/^[0-9+=-]$/","action":"getNodes(T)"}
{"count":12,"retval":[{"id":"15caa","text":"7"},{"id":"1606b","text":"8"},{"id":"1642c","text":"9"},{"id":"16bae","text":"4"},{"id":"16f6f","text":"5"},{"id":"17330","text":"6"},{"id":"17ab2","text":"1"},{"id":"17e73","text":"2"},{"id":"18234","text":"3"},{"id":"185f5","text":"+"},{"id":"18d77","text":"0"},{"id":"194f9","text":"="}]}
17ab2,17e73,18234,185f5,16bae,16f6f,17330
$ {"elements":["17ab2","17e73","18234","185f5","16bae","16f6f","17330","194f9"],"action":"forEach(nsClick);sleep(500)"}
{"count":2,"list":[{"count":8,"retval":[{"retval":true},{"retval":true},{"retval":true},{"retval":true},{"retval":true},{"retval":true},{"retval":true},{"retval":true}]},{"retval":true}]}
$ {"actions":"refresh(*R:.calc_edt_formula);getText"}
{"count":2,"list":[{"retval":true},{"retval":"579 Calculation result"}]}
Result is: 579


Example 5: To obtain the Tesla’s odometer reading, use the following commands:

Look for <number> miles:

>>  device.sendAai({action:"openApp(Tesla);scrollIntoView(T:/^[0-9,]+ miles$/);getText"});
{count: 3, list: [{retval: true},{retval: true},{retval: '9,999 miles'}]}

Look for the VIN and move up one line (OY:-1). The first matching format is very close to the ‘<number> mi’ battery-remaining display.

>>  device.sendAai({action:"openApp(Tesla);scrollIntoView(T:VIN:&&OY:-1);getText"});
{count: 3, list: [{retval: true},{retval: true},{retval: '9,999 miles'}]}

Mileage reading and VIN:

>>  device.sendAai({action:"openApp(Tesla);scrollIntoView(T:VIN:);getText(OY:-1);getText(OX:1)"});
{count: 4, list: [{retval: true},{retval: true},{retval: '9,999 miles'},{retval: 'XXXXXXXXXXXXXXXXX'}]}

TCHelp