UI Interaction click question

[Yogi]

Hi there,
Tried a little around with the UI click action. It works well so far.

The problem I encountered:
There are single characters in a defined area on the screen.
If we click on one character it is removed to a new position
which is located above this area.
(you know "4 pictures 1 word"? Guess the word with fixed
character length from random characters)

We get the available characters with the action "Read Screen Contents".
The characters have no "ID" but a fixed rectangular area where they reside.

So far so well. Got automation working if there are no multiple same characters.
If there are multiple same characters the already clicked and moved character
gets back to its original position - this behaviour is not wanted.

Now the question:
Is there at the moment the possibility to limit the area where the "UI Click Action"->Text content
has to take place?
Or another solution?

Would be nice to get the screen position (x,y,width,height) of the characters through action
"Read Screen Contents" so it would be a possible solution with the action "UI Click Action"->X,Y location

Regards,
Fritz

 

[mapriex]

you can use the developer mode on your device, enable x,y coordinates and then hold a half second on the screen, where you want to know the x,y coordinates. you can read the coordinates at dx for x and dy for y afterwards.
if you have more coordinates to remember, make a screenshot or write the coordinates down. so you can use them in your macro.

 

[Yogi]

you can use the developer mode on your device, enable x,y coordinates and then hold a half second on the screen, where you want to know the x,y coordinates. you can read the coordinates at dx for x and dy for y afterwards.
if you have more coordinates to remember, make a screenshot or write the coordinates down. so you can use them in your macro.

Thanks for your reply
The problem get not solved through knowing the position of the characters only - if the word is solved the
next random characters are there and you don't know which position belongs to which character.

I was just curious how to manage this problem in MD, I have a working solution already in Automate.
Through try and error you touch sometimes the limits...

 

[mapriex]

Thanks for your reply
The problem get not solved through knowing the position of the characters only - if the word is solved the
next random characters are there and you don't know which position belongs to which character.

I was just curious how to manage this problem in MD, I have a working solution already in Automate.
Through try and error you touch sometimes the limits...

i don't know how the game works with "random positions", but it looks like the maximal characters are on the 12 start positions and then you need to check every goal position of the 12 goal positions. Finally you have to make 479 001 600 checks or ui getures to make every match possible to find the solution of the game.

anyway i don't know why you wanna automate a game, since the goal of a game is to have fun. maybe you quit with this game and play another one

 

[Yogi]

i don't know how the game works with "random positions", but it looks like the maximal characters are on the 12 start positions and then you need to check every goal position of the 12 goal positions. Finally you have to make 479 001 600 checks or ui getures to make every match possible to find the solution of the game.

anyway i don't know why you wanna automate a game, since the goal of a game is to have fun. maybe you quit with this game and play another one

So did I explain wrong or gave too less information
The goal is to get out of the 12 random characters a word from 2 till about 10 characters.
If I don't get it in some time, I let the macro do this job:
get the characters available
get the length of the word to be found
run a website with Synonyms
there you get a list with possible words
selecting the word and let the macro do the rest - fill in ....

as said I have a running solution in Automate, but want to know how MD compares.
Playing is the easy way for learning because of fun,
so let's have fun

 

[Endercraft]

Maybe if you showed the solution with Automate we could help you recreate it.

 

[Yogi]

Maybe if you showed the solution with Automate we could help you recreate it.

It is rather complicated at the first glance because of using xPath expressions,
but I don't want to get into details, the flow looks like:

The variable txt contains a word e.g. "WINDOW"
The set Variable txt1 block consists of the following:
"//android.view.View[@android:contentDescription=\"" ++ char(txt[index]) ++ "\" and (substring-before(@android:layout_height,\"p\")>100 and substring-before(@android:layout_height,\"p\")<180) and (substring-before(@android:layout_y,\"p\")>2000 and substring-before(@android:layout_y,\"p\")<2500)]"

So I limited because of the absence of an ID the area where the characters are: layouty>2000px and <2500 px
As you can see you have to know the screen content - and hope it never changes

My part is finished here...

 

[mapriex]

It is rather complicated at the first glance because of using xPath expressions,
but I don't want to get into details, the flow looks like:
View attachment 7991
The variable txt contains a word e.g. "WINDOW"
The set Variable txt1 block consists of the following:
"//android.view.View[@android:contentDescription=\"" ++ char(txt[index]) ++ "\" and (substring-before(@android:layout_height,\"p\")>100 and substring-before(@android:layout_height,\"p\")<180) and (substring-before(@android:layout_y,\"p\")>2000 and substring-before(@android:layout_y,\"p\")<2500)]"

So I limited because of the absence of an ID the area where the characters are: layouty>2000px and <2500 px
As you can see you have to know the screen content - and hope it never changes

My part is finished here...

Is there at the moment the possibility to limit the area where the "UI Click Action"->Text content

as i know you can't select a specific part of a screen, where you can choose from.

Would be nice to get the screen position (x,y,width,height) of the characters through action
"Read Screen Contents" so it would be a possible solution with the action "UI Click Action"->X,Y location

not possible yet.

Or another solution?

you can only use the the full "read screen contents". extract to another script language or as text and filter variables that only have one letter. Now you have all letters as a variables and you can work with them what you want.

but i guess, its not exactly what you want.

 

[Yogi]

but i guess, its not exactly what you want.

Right

 

[Yogi]

For me the best solution is to have almost anything in one app.

At the moment I see there are apps who are really genious from
different point of views.

I like Automate with its visual components, having FTP etc.

On the other side
I like the concept of MD's triggers, actions and (individual) constraints.

So I combine the 2 worlds and also the widget app KWGT for what
I need or like.

It's somehow like which programming language is best. One likes this,
one that.

The best is what fits your needs

 

[mapriex]

So I combine the 2 worlds and also the widget app KWGT for what
I need or like.

It's somehow like which programming language is best. One likes this,
one that.

The best is what fits your needs

i think this is the current "easiest" solution to combining programming with android interaction as third party user in my opinion.


i am used with python language in private usage, not because it is so easy, hmmm also easy. the most important thing is the open source feature with its many libraries you can choose from.