Any way to add OCR/image recognition?

[Shajirr]

What I learned is that none of the methods in the Click action work on game apps.
So can't read text, can't search for elements, etc.

Any way to add OCR/image recognition capabilities instead? Like taking a screenshot,
sending it to some third-party app which will scan it, then return the data which you can load into variables, or something similar?

For many things I'd very much like to know what is being clicked,
as relying on just coordinates in those cases is not possible.

As an example, on PC I used Vis2 library for AHK, which in turn used Tesseract OCR engine

 

[Endercraft]

To give you an idea when it will be added, at least not until 2026 (unless everything goes very well).

 

[sampleuserhere]

It's doable now, there is ML kit on Android.

Afaik, The kit requires google play service to work. The device doesn't always need to be online though, as long as the language data is installed. It supports non-Latin text too!

Recognize text in images with ML Kit on Android | Google for Developers

developers.google.com

Automate has this one implemented with Text Recognition block.

Text recognition ⋅ Automate for Android ⋅ LlamaLab

llamalab.com

Now the cool thing is that ML kit offers a bunch of other cool stuff as well, e.g Face recognition.

ML Kit | Google for Developers

Google's on-device machine learning kit for mobile developers.

developers.google.com

 

[Shajirr]

Google ML kit is for the devs, not users. No idea how to use any of that.

To give you an idea when it will be added, at least not until 2026 (unless everything goes very well).

so not any time soon then

Has no one ever used any third-party OCR engines / apps with MacroDroid?
Since it supports sending/receiving Intents, it should be able to communicate with such apps.

 

[Dimlos]

You can use Tesseract with Termux, but I'm not interested in OCR for games so I don't know if it will work for your application.

 

[sampleuserhere]

Google ML kit is for the devs, not users. No idea how to use any of that.

I thought you were referring to adding functionalities as in requesting a feature. I guess it wasn't the case, then?

Has no one ever used any third-party OCR engines / apps with MacroDroid?
Since it supports sending/receiving Intents, it should be able to communicate with such apps.

I previously mentioned that Automate (automation app by llamalab) could do this. It's free up to 30 running blocks.

In Automate, creates the following flow.

Receive broadcast (e.g ocr.start) > Take Screenshot > Text Recognition > Send Broadcast (e.g ocr.result)

Then in Macrodroid.

Send Intent (e.g ocr.start) > Wait for Trigger, Intent Received (e.g ocr.result)

Ask in here for further question about Automate.

https://www.reddit.com/r/AutomateUser/

 

[Shajirr]

In Automate, creates the following flow.

Receive broadcast (e.g ocr.start) > Take Screenshot > Text Recognition > Send Broadcast (e.g ocr.result)

Tested the OCR, and it works extremely well - it read all the texts on the page, even located on uneven gradients, of various font sizes, various fonts, serif fonts with thin edges, text that is so small that I can barely even read it myself, etc. And extremely fast, completes nearly instantly too.

With that in mind, I might just move to that program entirely, as the main script logic can be created in both programs, while screenshot taking in MacroDroid is really bad and OCR non-existent, so wouldn't make sense to depend on 2 programs instead of one.