Screenshot to text recognition OCR

T

tRanChor

New member
I'd like to read the screen using text recognition OCR from a plugin or another app, by taking a screenshot using the action in macrodroid then immediately using OCR on the screenshot, then output the text variable either in the form of string or dictionary.

TouchTask has OCR action but I can't figure out a way to use the latest image as the target for the OCR action. Can someone help me with this?

The action flow in Macrodroid would be:
> Take screenshot
> Read screenshot with OCR app/plugin
> Store text in string or dictionary
> Check if string or dictionary contains target word/substring


Additional bonus requests:

It would be more awesome if you can also tell me the easiest way to loop through a dictionary or string, then see if the word/substring ur looking for exist.

I'll be even happier if u can also tell me how to check a certain x,y pixel in the screen to see if it has a certain color.
 
Endercraft

Endercraft

Moderator (& bug finder :D)
tRanChor said:
I'd like to read the screen using text recognition OCR from a plugin or another app, by taking a screenshot using the action in macrodroid then immediately using OCR on the screenshot, then output the text variable either in the form of string or dictionary.

TouchTask has OCR action but I can't figure out a way to use the latest image as the target for the OCR action. Can someone help me with this?

The action flow in Macrodroid would be:
> Take screenshot
> Read screenshot with OCR app/plugin
> Store text in string or dictionary
> Check if string or dictionary contains target word/substring


Additional bonus requests:

It would be more awesome if you can also tell me the easiest way to loop through a dictionary or string, then see if the word/substring ur looking for exist.

I'll be even happier if u can also tell me how to check a certain x,y pixel in the screen to see if it has a certain color.
Click to expand...
Do you know an OCR plugin ? if so try using the plugin action (search for plugin in actions and you should find it). I have never seen someone on this forum come up with a solution.
 
T

tRanChor

New member
Unfortunately Shell script isn't working for me, so the OCR doesn't work too. There's also the "Shell script timed out" on logs.


I've tested it multiple times and the variables are still empty.
Dimlos said:
I have created a sample macro to OCR the latest screenshot.
Click to expand...
 
D

Dimlos

Well-known member
Does the MacroDroid Helper log show any errors?
The destination for saving screenshots depends on the model.
My macro is designed to be saved in Pictures/Screenshots, so if it is saved in DCIM, it will not work unless you rewrite it.
 
T

tRanChor

New member
The image variable seems to be working now, but the OCR text variable still remains empty. Screenshot_20230114-092537.pngScreenshot_20230114-092546.pngScreenshot_20230114-092554.png


Dimlos said:
Does the MacroDroid Helper log show any errors?
The destination for saving screenshots depends on the model.
My macro is designed to be saved in Pictures/Screenshots, so if it is saved in DCIM, it will not work unless you rewrite it.
Click to expand...
 
T

tRanChor

New member
Nvm it seems to be working now for some reason. Thanks for the help.

Do u also know how to check if the string contains target susbtring?
 
T

tRanChor

New member
Nvm, ChatGPT already gave me the answer. Now I can click buttons by text on custom canvas HAHAHAHAHAHAHA.

Tho it would be even more convenient if the latest screenshot automatically deletes after the OCR.

Screenshot_20230114-095033.png

tRanChor said:
Nvm it seems to be working now for some reason. Thanks for the help.

Do u also know how to check if the string contains target susbtring?
Click to expand...
 
T

tRanChor

New member
Lol I tried "rm $img" shell script but it didn't work, so I just made the macro click the delete button that appears after taking a screenshot.Screenshot_20230114-101929.png
 
MacroDroidDev

MacroDroidDev

Administrator
Staff member
I know some users turn off auto updates, but I'm always a little suspicious of people with out of date versions as the majority of (certainly not all) bug reports from old versions are because they are pirate versions.
 
D

Dimlos

Well-known member
cza93 said:
Dont work for me... Anything changed?
Click to expand...
I tried the macro and it works.
It is OCR, but basically you have to use English to make it work.
If you set the language of MacroDroid to English and try it, it might work.
 
C

cza93

New member
Dimlos said:
I tried the macro and it works.
It is OCR, but basically you have to use English to make it work.
If you set the language of MacroDroid to English and try it, it might work.
Click to expand...
Even forcing the language to english (mine is portuguese) it doesn't work.

Screenshot_20230611-205826.png
 
D

Dimlos

Well-known member
It worked on OPPO with Android 12 but not on Xiaomi with Android 13.
It is not likely to recognize Portuguese anyway, so it may be better to give up on TouchTask.
I think Google Lens' character recognition function can probably recognize Portuguese.
 
You must log in or register to reply here.
Top