Webhooks speed problem

[JPM]

I use webhooks extensively so Macrodroid "reads aloud" (text to speech) information from my home automation system, think "Garage door left open" or "Intruder by basement motion detector" or another use case; when I arm the alarm at night, set Android to silent/no vibrate until I actually wake up and dis-arm the alarm, this prevents middle of the night "your cars warranty has expired" spam calls from waking me.

I realize trigger.macrodroid.com is being run a free service to all of us, but I am thinking the server gets congested or unavailable at times? It's possible I'm crazy, but I think that sometimes I miss "read aloud" events. I've experimented with the audio output stream and I've come to realize that "Alarm" is most reliable...regardless of the phone being connected in car (android auto) or headset (bluetooth) or just in my pocket (internal speaker)

Is it possible to run an equivalent service to trigger.macrodroid.com locally, as in self-hosted? In my case I pay for a static IP at home (but I suppose a dynamic DNS service such as DuckDNS or No-IP could be used as well) ideally I would want to self-host this "trigger" application that the Macrodroid app is "listening?" to (on Windows, using IIS if applicable) so that it's super fast and private as opposed to the public server.

Can this be accomplish somehow? I would gladly be a BETA tester.

 

[Jacob L]

Can you not use notification received? Sorry if that seems no good I'm not 100% sure of your setup. Maybe changing your phone's DNS to a nearer server to make it faster. This can be done in settings\wireless\private DNS on Android 9+, maybe 10.

How is your internet speed for other devices and site? Wired is always faster than wireless, I bought a cat8 cable and made it even faster, even when we don't get gig speeds.

Have you followed steps at dontkillmyapp.com?

 

[MacroDroidDev]

If you have hard evidence that the trigger is missing events or is being severely delayed then I would certainly be very interested. There is no doubt the usage of this server is way above what I had originally expected but I don't see any red flags in terms of my server diagnostics. I don't see CPU usage going above 25%, disk I/O and bandwidth are also well within scope.

I'm not really a server guy and just learnt as I went with all this. If there are real problems I have various solutions that I could implemnet.
1) Just upgrade my virtual server to a higher spec.
2) Add limits to prevent people over hammering the server (maybe stricter limits on free vs pro)
3) Limit free users to only one webhook trigger (to prevent free users swamping the server)

I don't have any plans to offer a self host service as it's all integrated into my whole back end and it would be too much effort to unbundle and package this for the handful of users who would be interested.

 

[Jacob L]

Is it an adblocker, VPN, firewall or other connection related issue for webhooks?

Updating router, phone OS and apps will make everything run more smoothly

Is MacroDroid being killed by the OS? Dontkillmyapp.com

 

[scubanarc]

If you have hard evidence that the trigger is missing events or is being severely delayed then I would certainly be very interested. There is no doubt the usage of this server is way above what I had originally expected but I don't see any red flags in terms of my server diagnostics. I don't see CPU usage going above 25%, disk I/O and bandwidth are also well within scope.

I'm not really a server guy and just learnt as I went with all this. If there are real problems I have various solutions that I could implemnet.
1) Just upgrade my virtual server to a higher spec.
2) Add limits to prevent people over hammering the server (maybe stricter limits on free vs pro)
3) Limit free users to only one webhook trigger (to prevent free users swamping the server)

I don't have any plans to offer a self host service as it's all integrated into my whole back end and it would be too much effort to unbundle and package this for the handful of users who would be interested.

I'd like to help out with this effort. I want to run a private webhooks server just for my own use. This is something that I very much want to do.

I would be happy to tear the integration out of your server code if you don't have time and share the solution with anyone else who wants it, or share it with you to share with others. I can do it in Github, or privately.

To be absolutely clear, I do not want to make something that is useable by other "non-server" people... that's what your server is for. I want to make something that is useable by individuals for just their own use, but with your app as the front-end.

 

[scubanarc]

Is it an adblocker, VPN, firewall or other connection related issue for webhooks?

Updating router, phone OS and apps will make everything run more smoothly

Is MacroDroid being killed by the OS? Dontkillmyapp.com

It's none of those, webhooks work fantastically well.

I already have servers and I'm a developer. I'd like to run the webhooks through my own servers for a few reasons:

1 - Reliability - I know if my servers are up, and if they are not up then I know immediately. If the MacroDroid webhooks server is down, I'm likely to not get my notifications, but I also don't know that I'm not getting them.

2 - Longevity - I'm looking for a long-term solution where I don't have to worry about changes in the next 5 years. I cannot guarantee that the webhooks server will be up, but with my own servers, I can.

Realistically reason #1 is the big one. I need a solution that I can trust 100%. If it's running through your servers then I cannot. If it's running through my own servers then I can. And like I said... if my servers are down then I already know it.

 

[420]

It's none of those, webhooks work fantastically well.

I already have servers and I'm a developer. I'd like to run the webhooks through my own servers for a few reasons:

1 - Reliability - I know if my servers are up, and if they are not up then I know immediately. If the MacroDroid webhooks server is down, I'm likely to not get my notifications, but I also don't know that I'm not getting them.

2 - Longevity - I'm looking for a long-term solution where I don't have to worry about changes in the next 5 years. I cannot guarantee that the webhooks server will be up, but with my own servers, I can.

Realistically reason #1 is the big one. I need a solution that I can trust 100%. If it's running through your servers then I cannot. If it's running through my own servers then I can. And like I said... if my servers are down then I already know it.

Well... if the macrodroid webhook server would be down, would you not be able to tell by the response? so it could alert you if it's down?

I gues this depends on where/how you are triggering your webhooks, if it's from another device through macrodroids HTTP GET action you can simply save the succes state and response to variables and add triggers for those variables not being true/ok to send you a notification and or try it again 30 seconds later etc.. If you are triggering them in another way then i really don't know if you can also do something like this..

 

[scubanarc]

I just use curl in a bash script to hit the webhooks endpoint (with some URI variables). I can certainly detect if I don't get a good response back.

However, it's a bit of a chicken and egg from there. If the macrodroid webhook is down, and I use macrodroid on my phone to receive alerts, and my server notices that it's down, then how can it send me an alert? I'd have to write a whole side system for alerting myself that macrodroid is down.

 

[420]

I just use curl in a bash script to hit the webhooks endpoint (with some URI variables). I can certainly detect if I don't get a good response back.

However, it's a bit of a chicken and egg from there. If the macrodroid webhook is down, and I use macrodroid on my phone to receive alerts, and my server notices that it's down, then how can it send me an alert? I'd have to write a whole side system for alerting myself that macrodroid is down.

Well (assuming you have your own webhook server) how would your server alert you if the server itself is down? You said you would know immediately when it is down, but how would you know this? If you are in the same room i can see it, but how would your server notifiy you on your phone that it is down while it is down? Doesn't this give the same (potential) problem?

 

[scubanarc]

Great question...

To start with, the servers are almost never down. I've had perhaps a couple of outages in maybe 5 years. We have very good uptime. But that aside...

I know if my servers are down for a variety of reasons. The most important way is a phone call. A very, very bad phone call that punches through "Do not disturb" mode

Also, all kinds of things immediately stop working. Motion lights stop turning on, the music stops (literally), VPN connections die, our terminals freeze, the SQL monitor wigs out... it's pretty glaringly obvious.

 

[420]

I just use curl in a bash script to hit the webhooks endpoint (with some URI variables). I can certainly detect if I don't get a good response back.

However, it's a bit of a chicken and egg from there. If the macrodroid webhook is down, and I use macrodroid on my phone to receive alerts, and my server notices that it's down, then how can it send me an alert? I'd have to write a whole side system for alerting myself that macrodroid is down.

Great question...

To start with, the servers are almost never down. I've had perhaps a couple of outages in maybe 5 years. We have very good uptime. But that aside...

I know if my servers are down for a variety of reasons. The most important way is a phone call. A very, very bad phone call that punches through "Do not disturb" mode

Also, all kinds of things immediately stop working. Motion lights stop turning on, the music stops (literally), VPN connections die, our terminals freeze, the SQL monitor wigs out... it's pretty glaringly obvious.

How you are doing this i really do not know, so i can only imagine how hard and time consuming it would be to have this (or another) side system do the same things when the macrodroid webhook server is down, but you have answered your own question!

 

[scubanarc]

How you are doing this i really do not know

It's literally my job I own a web business, so servers are both a hobby and a job.

so i can only imagine how hard and time consuming it would be to have this (or another) side system do the same things when the macrodroid webhook server is down

I feel like the .APK does the heaving lifting, I just need a "hook" into the FCM.

but you have answered your own question

Not sure what you mean....

Do I need to arrange for someone to call me when the webhooks server is down?

Do I need to have a VPN drop when the webhooks server is down?

What part of "I know when my systems are failing" even remotely equates to "I'd like to host my own webhooks system".

I think it would be a fun side project. I don't think it will be that hard. I don't know how to make the .APK but I do know how to run servers. I'm not sure why this is hard, unless there are security implications to allowing me to send FCM messages that the app can catch.

 

[420]

Not sure what you mean....

Do I need to arrange for someone to call me when the webhooks server is down?

Do I need to have a VPN drop when the webhooks server is down?

What part of "I know when my systems are failing" even remotely equates to "I'd like to host my own webhooks system".

What i mean is that you already have a system in place that can notify you if your server is down, and you also have a way to detect if you don't get a good response back from a macrodroid webhook, so you'd just have to combine the two right? - get a bad response on a macrodroid webhook = run all (or some of) the stuff that now already runs when your server is down.

 

[scubanarc]

get a bad response on a macrodroid webhook = run all (or some of) the stuff that now already runs when your server is down.

I don't have anything that contacts me when the servers are down. The shit hits the fan when the servers are down and I immediately know it. This is not the same as "my curl got an HTTP error response". I have no system (other than webhooks) of immediately alerting my phone in this case.

Literally, webhooks is my preferred way of alerting me when things are not right. If they are down I have no other systems.

I like webhooks and I want to run them on a private server so that I know that they are working. That is all.

 

[MacroDroidDev]

Apologies but self host is not something that is going to be viable for me to consider in the short term.

 

[scubanarc]

Apologies but self host is not something that is going to be viable for me to consider in the short term.

I understand. I'm really looking to dev it myself and take as little of your time as possible.

Can you explain why it's difficult for me to self-host? Is it security-related?

Am I correct that the only way to send a FCM message to the MacroDroid app is to have the private API key?

Just looking to understand a little bit better.

 

[MacroDroidDev]

MacroDroid has to keep track of your FCM token which can be changed at any point and this is all associated in my back end with your device id in a database. Of course none of this is rocket science but I just don't have the time right now to unbundle anything for what is a very niche use case when I have 100s of very pressing things to do that will benefit many users.

 

[JPM]

I appreciate the response and thought that went into this - to be honest I forgot about this thread ... as I felt, rather un-scientifically, that webhooks were working better since my original post. I did upgrade a few things on my end, such as offloading some processes on my home automation server to another server, updating curl, and correcting some ISP issues that plague my neighborhood.

So in any case ... I can't determine if your end or my end is better - but it's working better.

Thanks!

 

[JPM]

I feel like in the last month, notifications have slowed again. I can't determine if it's a WiFi sleep issue, or sever response time.

 

[MacroDroidDev]

What sort of delays are you talking about here?