I experienced the same issue on my Oppo Find X3 Pro when running Android 10 and 11.
I use gesture navigation and my understanding is that android 10 & 11 have known issues with apps that use gestures, multiple attempts and slow to respond - Nova for example includes a warning.
Recently the phone updated to Android 12 and I haven't experienced the issue. My phone settings are the same and I have to assume Google has improved gesture compatibility.
Swiping to open the drawer works every time with no delay.
As an aside in Nova, the up and down gestures on icons or folders work first time as well.