Realtime OpenCV software virtual gimbal over network for openHD/wifibroadcast/wfb-ng FPV, etc

Описание: Disclaimer: I'm not a developer, just a tinkerer. I hope some actual developers might pick this up and run with it

This is a proof of concept virtual gimbal or "realtime gyroflow" for time-sensitive live FPV video using IMU data sent over udp sockets to the groundstation which is running a simple opencv python script receiving that data and applying it with cv2.warpAffine and imutils.rotate.

I have DJI FPV for all my whirlythings but for ground stuff I like being able to hack my own solutions and eventually want to build a dual screen off-road RC car racing station controlled with xbox style controllers. Would be nice to build custom graphics (e.g. race track overlay, RPM, speed, etc). For now I'm using a brushless gimbal on the fpv cars but it is tricky to make it work and then the bearings seem to loosen up with all the bumps then things get all shaky again.

I'm playing with this imu-driven virtual gimbal idea because it doesn't seem possible to leverage pure computer vision software to stabilize sub 100ms real-time streaming video for FPV racing ground vehicles, at least not without crazy processing power at the receiving end. Using imu data and opencv to rotate and shift the image around requires comparably little processing power and adds no discernable delay to the video stream. This is all well-known and there are plenty of scholarly articles about it and I'm sure there are existing implementations but I couldn't find any that I could use.

I would think that ideally the imu data should be fused into the video stream as metadata in gstreamer to keep them perfectly in sync frame by frame but maybe syncing is irrelevant since we're doing all of this over wifi radios in monitor mode. Writing plugins for gstreamer is way over my head at this point.

As it sits in this video this separate-stream method could work for slow things like rock crawlers and cruising around but I need to play with filtering so it doesn't lose sync on fast movements. I found that if I over-filter the imu data I can delay it enough to match the video stream for slow and medium speed movements but for anything faster the opencv rotate expectedly lags behind the video due to the filter. I'll keep playing with it.

Would be super cool to just hard mount a raspicam or usb camera to a car and let the IMU and some lightweight software do the rest.