Device Manager sees mouse but can't use it

I'm trying to connect a usb mouse, the device manager sees it and shows that the device is working properly but moving the mouse does nothing. I have nothing to show from my logs that indicates there is an issue with the device. I have been able to pass through other devices (ex: mass storage) without issue. I'm at a loss for what my next step should be in troubleshooting the issue.

Things I've tried:
Using a different mouse
Using a different machine as the server (Tried rpi 3 running Raspbian and x64 machine running ubuntu server)
Using a different remote desktop
Combos of this.

Any help is appreciated.

#2

Can you try vnc server/client instead of remote desktop just to see if that works . I suspect remote desktop is somehow trying to re-redirect the mouse and its conflicting with the mouse being passed via virtualhere.