Sound desync with video

Ok, first some background:
I have a wireless joystick and throttle setup using a raspberry pi4 with VirtualHere server. I decided I wanted audio on it to. Through lots of experimentation, I was able to get quite reliable audio in and out by having the Pi connect to a dedicated 5ghz router connected to a dedicated ethernet adapter on my PC. The USB audio I've been using is quite old, only 16bit, 48khz quality, but the popping is quite minimal and the audio has never lost sync with video.

Now I have a much higher quality audio device I want to use. the problem is that audio from this device seems to slowing lose sync with video, eventually making the delay too jarring to watch. Even with the bitrate and sampling rate set exactly the same as the old device, the audio still slowly loses sync.

So Now, I've been changing settings on the router and playing with compression levels on the server and nothing helps. Transmitting audio seems to use more CPU on the Pi than the old audio, but it still doesn't go any higher than 25% according to htop on the Pi, so I don't think that's it.

Now I've noticed that when the Virtualhere client shows my old sound device, it displays a speaker icon next to it, indicating that it is a sound device. The new sound device only shows a star icon as it does for my joystick and throttle. Does the server or client normally detect when the device is for audio and handles data transmission differently as a result? If so, this would explain why the audio always loses sync. And if that's the case, is there any way to get the client or server to treat the device differently?

I'm at a loss at this point. So if anyone has any other idea, please throw them at me.

Also, the Pi is not running Raspbian. I have Ubuntu 19.10 installed and it is all up to date as of yesterday.

#2

I think you might have to use ethernet cable for the full connection rather than wifi from the pi to the router.

#3

I may have a solution already, but it came from the maker of the DAC I'm using. The device is very new and has some kinks to work out. They just released a firmware update for issues Apple devices are having, and it seems to have corrected my issue. Just finished watching some youtube videos as a test and the audio never went out of sync. I'll continue to test and post another update later.

I had been worried about the wifi working as well, but having a dedicated connection just for the Pi on 5ghz does wonders for performance. It may also help that I'm using a Mikrotik router, which is a bit above regular consumer routers.

#4

OK thats good new. Yes wifi will have variable latency is which not ideal when streaming isochronous data like audio/video. Its ok but the 5ghz wifi on the pi does not have a very good aerial and is only 1T1R which further limits range

#5

So it looks like it didn't totally solve my problem. but it has improved. Seems to actually depended on system load or something. Like if I start up an intensive game, at a certain point when loading, the audio will skip a bunch, then any video that is playing will be badly out of sync with audio. I know the WiFi on the Pi 4 isn't very good, but I'm using it in the same room. The point was just to eliminate cables strung too the chair I have the stick and throttle mounted on, so I didn't need much range. Having MIMO would probably help, but this issue seems signal and throughput independent, since setting my audio as low quality as possible drastically reduces network load, but the problem still persists.

I never ran into this issue with the old USB audio adapter I was using, so I'm starting to think it's driver and hardware related since the new DAC uses the USB Audio 2.0 driver and the old audio uses an older USB audio driver. The DAC is also a USB-C dongle, and while it works perfectly in windows, I'd bet more of the audio processing is being done on the software side, so any heavy load on the PC is causing issues when it communicates though the Virtual USB driver. It's all theory, but it sounds likely at least. I'll just have to keep experimenting. Maybe get a MIMO wireless adapter for the Pi instead and see if that helps.

#6

Also you could try this, in windows bring up Task Manager -> Details->Right click vhui64.exe->Set Priority-> High

Also get a good wifi 5Ghz ac adapter with 2T2R that works with the pi

Try just ethernet cable only to confirm its the limitation with the pi built-in wifi

#7

I connected the Pi through ethernet and tested it out. I could still get a very small amount of desync, even with the client set to high priority, but it took some work.

But I may have stumbled over the real problem. When trying more wireless configuration changes, I noticed and LED on the DAC change colors. The LED is set to indicate the quality level of the output it's providing: Purple for MQA, Red for over 48khz, and blue for 48khz and lower or when it's asleep. When the wireless connection got really bad, the light, which would show red for how I have it configured, changed to blue. It also changes to blue when I don't have it connected to the PC through the client. This was actually flickering between the two colors.

I think when the connection to the PC gets interrupted in some way, like a late data frame or excessive system load, the DAC is so sensitive to this that it is resetting it's output quality or going into a sleep state, and then switching back when it's talking to the PC again. I bet that's throwing everything with videos out of sync. It probably also happens so fast that I usually can't see that light change colors. That would also be why my older audio works fine, since it's never goes into any sort of reset or sleep state when it's connection is interrupted. I need another DAC that uses USB Audio 2.0 and doesn't like to go to sleep to verify it, but this might be the answer, which sucks. This little DAC sounds awesome and I got it specifically because it should be easy on power usage since I run the Pi off of a battery pack.

#8

Something else to try...

In Windows -> Power and Sleep Settings->Additional Power Settings->Change Plan Settings->Change Advanced Power Settings-> USB Settings->Selective suspend -> Disable

#9

Believe it or not, that's one of the first things I tried :). Didn't help. I was actually able to reproduce this in a big way by turning down the process priority to low for the client. This caused the LED on the DAC turn change color almost as soon as I started to load the game and video started to desync immediately. I'm pretty confident the DAC going to sleep is the cause. It just seems to need constant communication to stay awake.

The only thing I could think of was to set AutoAttachToKernel to 1 to try and get the DAC to stay awake by having it connect to the Pi kernel if it lost connection. Combined with setting the process priority for the client higher seems to reduce popping, but the desync is still there, just maybe a little less aggressive.