I lose other performers when I hit a big cymbal crash

[billthedrummer]

Describe the bug

I have a Roland VAD-506 e-kit, and after I upgraded to the 3.7 client, when I hit a big cymbal crash, I can't hear any of the performers any more. I'm also on a 3.7 server, so I'm not sure what's causing the problem. I have to stop and restart Jamulus. I haven't changed any of my volume settings.
To Reproduce

Hit a loud cymbal crash or double cymbal crash on an e-kit when in a session with other performers.

Expected behavior

It shouldn't kick off the other performers.
Screenshots

Operating system

Windows 10 client from OpenSource, Vultr server
Version of Jamulus 3.7

Additional context

I'm going to downgrade my client and see if that fixes the problem.

[gene96817]

1- when you hit a big cymbal crash, does your channel clip? How long does the clipping condition last?
2- does the condition clear 30 seconds after the clipping condition clears?
3-

I can't hear any of the performers any more

It shouldn't kick off the other performers.

That's two different conditions. In the first, you don't hear the other performers but they are still connected. Does their VU meter show activity? The second implies, everyone is disconnected and you are the only person connected?

4- after the crash, if the other performers are connected, turn off Settings/Enable Small Network Buffers and then turn it back on again. That should flush your buffers. Do you now hear the other performers?

[billthedrummer]

Hi thanks for the quick response. Sorry, that wasn’t clear. I do see their meters, I just don’t hear them any longer. My channel doesn’t clip, and it doesn’t clear. I had Enable Small Network Buffers turned off. I understood that didn’t matter. I turned them on, though, and will try that. From: gene96817 ***@***.***> Sent: Saturday, April 3, 2021 6:34 PM To: jamulussoftware/jamulus ***@***.***> Cc: Bill Pearce ***@***.***>; Author ***@***.***> Subject: Re: [jamulussoftware/jamulus] I lose other performers when I hit a big cymbal crash (#1445) 1- when you hit a big cymbal crash, does your channel clip? How long does the clipping condition last? 2- does the condition clear 30 seconds after the clipping condition clears? 3- I can't hear any of the performers any more It shouldn't kick off the other performers. That's two different conditions. In the first, you don't hear the other performers but they are still connected. Does their VU meter show activity? The second implies, everyone is disconnected and you are the only person connected? 4- after the crash, if the other performers are connected, turn off Settings/Enable Small Network Buffers and then turn it back on again. That should flush your buffers. Do you now hear the other performers? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#1445 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ATQYGVPHR74NDN4LCZ2UBHLTG66WVANCNFSM42K2ID3A>.

[gene96817]

I suggest turning on Enable Small Network Buffers, at the largest setting. Then when you get this condition, set the buffers to the smallest setting for 5 seconds and then set it back to the large setting. (The other sequence, small to large back to small, might not work.

Also, when you don't hear the other singers, do their meters move?

[dcorson-ticino-com]

I'm not sure how that's going to help when I'm in the middle of a performance.

As you are the only person we know of with this problem, you are going to have to help us find out where this is coming from, performance or not.

Please answer our questions. Are the others still online, do their meters move, do your meters move, does changing the buffer settings restart the sound ? Do the others have this sound interruption ?
This information will help us to figure out what is wrong and to find a solution so it stops happening and you can play without interruptions.

[gilgongo]

@dcorson-ticino-com META: We will I hope shortly be adding some guidelines that we can link to for helping people posting in the discussions :-)

[billthedrummer]

I answered your question about the meters - see the response just below this. You said it was a problem with the audio interface, which I haven't changed. I will try the suggestion about buffers, and try turning the volume up and down. I just haven't had a chance yet, so please bear with me. And if I'm not following guidelines for posting, please let me know what I'm doing wrong.

[hoffie]

I think everything is fine and people just wanted to clarify that this is a very specific problem where, unlike more generally-reproducable issues, most of the debugging work will have to be done by you as the only known person which is affected. :)

If every possible workarounds/remedies have been tried, I would still be interested in getting a packet dump (Wireshark) from the traffic when the problem occurs.

[billthedrummer]

Thanks. I'll try the other suggestions first. Hopefully that will fix the problem and/or give you more information.

[dcorson-ticino-com]

I do see their meters, I just don’t hear them any longer. My channel doesn’t clip, and it doesn’t clear.

That sounds to me like a problem with your audio interface. If Jamulus doesn't even clip and the meters run normally it doesn't look like a Jamulus problem. When Jamulus restarts it also resets the audio interface.
What happens if you turn down the input volume some?
When you say restart Jamulus, you mean the client or the server or both ?

[billthedrummer]

I use a Studio 24c, and haven't changed anything for a long time. Same interface, same drivers, same PC. I checked that the input and output was set correctly this morning, and it is.

[ghost]

Please investigate this by turning your input volume up, and also down, to see if this causes the problem to increase or decrease. A digital signal can clip or wrap around if it is too large... this can affect DC levels and create mayhem.

[DominikSchaller]

You're saying it only happens after upgrading to 3.7, right? Before everything worked.

[billthedrummer]

Yes. However, I downgraded to 3.6.2 and it happened during a practice this morning. Inconsistent though - I did three big double-cymbal crashes, and everything was fine. Then in the middle of a song, I hit one single crash pretty hard, and I couldn't hear the other performers. I didn't try resetting the buffers, though. Was trying to get back on and finish up a practice.

[bspratt]

This was all happening on the WorldJam "Studio Seattle" server running 3.7.0 built from the release source tarball. I'm rolling it back to 3.6.2 built from source to see if thing improve for Bill. I've also noticed a couple of mysterious jamulus restarts that didn't used to happen, so I'm curious to see if that stops as well.

I should probably start a new discussion for this, but it would be nice to have a .deb for 3.6.2 in case folks want to roll back from 3.7.0, and didn't build it from source.

[dcorson-ticino-com]

This will be interesting to see.
But if it was the server wouldn't the others online be effected too?
3.7 server has been stable for me, but most of the time I have been running the dev version for delay-pan.
It is not necessary that you changed something, hardware as well as software can develop problems.

[hoffie]

I've also noticed a couple of mysterious jamulus restarts that didn't used to happen, so I'm curious to see if that stops as well.

Do you have details on them? Are you seeing segfaults (dmesg)? Are you running via systemd? Is there anything interesting in systemctl status jamulus* or journalctl -u jamulus*?

It would be good to have a way to reproduce this. If we really assume this is about something odd in the audio stream, it would be helpful to have a recording of that sound. We could then check if the recording produces the crash as well. I really have a hard time to come up with theories what would lead to this.

[billthedrummer]

I don't know if a recording of the session would help, I was on Studio Seattle this morning and I think it was recorded. I did the three hard crashes pretty early on which didn't kick me off, and the one that did kick me off was probably around 11:15?

[hoffie]

Maybe a network traffic capture using Wireshark would help. The most realistic theory I can come up with: Your sound triggers a specific bit pattern which triggers some kind of intrusion prevention system on a network device somewhere between you and your server...

[bspratt]

Unfortunately recording wasn't on today (it is now). That's consistent with a restart, though - it's using --norecord, but I usually start it once the WorldJam show is over.

It's under systemd, yes. Nothing useful in the logs, unfortunately. Next time I think I've had a restart I'll check the logs and report back.

[dakhubgit]

It may be completely irrelevant to your use case, but I have this effect (not with a cymbal crash but with other hard clipping) because I am wearing wireless headphones (of the analog RF type, so no additional latency) where the sender somewhat annoyingly does automatic gain control with a rather long decay time. If you have any automatism in your setup somewhere of this kind, it may trigger?

[billthedrummer]

Thank you! I'm sort of happy to hear that, although sorry it's happening to you. Unfortunately, not the same use case. I'm using wired headphones. Since the cases are limited, it does sound like an issue caused by some weirdness in the front-end, although I'm not sure why Jamulus would react the way it does. I have an intense, short practice tonight, but hope I can test out some of the solutions.

[bspratt]

OK, here's a data point: I can see from systemctl that my 3.7.0 server "PNWJam Tue8pmSun730p" (Any Genre 2) restarted Mon 2021-04-05 04:55:53 UTC. That's unexpected.

Likely it crashed and systemctl restarted it. Nothing shows in dmesg or journalctl, it was too long ago now I guess.

I'll turn on the server -log in case that helps, is there anything else I can do that would aid in diagnosis?

[hoffie]

Yes. Depending on the system config there may be no classical syslog anymore, but you should see this in journalctl -u jamulus* or the generic journalctl.

[bspratt]

Thanks, I'll monitor the situation,

(FWIW good old /var/log/syslog abides!)

[bspratt]

Hmm, it looks like it might have died while closing out a recording session. The .lof and .rpp files for the last session before the restart are empty. There were 66 tracks to deal with, and all but two of those are 0 bytes long. This is on a Vultr "High Frequency" insstance with 2GB memory.

[hoffie]

it looks like it might have died while closing out a recording session

I think there is at least one open issue about the recorder preventing Jamulus from working when the disk is full (#1163). There might be other cases (permissions). Can you verify if anything like that may be the cause for the crashes?

[bspratt]

Good catch - disk was quite full, this is likely the cause for the restart. (Clearly I need to revisit my cron job that's supposed to keep that recording directory hosed out)

Still on the lookout for Bill's issue.

[dakhubgit]

Christian Hoffmann ***@***.***> writes:

Hrm, what I can think of would be: - Jamulus exits cleanly (although unexpected for you). This should be indicated in the log file and sounds unlikely. - Jamulus crashes due to a bug (bad memory access). This should be visible as a SEGFAULT in `dmesg` and may be caught be coredump handlers (e.g. abrtd on RHEL) - Jamulus is killed by a supervisor (such as systemd) due to resource limit exhaustion. This should be logged by systemd.

Well, if the problem happens with loud crashes, that would point to either a problem with the Codec or a problem when converting the resulting floats to short. So it could be some kind of divide by zero or similar error. The system logs might have indicated the kind of signal that the server exited with.

…

-- David Kastrup

[bspratt]

Next it happens to Bill on a server that I run, I'm sure he'll alert me and I can see what I can find out.

[billthedrummer]

Yep will do. I have a practice at Studio Seattle tonight at 6. It's a loud one!

[bspratt]

OK, I'm moving that back to running 3.7.0 in hopes of running this down.

[billthedrummer]

Welp, I figured out the problem. I turned down the volume on my cymbals and still got kicked out. I noticed it happened on a single crash on my right crash, which was the closest one to my audio interface, so I thought I'd move it back a little When I did that, got kicked out again. Sooooo, the right crash must have swung into a cable or something. Sorry for all this, but I hope maybe Brian learned something. I'm off to order some sackcloth and ashes from Amazon...

[gene96817]

of all things, a hardware problem. :P

[bspratt]

That's good news!

[dcorson-ticino-com]

Great to hear you found it !

[gilgongo]

Although... #1509 :-)