Volumio restarts AirPlay due to 'fatal error'

Volumio Version: 2.907
Hardware: RPi
DAC: Boss

Today, I wanted to use my audio setup to play back music from my Mac using AirPlay. It worked great for a while, but then just stopped after some time (about an hour or so). It restarts just fine, but then again stops by itself after a while… I debugged Wifi issues, but this even happens when the RPi and the Mac both have their Wifi turned off and are connected into the same Ethernet switch.

It turns out that Airplay stops because shairport-sync gets restarted: typing sudo journalctl -n200 | grep shairport I see an entry like

Sep 12 13:48:13 volumio sudo[3985]: volumio : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl restart shairport-sync

(Actually when this appears, the same entry always shows up twice in a row with the same timestamp.) Any idea what causes this and what I can do on my end to avoid having shairport-sync being restarted every now and then?

Edit: See below for more details of what’s happening.

OK, mystery solved (I think). The problem was that I had the “Wifireconnect” plugin installed and enabled. Every now and then, the router would drop a couple of packets which would then cause Wifireconnect to reset the network interface. Apparently this then also resets shairport-sync (which is reasonable I guess). I just disabled Wifireconnect which appears to have solved the problem.

It appears that Wifireconnect wasn’t the culprit after all: shairport-sync just got restarted again even though Wifireconnect is now disabled (and I restarted volumio after disabling it, for good measure). It’s just that this time it took over two hours for the problem to manifest itself. I guess that’s some progress…

If you edit the volumio script (via ssh or keyboard and monitor) …

nano /usr/local/bin/volumio

And comment out the “curl” line under “airplayinactive” (hit control W, search for airplayinactive and add a # at the beginning of the curl line) does this help? Use control X to save.

If not, remove the # and all will go back to normal.

You might want to save a backup of this file before tinkering with it.

cp /usr/local/bin/volumio ~/volumio

To restore the backup …

cp ~/volumio /usr/local/bin/volumio

Thanks a lot Evan for the suggestion. I’m afraid that it didn’t make any difference. I tried it again this morning and shairport-sync was restarted after about half an hour.

OK, so this time I kept journalctl -f running to try to see what triggers the restart of shairport-sync. It turns out that the reason seems to be a “fatal error” which causes the whole volumio subsystem to relaunch itself, not just shairport-sync (although it must be related to AirPlay since I’ve never experienced that kind of interruption before). Specifically, I get the following in the log:

Sep 13 20:59:23 volumio volumio[838]: |||||||||||||||||||||||| WARNING: FATAL ERROR |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Sep 13 20:59:23 volumio volumio[838]: { Error: connect ETIMEDOUT 151.101.18.79:443
Sep 13 20:59:23 volumio volumio[838]: at Object._errnoException (util.js:1022:11)
Sep 13 20:59:23 volumio volumio[838]: at _exceptionWithHostPort (util.js:1044:20)
Sep 13 20:59:23 volumio volumio[838]: at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
Sep 13 20:59:23 volumio volumio[838]: code: ‘ETIMEDOUT’,
Sep 13 20:59:23 volumio volumio[838]: errno: ‘ETIMEDOUT’,
Sep 13 20:59:23 volumio volumio[838]: syscall: ‘connect’,
Sep 13 20:59:23 volumio volumio[838]: address: ‘151.101.18.79’,
Sep 13 20:59:24 volumio volumio[838]: port: 443 }
Sep 13 20:59:24 volumio volumio[838]: |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Sep 13 20:59:24 volumio sudo[25371]: volumio : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/journalctl --since=2021-09-13 20:58
Sep 13 20:59:24 volumio sudo[25371]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 13 20:59:24 volumio sudo[25371]: pam_unix(sudo:session): session closed for user root
Sep 13 20:59:24 volumio systemd[1]: volumio.service: main process exited, code=exited, status=1/FAILURE
Sep 13 20:59:24 volumio volumio-remote-updater[716]: [2021-09-13 20:59:24] [error] handle_read_frame error: websocketpp.transport:7 (End of File)
Sep 13 20:59:24 volumio volumio-remote-updater[716]: [2021-09-13 20:59:24] [disconnect] Disconnect close local:[1006,End of File] remote:[1006]
Sep 13 20:59:24 volumio systemd[1]: Unit volumio.service entered failed state.
Sep 13 20:59:24 volumio systemd[1]: Starting dynamicswap service…
Sep 13 20:59:24 volumio systemd[1]: Started dynamicswap service.
Sep 13 20:59:25 volumio systemd[1]: volumio.service holdoff time over, scheduling restart.
Sep 13 20:59:25 volumio systemd[1]: Starting dynamicswap service…
Sep 13 20:59:25 volumio systemd[1]: Started dynamicswap service.
Sep 13 20:59:25 volumio systemd[1]: Stopping Volumio Backend Module…
Sep 13 20:59:25 volumio systemd[1]: Starting Volumio Backend Module…
Sep 13 20:59:25 volumio systemd[1]: Started Volumio Backend Module.
Sep 13 20:59:27 volumio volumio[25393]: info: -------------------------------------------
Sep 13 20:59:27 volumio volumio[25393]: info: ----- Volumio2 ----
Sep 13 20:59:27 volumio volumio[25393]: info: -------------------------------------------
Sep 13 20:59:27 volumio volumio[25393]: info: ----- System startup ----
Sep 13 20:59:27 volumio volumio[25393]: info: -------------------------------------------

This is followed by another couple of hundred lines which in particular include the relaunch of shairport-sync as well as various other services. This is still mysterious to me: why does volumio try to connect to 151.101.18.79 (an IP address that appears to belong to the bay area cloud services provider Fastly) and why does it basically commit suicide when that fails? (Possibly due to an invalid SSL certificate, at least that’s what I get when trying to connect to that IP address with my browser.)

Most importantly, how can I prevent this from happening?

I was able to mitigate the problem by setting EXIT_ON_EXCEPTION to false in volumio’s environment file. This tells volumio not to relaunch itself after an uncaught exception. This is probably not recommended in general since it might find itself in a broken state if some other bug occurs. On the other hand, it does have the effect of simply ignoring the failed connections to that IP address, so that AirPlay now works fine without interruption.

The ideal solution of course would be to find a way to avoid having this exception being raised in the first place. (Or to have the code fixed so that it gets caught and handled gracefully.)

Could you share the log entries right before the FATAL ERROR? Also a standard debugging log would also be helpful!

The log just before the fatal error is usually unremarkable. Last time it happened, it looked like this:

Sep 14 21:01:36 volumio volumio[1762]: info: CoreCommandRouter::executeOnPlugin: volumiodiscovery , saveDeviceInfo
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::servicePushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreStateMachine::pushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::volumioPushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::executeOnPlugin: volumiodiscovery , saveDeviceInfo
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::servicePushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreStateMachine::pushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::volumioPushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::executeOnPlugin: volumiodiscovery , saveDeviceInfo
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::servicePushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreStateMachine::pushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::volumioPushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::executeOnPlugin: volumiodiscovery , saveDeviceInfo
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::servicePushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreStateMachine::pushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::volumioPushState
Sep 14 21:05:08 volumio volumio[1762]: info: CoreCommandRouter::executeOnPlugin: volumiodiscovery , saveDeviceInfo
Sep 14 21:05:41 volumio volumio[1762]: |||||||||||||||||||||||| WARNING: FATAL ERROR |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Sep 14 21:05:41 volumio volumio[1762]: { Error: connect ETIMEDOUT 151.101.18.79:443
Sep 14 21:05:41 volumio volumio[1762]: at Object._errnoException (util.js:1022:11)
Sep 14 21:05:41 volumio volumio[1762]: at _exceptionWithHostPort (util.js:1044:20)
Sep 14 21:05:41 volumio volumio[1762]: at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)
Sep 14 21:05:41 volumio volumio[1762]: code: ‘ETIMEDOUT’,
Sep 14 21:05:41 volumio volumio[1762]: errno: ‘ETIMEDOUT’,
Sep 14 21:05:41 volumio volumio[1762]: syscall: ‘connect’,
Sep 14 21:05:41 volumio volumio[1762]: address: ‘151.101.18.79’,
Sep 14 21:05:41 volumio volumio[1762]: port: 443 }
Sep 14 21:05:41 volumio volumio[1762]: |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

The debugging log at http://logs.volumio.org/volumio/9jiUCnG.html was generated shortly after one of these events happened.

You are right, it is rather unremarkable!
Given the IP resolves to some fastly domain, I will reckon a guess that it is probably something to do with some myVolumio service - could you try logging out of your myVolumio account and see if that stops the crashes?

Hi, I don’t actually have a myVolumio account (as far as I remember: it says ‘not connected’ under ‘MyVolumio’ at the top left), so I guess that I’m already logged out. I could try creating an account and logging in…

OK, I have reenabled EXIT_ON_EXCEPTION and logged into a MyVolumio account. The problem still persists with the same symptoms. Debugging log generated just after volumio relaunched itself can be found at http://logs.volumio.org/volumio/PFO9ofW.html.

This afternoon however I had AirPlay running for over two hours without any interruption and no problem. I then installed ufw and instructed it to block the 151.101.18.xx IP range. Within about 15 minutes or so the same error occurred again. So the takeaway is that

  1. It may well be that there was a glitch at Fastly which got resolved, so that the problem is more or less taken care of.
  2. There is a bug in volumio that causes an uncaught exception to be raised when that IP range is inaccessible (or doesn’t respond fast enough) for some reason. If you want me to do some specific test on my end to help locate the bug, please let me know.
  3. The problem only appears to manifest itself when AirPlay is active.

By the way, the MyVolumio password reset emails fail quite a large number of spamassassin tests so get a spam score of 5.8 on my system: DCC_CHECK, DKIM_INVALID,DKIM_SIGNED, HTML_MESSAGE, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, URI_FIREBASEAPP, URI_TRY_3LD.

Ah, I didn’t catch that part at first. Then the myVolumio stuff was a complete wrong road, sorry!

What are you playing on airplay? Is album art pulled from fastly by any chance?

Ah, good question. I was playing the “my station” stream on Apple Music. I’ll try to play from my own library to see if that makes any difference… OK, just tried it, it still triggers the error, this time almost instantly. (I did have ufw enabled to make it more reproducible.)