Yes, but it’s partly dependent on your definition of a performance difference. Most obviously, the Pi Zero has a far longer bootup time for Volumio: put simply, a long bootup for Zero versus a short bootup for Pi 3. If you’re in the habit of shutting down and booting up for each listening session, that might bother you; then again it might not.
If you mean the music itself, there is no difference between the audio capabilities of the Pi Zero and the the Pi 3: if you feed the signal out over HDMI, you can have digital sound up to 24/192 fed to a DAC fed in turn to an amplifier in a high end audio system. (Alternatively you can add a digi+ board directly to a Pi Zero or Pi 3 and feed out a digital signal over coaxial cable to a DAC; my preferred method.) The (potentially) weakest link in that chain is not the Pi, it’s the DAC. Place a great DAC in the chain and you’ll get great sound. The Pi itself can never transcend the downstream DAC, so the DAC is far more important than the Pi or other upstream device. A Pi into a truly high end DAC should perform extremely well, and the Pi model should be immaterial.
In terms of things like freeze ups, dropouts, or wifi issues, I’d say there’s no noticeable difference. However, if you write data (music files) to the internal SD card while playing music, it will probably affect the Zero more severely, causing a lot of dropouts.
The final thing I would consider - you can decide yourself whether you call it performance related or not - is “plug-and-playability”. The Pi 3 has all the USB ports you need, plus a full size HDMI out and an Ethernet port. It’s fully plug-and-play. But if you want Ethernet with the Zero, get a USB adapter. If you to want to use any USB device or adapter with the Zero, first get a mini-to- full size USB adapter for the Zero’s USB-B port, and if you need to add more than one USB device to the Pi, get a USB hub as well. For a non-wifi Zero, you need two hub ports just for wifi and a hard drive; add Ethernet capability and you need to use three. There is a mini-HDMI port, so you need either the cable with the right connectors at each end (one mini connector, one full size) or yet another adapter. So the Zero is the opposite of plug-and-play, but if you don’t mind doing what it takes to get it running, its audio performance is as good as the Pi 3.
The remaining question really, is what is all that miniaturization actually for, especially as you can’t get the thing to sit flat on a desk with the cable connectors being heavier than the actual unit. The Pi 3 is already small enough. My reason for doing a Pi Zero in addition to the 3 was to see how it worked, and if you have the same idea of experimentation, then it’s fine. But you don’t really need a player this small. That said, I can plug it into a serious audio system and it is the equal of any other component I put in there - the CD player, turntable, etc. But it has to go through a DAC to achieve that - all Pi models are sonically lousy if you rely on line out.