Volumio Dev Team Weekly Diary

Volumio Community Forums News and Announcements
1

Dear Volumio users,

this thread wants to provide to the community a bit of a more “insight” into what the development team is doing, as most of the times what we do is behind the courtain work, which is barely visibile but that keeps the whole thing going.

I think that, as always, that transparency and openness is what foster the sense of community that we are nurturing here.

Hope also that this will be a pleasant read also for the techies out there, and to provide some insights of what means managing a project of the scale of Volumio.

Enjoy!

8 Likes
2

25 February 2022

This week we have been focusing on scaling up some of our cloud systems, as since the launch of Volumio 3 we’ve seen a great spike in usage of Volumio and therefore the vital cloud parts of it.

Specifically, we increased the capacity of the infrastructure that makes it possible to access your devices from myvolumio.org, adding automatic ssl renewal to them, experimenting with a new brilliant piece of software (caddy server) and setting up 6 new servers in Asia, Europe and North America.

We also further expanded auto-checks on many new endpoints, following a shameful incident that made some of our repositories unavailable due to expired certificates.

We upgraded the musicbrainz mirror (thanks to the help of musicbrainz dev team themselves) and added a robust caching mechanism. This improves response times of metadata retrieval and also refined the algorithm to provide more accurate results and much more hits.

BTW, if you are curious to know how many servers\cloud servers\services are in Volumio’s architecture, head to

https://status.volumio.com

It makes very apparent (to us in the first place) how complex is the architecture that we have to host to keep the project going. But this is just reflecting how technology has changed in the last years.
When we started Volumio we just needed one server (for downloads and OTA updates, and this server is still running), but now, a music player needs to interface itself with a myriad of third-party services to provide an adequate level of quality and integrations. This is also because in my opinion, a pure FOSS solution is not enough anymore in this day and age: you need to host a bunch of stuff and pay for it, to make it work.

We finally managed to track down a nasty bug that was affecting users with ISPs not using IPV6 or applying NAT, causing frequent stops in TIDAL, QOBUZ and everything that was streamed over HTTP2. This was due to a bug in curl (on buster) which made http2 requests fail under some conditions. We are currently doing QA on this fix, and if all goes well, next week we will release this update (for the joy of many users and our QA lead Davide).

Works on FusionDSP is progressing nicely, thanks to the effort of two brilliant members of the team and community, we are setting up a new open source component to make Fusion DSP finally work with Volumio’s AAMPP and we hope it can go public very soon.

We are also working on the new version of the Android\iOS app (which will be free). This new version will bring a super easy onboarding for the first time users, allowing them to auto update their devices and to connect to their wireless network in a very easy way.
This in our intention will make life easier for newcomers both of Volumio on raspberry PI but also on Volumio products (that’s another thing we’re working hard on… stay tuned…).

Unfortunately, what we are really behind schedule is improving the plugins store submission procedure and testing. We have still a huge backlog of plugins to test and no-one can really focus on it as the dev team is fully occupied. We are not holding it back however as we are actively looking for someone to take care of it as its firts and only activity (if you are interested on being this person, drop me a message).

We also made some minor fixes to CD playback plugin, Tidal Connect, Metavolumio.

That’s it for now, see you next week

11 Likes
3
4

Hi guys, please take a look at:

Your help is appreciated!

Many thanks!

Davide

5

March 4 2022

First of all, all our thoughts are for Ukraine and ALL people affected by this tragedy, which is beyond imagination. The team is discussing on how we can send tangible support and we hope we can take action next week.
We decided not to take a political stance, nor to make any statement on the situation but rather focus on doing something tangible to help people in need.

Going back to frivolous stuff, our work this week has been focused on continuing to harden our cloud server infrastructure for remote connection, gradually moving all servers to the new and improved caddy configuration.

On Monday we released a new update which fixed quite some bugs, but unfortunately, the Tinkerboard image suffered from a temporary Github outage, which prevented it from installing some core packages like curl and the remote updater. We then issued an emergency update on Wednesday and our support team is helping those affected.
We are improving the build script in a way that the build will fail if this occurs again, avoiding us to promote releases with missing core-packages.

Two new open-source alsa modules have been released and integrated in the main images, allowing the finalization of the FusionDSP works (which seems very close) and a useful facility for plugin developers.

There has been also bugfixing done for Metadata display on premium plans (from the search function) and some work on CD Playback.

Development on the new apps is currently on halt, awaiting results from testing from Volumio’s moderators and devs.

That’s it for this week. I hope that the next post on this diary will be more cheerful because peace has been reached (although it seems unilikely, but that’s what we hope).

6

March 11 2022

This week it has been almost all “behind the curtains” work for the team, boring but very important stuff…

First, there has been an incident when releasing version 3.233: a temporary github outage prevented the build system to download and install some critical components of Volumio image. Basically, the Tinkerboard platform ended up without some very important libraries (curl, flac) and the updater. Other platforms, like the PI, instead were built without the classic UI.
But you already knew about that.

The team work has been directed to debrief the incident and to take proactive measures against it happening again: we tweaked the build system in a way that it will fail the build if that should happen again (thanks Ash!) and add some more tests into our QA checklist for new releases (thanks Davide for the hard work there!).

We will gradually refactor the build system to report (and fail) every minor issue in the build process. Besides increasing trust in release repeatability, it will be a very important step in deploying an automated build system (on a CI) which we are considering since long time.

In the last times, the complexity of the project has increased substantially, so we are at the point where we need to set up robust procedures (quality, development, support) and debrief very often.

On an organizational level, this is increasing a little bit internal bureaucracy, but we reckon that it’s a very important task to carry out to ensure we guarantee high quality standards for our users. The mantra that we are having is: never make the same mistake twice.

Then, works on the new iOS and Android app is progressing, although with some showstoppers: we are trying to solve some compatibility issues with old android versions with the new Wireless set up flows: it appears that older Android versions are not able to receive properly the list of wireless networks to connect to.

We continue monitoring the cloud infrastructure, following the big upgrade that happened in the past few weeks. The scaling of servers and architecture seems to hold up pretty well, and the increased load is handled beautifully.

Last, but not least, since the biggest (and most daunting) bugs of Volumio 3 has been fixed, we are now focusing on minor bug fixing and tweaks, and we have a new version under QA testing, hopefully to be released next week. It will include fixes for Ui, network attached storage mounting, metadata discovery and more.

As you might notice, developing, maintaining and making Volumio grow, involves lot of work which is not obvious nor visible to the general public, but it’s equally important to make it work.

That’s it for this week, see you the next !

7 Likes
7

I have been much luckier with 3.233 installed 11 March (no problems so far) and I have to say that whatever design tweaks you have made, sound quality has improved considerably. Nothing else has changed in my system but it sounds like I made a significant upgrade. I was hugely disappointed when v3 was launched because expectations were high only to be met with too many issues. Well done!

1 Like
8

I find this type of authentic, no spin communication from @volumio really refreshing. Regular updates, open and honest about issues faced and mistakes made and learnt from. This is precisely what I would want to hear from a growing organisation that is facing the complexity of a maturing service, whilst still fully engaged with the diverse expectations and sometimes frustrations of the community that it serves.

3 Likes
9

March 18 2022

This has been a good week here at Volumio HQ. The main reason for that is that we somehow feel that we’ve managed to turn around the “bad climate” that we had in this community lately and re-establish the joyful vibe that we strive to cultivate in this place of the net.

We reckon that the release of Volumio 3 hasn’t been without issues: many people were frustrated by a couple of nasty bugs (which we solved) and the lack of plugins in V3. This meant that the community was filled with angry comments by users (and the famous vocal unhappy minority), which were expressing frustration about those issues. This vibe not only made this a quite less interesting place, but had a huge toll on the team’s morale and contributor’s “attachment” to the project.

Thanks to the suggestions of the private moderators’ group we decided to act and put a lot more effort in communicating proactively with the community (what you read is an outcome of this) and the users who were experiencing issues.

It looks like it’s paying off. We can sense that the spirit of this community is getting back. But we are not relaxing, this challenging time was an occasion for us to learn how to do better, and this is what we want to continue to do.

One process which is being reshaped a lot is our technical support and customer help department. First, we wanted to have a better “dispatching” of requests to the proper person in the organization. We achieved so by restructuring the contact page in a more “directive” way, to forward requests to the appropriate person (shop requests\returns, billing, technical support, partnerships, distribution etc).

Then we wanted to provide faster answers to the people requesting technical support.

By analyzing supports requests we found out that about half of them could be solved pretty quickly without the need of our support people, by just providing small written hints.
So we implemented a chat-bot with automatically matches users’ requests with our knowledge base (which we are filling on an ongoing basis) and provide immediate answers to known problems or situations.

This has proven to work well for some kinds of requests, but really it did not for others. Analyzing the flow and some feedback by our users this week, we found out that this system was not providing enough feedback on the required actions for some users (we required some further action from user before initiating the support session) which felt frustrated of not receiving support and contacted us on other channels (Facebook, forum, etc).

So, we are working to change this part, by changing the user contact flow in those kinds of situations. We expect to have this change live in about 10 days.

So, for those users which felt neglected in the past month, we acknowledge the problem and we are working to fix it, so the next time you need us you’ll find a better experience.

On the development side, work this week has been again on our cloud ecosystem. If you wonder why that is, that we spent all this time on cloud stuff and not on the “Volumio” side of things, here’s why.

Back in 2018 we took a very important architectural decision: move all the integration intelligence to the cloud, rather than on the device.

Example: our albumart retrieval server used to run on the device. It basically identified the artist, album and track and made some queries online to fetch and download the most appropriate albumart for a specific track. This worked well. Or at least it did until one of those online services shut down or change their API (this happens more frequently that you might think).

So, you have 2 ways to solve the issue:

  • Update all the software you have on the field (in our case more than 430k installs). This means that many users will experience the problem until they update their software. It also means that spotting this problem will come only after you have some users reports.
  • Move the logic that aggregates online content to the cloud and have the albumart facility in Volumio just rely on that.

We took the second step. This brings some additional advantages:

  • We can understand quickly if something goes wrong, since we can set up automatic monitoring of all the service we use.
  • When we spot a problem and fix it, the fix is istantaneous for all our users, without the need for update.
  • Doing some optimizations like caching and load balancing we can offer faster response times compared to an algorithm running on the device itself, and make in turn the whole experience better.

This comes also with some disadvantages:

  • It costs money to run those services. If we use the “distributed” computing power of the devices, it does not cost anything. But if we need to set up, mantain and host such systems on a scale required for Volumio, this has a cost.
    Basically, every time you look for an albumart, Volumio has to pay for it. Considering the service we provide for some of those endpoint is free, we have to be very clever to optimize our technology to not be overly expensive (in those cases, we don’t use AWS but a mix of Kubernetes + dedicated VPS with high microservices density).
  • In case it fails, it fails for everyone at the same time. So you have to put in place automatic recovery strategies and monitoring and ensure adequate redundancies.

Some examples of the services that use this architecture are:

  • Oauth login for Spotify, QOBUZ, TIDAL (and possibly more like LastFm)
  • Albumart retrieval
  • Browsing of QOBUZ and TIDAL
  • Plugins store
  • Webradio selection
  • Metadata and credits retrieval

And yes, what needed some love this week was the albumart provider. A change in an upstream API deteriorated the performace of retrieval of artist arts. We were able to fix it and deploy a fix without requiring anyone to update. And the best part of it, is that most of you didn’t notice.

We are continuing with this architectural pattern, especially when we enrich functionalities, that might explain why you see less commits on the Volumio backend regarding such aspects.
Some of the things we already worked on (we are waiting for frontend parts to support it) are live concerts notifications and lyrics.

That’s it for this week! Hope you got a better glimpse of how also the community can impact Volumio, both in positives and negatives ways. Remember that and keep on helping making this place a good portion of the net.

14 Likes
10

Right! You are no doubt aware that more than half the plugins are not compatible.

11

Right! You are no doubt aware that the plugins are created by the community, not by Volumio?

3 Likes
12

Small remark in that regard, I think many plugins actually are (forward) compatible, but require recompilation of modules, this takes a little bit of time. Which indeed can be frustrating, especially if you haven’t got the faintest on how to do it yourself. But as said above, I think we’re progressing nicely.

Also, developing (and patching) plugins can be time consuming, so I guess it will take some time before all plugins have been ported (or at least the ones in high demand). :wink:

13

Hello!

Not if I would rush anything nor claim at all, but when can we expect the update you’ve mentioned in this post (possibly and anticipatedly also with that fix for the Thread’s issue)? Its been a while now…

Thank you!

14

We managed to fix the last issue that prevented us to release the update. Since we never release on Friday, release will happen next week.

image

15

March 25 2022

First of all, we managed to package together the update which we will release next week. It includes various fixes for multiroom, utf-8 characters and the longly awaited AAMMP update (which will allow us to release the amazing Fusion DSP plugin).

What held us up from releasing was a mysterious bug that affected multiroom, which we took a while to identify (turned out to be a very silly mistake, but very hidden and difficult to spot).

Then, finally, we managed to get the new Android and iOS app to a level where we are happy with: the new improved and simplified onboarding flow works well in all circumstances.

It took a lot of debugging, testing, swearing… You can’t imagine how tedious sometimes can be to design and implement a very simple process…
Next time you try a piece of technology that just works, consider that a developer could have lost all his hairs to make it work properly :wink:
Our plan is to release a beta test of the new app next week.

This week focus has also been a careful analysis of our current state (as a project and as a company) and strategic planning. As they say, “tactics without strategy is the noise before defeat”.

We held various team meetings to pivot our strategy in 2 main areas.

The first is the OEM long-term strategy and approach. As you might know, Volumio OEM is a service we provide to quite a lot of Hi-Fi companies around the world. We basically empower them to release Hi-Fi products using Volumio Technology and experience. We are proud that more than 30 Hi-Fi companies worldwide choose Volumio to power their audiophile products.

This has worked well both for us and them for the past years and Volumio is what it is today also thanks to the financial and design contribution of our partners.

We however think that we can do better and be more ambitious with the OEM Partner program.
We saw a huge increase in requests to become Volumio OEM Partners since the release of Volumio 3 and a slight change in what new partners require from our technology.

Compared to 3 or 4 years ago, Hi-Fi companies tend now to have in-house IT departments and are asking us less direct involvement and more freedom when implementing our technology and a more SDK-like approach.

So, we are redesigning our OEM program very similarly to what Android does. So yes, our plan is that Volumio will become the Android of the music technology world.
Of course, following our “co-design” approach, we are involving some of our best partners in the decision and design processes.

The other planning involved Hardware. We have ambitious plans and some really, really nice things that we are working on…
We can’t disclose anything at the moment, but, if I were you, I would consider visiting this year’s
HIGH END Munich trade show…

That’s it for this week!

4 Likes
16

Munich, Finally something is going next/replace Primo?

1 Like
17

Primo is here to stay :wink:

18

You did announce a linear power supply for the primo a couple of years ago and I’m still waiting!

19

At Volumio we want to release only products that we are proud of. This PSU, while absolutely fantastic, was not hitting our price\performance ratio desiderata. We are not done with that yet, but no PSU from us in the near future.

20

come and see

1 Like