Out of Disk Space (overlayfs seems incorrect)

Help and Support
1

Hi,

In short - I’m out of disk space, on the root fs.

Somethings awry, but I’m not sure what.

Filesystem      Size  Used Avail Use% Mounted on
overlay         262M  256M     0 100% /

Checking via fdisk, we can see that the last partition has been resized correctly:

Disk /dev/mmcblk0: 29.8 GiB, 32026656768 bytes, 62552064 sectors
...
/dev/mmcblk0p3      4882813 62552063 57669251  27.5G 83 Linux

Looking at overlayfs itself…

overlay on / type overlay (rw,relatime,lowerdir=/mnt/static,upperdir=/mnt/ext/dyn,workdir=/mnt/ext/work)

And checking those directories, they’re non-existant - is that correct ?

root@volumio:/mnt# ls -l /mnt/{static,ext/dyn,ext/work}
ls: cannot access /mnt/static: No such file or directory
ls: cannot access /mnt/ext/dyn: No such file or directory
ls: cannot access /mnt/ext/work: No such file or directory

The contents of /mnt for clarity are:

root@volumio:/mnt# ls -l 
total 8
lrwxrwxrwx 1 root root   14 Oct  2  2020 INTERNAL -> /data/INTERNAL
drwxrwxrwx 1 root root 4096 May 11 09:23 NAS
drwxrwxrwx 2 root root    3 Oct  2  2020 UPNP
lrwxrwxrwx 1 root root    6 Oct  2  2020 USB -> /media

Any help in rejigging the filesystems to use the intended space appreciated…

2

What about telling is a little more, are you referring to a PI, a Tinkerboard, an x86 installation or perhaps something else?
And what version are you using? Is is a fresh install? Did you do anything specific, like a factory reset?
In short, please tell us what you did and could have lead to this issue.

3

Sure, it’s on a raspberry Pi. Volumio 2.834. Fresh install, no factory resets or anything.

" In short, please tell us what you did and could have lead to this issue."

I literally just installed it yesterday - not messed with anything under the covers. Just tried to do an apt-get of some packages…

4

Please be a bit more generous with your info :roll_eyes:
Which PI?
And the info you shared in the first post, was that before or after doing apt-get?
Trust you did not do an apt-get upgrade (as update is all you are allowed to).
Which brand SD card are you using? Class?

To help with this, it would be very helpful if you could do another fresh install and share a full log.
Preferably immediately after the first boot, before doing anything else.

5

I’m not sure how this helps you, given the issue is with the overlayfs, but its:

  • a pi4
  • after doing apt-get, but regardless it would appear if anything wrote to the rootfs
  • no, just an apt update and tried to install vim
  • the SD card is fine, there’s no reason to walk down that path
  • I haven’t reinstalled, but I’m not sure what that would achieve http://logs.volumio.org/volumio/uLmezY6.html
6

No, the issue is NOT overlayfs, there is something going wrong with resizing and the partition table. The overlayfs configuration in our initramfs uses the 3rd partition as-is after resizing and I would like to see any error message, warning, whatever. That is what the log is for. But if you can’t be bothered, up to you to decide whether you want help…

7

Fresh install. Log uploaded to here: http://logs.volumio.org/volumio/OHHPDla.html

Tried to install some large packages, to attempt to overfill /. Success:

dpkg: error processing archive /var/cache/apt/archives/golang-go.tools_0.0~hg20140703-4_armhf.deb (--unpack):
 cannot copy extracted data for './usr/bin/godex' to '/usr/bin/godex.dpkg-new': failed to write (No space left on device)
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

Disk has been expanded correctly upon bootup:

Disk /dev/mmcblk0: 29.8 GiB, 32026656768 bytes, 62552064 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x67b373ff
Device         Boot   Start      End  Sectors   Size Id Type
/dev/mmcblk0p1 *          1   500000   500000 244.1M  c W95 FAT32 (LBA)
/dev/mmcblk0p2       500001  4882812  4382812   2.1G 83 Linux
/dev/mmcblk0p3      4882813 62552063 57669251  27.5G 83 Linux

Output of ‘df’

volumio@volumio:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  2.0G  852M  1.1G  45% /imgpart
/dev/loop0      349M  349M     0 100% /static
overlay         262M  234M  7.1M  98% /
devtmpfs        948M     0  948M   0% /dev
tmpfs           992M     0  992M   0% /dev/shm
tmpfs           992M  8.7M  984M   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           992M     0  992M   0% /sys/fs/cgroup
tmpfs           992M   28K  992M   1% /tmp
tmpfs            20M   48K   20M   1% /var/log
tmpfs           992M     0  992M   0% /var/spool/cups
tmpfs           992M     0  992M   0% /var/spool/cups/tmp
/dev/mmcblk0p1  241M   74M  167M  31% /boot
tmpfs           199M     0  199M   0% /run/user/1000

Mounts:

proc on /proc type proc (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=970028k,nr_inodes=181911,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,noatime,mode=755)
tmpfs on /var/log type tmpfs (rw,nodev,relatime,size=20480k,mode=777,uid=1000,gid=4)
tmpfs on /var/spool/cups type tmpfs (rw,noatime,mode=755)
tmpfs on /var/spool/cups/tmp type tmpfs (rw,noatime,mode=755)
/dev/mmcblk0p1 on /boot type vfat (rw,nosuid,nodev,noexec,relatime,fmask=0111,dmask=0000,allow_utime=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=203116k,mode=700,uid=1000,gid=1000)

Overlayfs

volumio@volumio:/$ mount | grep overlay
overlay on / type overlay (rw,relatime,lowerdir=/mnt/static,upperdir=/mnt/ext/dyn,workdir=/mnt/ext/work)
volumio@volumio:/$ ls -l /mnt/{static,ext/dyn,ext/work}
ls: cannot access /mnt/static: No such file or directory
ls: cannot access /mnt/ext/dyn: No such file or directory
ls: cannot access /mnt/ext/work: No such file or directory
8

I’ve manually fixed this, by writing the image to an SD card and then resizing it manually using gparted, before first boot.

volumio@volumio:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay          27G   15M   26G   1% /
9

This is a weird situation, no errors.
Overlayfs is created with the squashfs file from mmcblk0p2 and two new folders + workdir on mmcblk0p3.
There is something really wrong with the partition on mmcblk0p3, it looks like it is re-sized to max, but overlay is still reported with 262M as if nothing happened.
I have (RPi 4 with 16Gb SD) showing it as 13G.
We’re on version 2.882, which might give slightly different storage figures.

Checking the dynamic and work directory your way (via grep overlay) may not work correctly.
Could you try mounting /dev/mmcblk0p3 to a temporary folder and do ls -l from there?
You should have dyn, union and work.

10

Confirmed:

volumio@volumio:/tmp/xx$ ls -la
total 36
drwxr-xr-x 6 root root  4096 Jan  1  1970 .
drwxrwxrwt 9 root root   360 May 12 15:13 ..
drwxrwxrwx 6 root root  4096 Jan  1  1970 dyn
drwx------ 2 root root 16384 Oct  2  2020 lost+found
drwxrwxrwx 2 root root  4096 Jan  1  1970 union
drwxrwxrwx 3 root root  4096 Jan  1  1970 work

I suspect that this is due to the fact the image didn’t originate from Volumio - it’s the AOIDE DAC image which they patch to support their dac chipset. I believe they have to do this, as there’s no native support in the kernel due to licensing issues. They do provide the binary blobs though, in case there was appetite to support this dac card by Volumio, GitHub - howardqiao/aoide-dac-drivers

11

I had the same issue last week, I checked what you are saying and actually everything was fine, the partition was the right size, but df was reporting it wrong.
Eventually I did an fsck of the partition and a resize2fs and that fixed it.
Volumio latest version running on Pi4

12

Please, could you please please tell us that the next time. Do we have to pull every bit of info out of you?
We do not support externally modified images, who knows what else they changed?
My work is done.

1 Like
14

Had I said that, the only ‘help’ I would have received from Volumio Support would have been to “use our official image” and “buy a different DAC”.

Neither would have worked for me.

I wasn’t asking for help with volumio per se, I was asking for help in understanding the underlying filesystem setup. I’m aware I’m on my own with this image, but this isn’t tech support - this is a user forum.

That said, should you be in a position to support this DAC, this situation wouldn’t occur. Can you add support for it, and remove the chance of this happening to the next person?

15

In case others find this thread, would you outline what the steps you took were to achieve it?

16

This is unlikely to happen. We do not maintain the RPi kernel, it is Aoide’s job to add the drivers to the kernel, it cannot be ours.
The fact, that they were not added, makes me believe they either have no (or not enough) interest or the driver sources do not comply with kernel coding standards.
Ask Aoide whether they want to develop a plugin to add support, others have done that as well, stops having to patch our images and enables the use of the latest Volumio service.

17

I read this part of the init script

# if data partition needs to be resized
#mount -t auto /dev/${BOOTDEV}p1 /boot
if [ -e "/boot/resize-volumio-datapart" ]; then
print_msg "Re-sizing Volumio data partition"
  END="$(parted -s /dev/${BOOTDEV} unit MB print free | grep Free | tail -1 | awk '{print $2}' | grep -o '[0-9]\+')"
  parted -s /dev/${BOOTDEV} resizepart 3 ${END}
  e2fsck -fy /dev/${BOOTDEV}p3
  resize2fs /dev/${BOOTDEV}p3
  print_msg "Volumio data partition succesfully resized"
  parted -s /dev/${BOOTDEV} unit MB print
  rm /boot/resize-volumio-datapart
fi

Based on that I ran e2fsck -fy /dev/mmcblock0p3 but that failed due to the fact that the partition was mounted, so I skipped to resize2fs /de/mmcblock0p3, then a reboot.
Sorry, i have fading memories, but those should be the steps. Later on today, I will try again with a fresh SD and if it should fail, I will note down the steps I will take.

For me it seems that the above script is failing somehow in the resize2fs part.
The partition was already properly resized, so the parted -s /dev/${BOOTDEV} resizepart 3 ${END} worked and I could see the message Volumio data partition succesfully resized in the logs, but the filesystem was not resized

18

Here we go!
This is the partition table at first boot

root@volumio:/home/volumio# parted /dev/mmcblk0 print
Model: SD USDU1 (sd/mmc)
Disk /dev/mmcblk0: 15.8GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      512B    64.0MB  64.0MB  primary  fat32        boot, lba
 2      64.0MB  2500MB  2436MB  primary  ext4
 3      2500MB  15.8GB  13.3GB  primary  ext4

So the partition has been resized, but

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  2.2G  588M  1.5G  29% /imgpart
/dev/loop0      294M  294M     0 100% /static
overlay         262M   14M  228M   6% /

so the resize2fs has not been run.
I can’t directly resize mmcblock0p3, it is failing

root@volumio:/# resize2fs /dev/mmcblk0p3
resize2fs 1.42.12 (29-Aug-2014)
resize2fs: Device or resource busy while trying to open /dev/mmcblk0p3
Couldn't find valid filesystem superblock.

but I can mount it and then resize it (weird, I know)

root@volumio:/# mount /dev/mmcblk0p3 /tmp
root@volumio:/# ls /tmp/
dyn  lost+found  union  work
root@volumio:/# resize2fs /dev/mmcblk0p3
resize2fs 1.42.12 (29-Aug-2014)
Filesystem at /dev/mmcblk0p3 is mounted on /tmp; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/mmcblk0p3 is now 3256016 (4k) blocks long.
root@volumio:/# umount /tmp
root@volumio:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  2.2G  588M  1.5G  29% /imgpart
/dev/loop0      294M  294M     0 100% /static
overlay          13G  143M   12G   2% /

Sorted :wink:
I don’t have time to investigate why the partition is resized, but the FS is not, sorry

2 Likes
19

Weird, same results here.

root@volumio:/home/volumio# resize2fs /dev/mmcblk0p3
resize2fs 1.42.12 (29-Aug-2014)

resize2fs: Device or resource busy while trying to open /dev/mmcblk0p3
Couldn't find valid filesystem superblock.

root@volumio:/home/volumio# mount /dev/mmcblk0p3 /tmp/xx/
root@volumio:/home/volumio# resize2fs /dev/mmcblk0p3
resize2fs 1.42.12 (29-Aug-2014)

The filesystem is already 7208656 (4k) blocks long. Nothing to do!

But, at least there’s a way to fix this before installation, and after installation also now.

20

I just resolved the same problem on a system started from an official image.
The steps you describe are the same steps I did (only difference is that I mounted /dev/mmcblk0p3 on /mnt/USB)

21

Hey, welcome to Volumio community! :slight_smile:
Yes, you can mount the partition wherever you wish, actually /mnt/USB makes more sense, if you are not using an USB key, I used /tmp by habit: I use tmp for this quick kind of tasks on servers too, although /mnt is safer for longer running tasks as you are hiding /tmp content for a while :wink: