fsck: recovering journal every reboot [Solved]

[rych]

Could someone with changes=/porteux on an ext4 help me test this? Reboot with the 'fsck' cheatcode and pay attention at the very beginning of the booting black screen as it's checking the filesystems. In my case it's "recovering journal" every reboot with frequent "clearing orphaned inode"

All I do is boot into my new PorteuX and then immediately go for a reboot -- but of course some changes are still written under /porteux -- and I see those messages.

It might be that PorteuX is so fast to shut down that it doesn't unmount, or doesn't wait for the completion of unmount.

My disk is NVMe SSD in a USB 3.1 enclosure and instead of using usb-storage it uses uas as a driver:

Code: Select all

root@porteus:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 10000M
    |__ Port 1: Dev 2, If 0, Class=, Driver=usb-storage, 5000M
    |__ Port 2: Dev 8, If 0, Class=, Driver=uas, 10000M

However, Porteus 5.0 on a different USB with both USB plugged in shuts them all down correctly (and many seconds slower): I can write to the USB disk something and the next reboot all file systems on both USBs are clean. Also, Porteus 5.0 installed on the same USB as PorteuX, in a different partition, after a test write to the PorteuX partition reports all clean filesystems on next reboot.

My hypothesis: PorteuX is too fast at shutting down, not having the good old 3-5 sec delays here and there

I've tried changes=EXIT:/porteux hoping it would do a final write and unmount more thoroughly/slowly -- but it's the same

[beny]

hi in porteux 4 current we have the /etc/rc.6 script modified a lot so you have a fast halt of the system, try to insert a stock script to see if the shutdown is different an allow to the nvme to end the write task.

[Ed_P]

the /etc/rc.6 script

/etc/rc.d/rc.6

[porteux]

I can confirm this issue. This is way more complicated that it seems, or maybe I'm missing something:

1- I fixed the union unmount step that was missing in PorteuX -- to be released in 0.5 version. This fixes the 'clearing orphaned inode' messages, but unfortunately it doesn't fix the 'recovering journal' ones.
2- after sysvinit 2.96 (located in 001-core) the binary /sbin/init has been updated in such way that it's causing /union to not unmount completely anymore (/union/sys/fs/cgroup and /union/sys/fs/cgroup/elogind get stuck). I'm not sure what to do about this.
3- even using Porteus 5 on a generic USB stick on my machine I need to but a sleep 3 before running fsck (in initrd) otherwise the system won't find the storage unit on time therefore fsck won't do anything.

[rych]

/union to not unmount completely anymore (/union/sys/fs/cgroup and /union/sys/fs/cgroup/elogind get stuck)

Sorry it turns out to be so complex and esoteric. Reminds me of how difficult it used to be to cleanly eject a USB under Windows: always something is holding it

Is there a command in Linux to flush writing operations through all levels: from filesystem down to the USB controller hoping it gets released then?

Porteus 5 on a generic USB stick on my machine I need to but a sleep 3 before running fsck

Yes, I remember that magic delay 3, it must be 3 USB has always been flaky and a little mysterious. As a user I don't mind waiting an extra 3 seconds, especially since I use the changes=EXIT:/porteux and expect to wait a bit longer at the end. Otherwise, great job making PorteuX shut down instantaneously!

[porteux]

After some investigation I realized that it's not exactly the version of /sbin/init (which is included in sysvinit) that makes the difference, but rather how it's built. It turns out that the version of /sbin/init that Porteus uses has never been the one in the official Slackware repository:

/sbin/init version 2.88:
porteus 4 -> 69.8 kb
slackware -> 39.5 kb

/sbin/init version 2.96:
porteus 5 -> 89.7 kb
slackware -> 47.6 kb

/sbin/init version 3.01:
porteus -> N/A
slackware: 47.6 kb

As you can see, Porteus has been shipping a much larger /sbin/init that for unknown reasons (at least to me) allows the system to perfectly unmount things during shutdown/reboot so fsck doesn't complain.

Consider the following commands in /mnt/live/cleanup script being executed after kill -15 `echo $PID` >/dev/null 2>&1

Code: Select all

umount -nl /union
umount /union
umount /memory/changes
umount /memory/images/*

If we run the code above using /sbin/init (no matter which version) from Slackware it will always fail from the second command (umount /union) onwards because the devices are busy. If we use /sbin/init from either porteus 4 or 5 it works flawlessly.

[Ed_P]

So, who is maintaining the Porteus /sbin/init code?

[porteux]

So, who is maintaining the Porteus /sbin/init code?

No idea. But the fact that Porteus is stuck in sysvinit 2.96 (released almost 4 years ago) is a sign that something went wrong.

[Ed_P]

Well, that is about the time brokeman left. May 2019 was the last I heard from him. Since /sbin/init is still working an update is not needed at this point but finding brokenman's source code would be nice.

[porteux]

Well, that is about the time brokeman left. May 2019 was the last I heard from him. Since /sbin/init is still working an update is not needed at this point but finding brokenman's source code would be nice.

In his build script packages (http://brokenman.porteus.org/current/build-scripts/) /sbin/init is always there already built and no single clue on how it was built nor by whom.

[rych]

Some prehistoric remarks regarding the /sbin/init:
Re: [IMPORTANT] Boot loader for next release. (Post by fanthom #538):

yes - 64bit version of Porteus currently uses 32bit /sbin/init as we need statically compiled executable for proper shutdown procedure (otherwise aufs will be 'busy').
64bit, compiled statically /sbin/init gives segfaults so i decided to go with 32bit and works nice

Re: [HOWTO] trouble shooting boot problems (Post by Ahau #4353):

/sbin/init from 001-core must be copied to the ramdisk in order to allow the proper shutdown procedure

Indeed it's still 32bit and statically linked in Porteus 5.0:

Code: Select all

root@porteus:/sbin# file init
init: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, stripped

As for unmounting AUFS:
https://askubuntu.com/a/487100 :

You can never unmount root filesystem, sorry -- no matter if you're using squashfs+aufs, or just plain ext2/ext3/ext4... What you can do is remount it read-only, which is the same from data integrity standpoint... note that aufs is somewhat different as you should make sure the rw-backing of it still writeable while aufs itself is being put read-only... for example, assuming /mnt/sq/rw is ext4 read-write part of your aufs "/" mount:... In short, you flush to disk and disable aufs features that require writing to ext4 overlay, then force ext4 itself to sync and go read-only, so it will be marked as cleanly unmounted next time you boot.

and this
https://superuser.com/questions/903963/ ... filesystem

[porteux]

@rych, thanks a lot! It's funny because I was looking at the same information but from different sources (although from the same person!): https://www.slax.org/blog/18417-Switchi ... sybox.html

So, yes, compiling it statically fixes the problem! The challenge now is learning how to compile it against uClibc so we can have sane file sizes (without it static /sbin/init takes 960 kb).

Nice finding!

[Ed_P]

I agree, nice finds rych.

The challenge now is learning how to compile it against uClibc so we can have sane file sizes (without it static /sbin/init takes 960 kb).

Remember, fixing things that are working is not as productive as fixing things not working.

[rych]

static /sbin/init takes 960 kb

That's so much larger than Porteus's 89.7 kb, is it because it's 64-bit now? Also, I wonder why static build correctly unmounts? Is it because reliance on dynamic libraries also puts a hold on the filesystem being unmounted?

Back to the 960 kb "problem", even if it can't be helped, it's no problem in 2023 anymore. For example, my current "USB stick" is 1 freaking Terabyte, plugged in a super fast USB 3.1 10Gbps port which is even faster than some of the host system disks! See Booting from NVMe USB enclosure

[porteux]

Using buildroot environment generated a /sbin/init with less than 100 kb.

@rych, yes, that's the reason why building a static version fixes the issue. It makes sense, right?

Unfortunately, since kernel 6.3.x the union can't be properly unmounted anymore (see here). I don't know if there's anything we can do to address that.