XZ (LZMA2) compression added to 2.6.38-rc1 kernel

[fanthom]

Besides of XZ great news - 2.6.38 seems to be really an interesting release:
http://www.phoronix.com/scan.php?page=n ... &px=OTAyNw

Q. What does it meant for Porteus project?
A. TROUBLES

Q. Why?
A. LZMA and XZ are not compatible and conversion between them is necessary (damn - again!?)

Q. Do we need to use XZ?
A. Don't know yet but probably yes as i believe that everyone gonna switch from LZMA to XZ and LZMA support for squashfs will be simply dropped (2.6.36 and 2.6.37 kernels had Lzma support ported from 2.6.35 kernel) in favor of XZ.

Q. How wide is XZ kernel adoption?
A. Everything can be compressed with XZ algorithm: vmlinuz (kernel), initrd.xz (not lz), and modules (XZM instead of LZM???)
I'm wonder what other live distros gonna do: stick to LZM name (for historical reasons) or switch to XZM?

i was really excited with this news so i have tried 2.6.38-rc1 straight away. unfortunately Aufs doesn't support it yet (but Junjiro is working on it) so couldn't test Porteus with XZ compressed modules.
Philip Lougher prepared squashfs-tools (latest cvs pull) for xz support so simple compression/decompression tests were possible:
LZMA compression:

Code: Select all

time mksquashfs squashfs-root/ 001-core.lzm -b 256K
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on 001-core.lz, block size 262144.
[======================================================================================================================\] 14162/14162 100%
Exportable Squashfs 4.0 filesystem, lzma compressed, data block size 262144
     compressed data, compressed metadata, compressed fragments, compressed xattrs
     duplicates are removed
Filesystem size 54077.52 Kbytes (52.81 Mbytes)
XXXXXXXXXXXXXXXXXXXXXXXXXXX

real 0m51.547s

XZ compression (with x86 optimization filter)

Code: Select all

time mksquashfs squashfs-root/ 001-core.xzm -b 256K -Xbcj x86
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on 001-core.xz, block size 262144.
[======================================================================================================================/] 14162/14162 100%
Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
     compressed data, compressed metadata, compressed fragments, compressed xattrs
     duplicates are removed
Filesystem size 52548.66 Kbytes (51.32 Mbytes)
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
real 1m40.081s

Result: LZMA compressed 001-core into 52.8 MB module in 51 seconds while XZ needed 100 seconds but size was 1.5 MB smaller
Compression time does not matter - you are doing it only once. The more important things are: size and decompression time (Porteus speed).
Lets check decompression now.
LZMA:

Code: Select all

time unsquashfs 001-core.lzm 
Parallel unsquashfs: Using 2 processors
15890 inodes (15805 blocks) to write

[======================================================================================================================/] 15805/15805 100%
created 14019 files
created 774 directories
created 1636 symlinks
created 6 devices
created 1 fifos

real 0m3.282s

XZ:

Code: Select all

time unsquashfs 001-core.xzm 
Parallel unsquashfs: Using 2 processors
15890 inodes (15805 blocks) to write

[======================================================================================================================-] 15805/15805 100%
created 14019 files
created 774 directories
created 1636 symlinks
created 6 devices
created 1 fifos

real 0m2.731s

WOW!
LZMA needed 3.282s while XZ only 2.731s which is nearly 17% faster
Cant wait to test Porteus with 2.6.38 kernel and XZ compressed modules.

BTW - I think we can change the roadmap a bit and ship Porteus-1.0 with 2.6.38 kernel instead of 2.6.37.x

[brokenman]

Exciting. I look forward to playing with the new compression. The faster decompression looks to be the positive thing.

Please keep us up to date as to when junjiro has cracked the egg.

[KSA_ARAB]

Good news !

I think we can change "LZM" to "XZM", because it's not first time to change extension we converted "MO" to "LZM" before on old slax,
also we convert slackware "TXZ" to "LZM" currently , so..

as you said "compression will be once", and keep going with new kernel always.

thank you.

[brokenman]

I agree. Keep up with the times. Anything with an X in it is way cooler.

[Quax]

...I think we can change "LZM" to "XZM", because it's not first time to change extension we converted "MO" to "LZM" before on old slax,
also we convert slackware "TXZ" to "LZM" currently...

Remembering my struggle with Tomas concerning lzmo/lzm, I'm glad that now there is a chance to make it transparent to the user which
module has been compressed with legacy lzma and which with xz.

This way users can separate Slax6 modules from Porteus and others.

Regards, Quax

p.s. I think we should 'inject' the idea of .xzm into other communities like Salix or Zenwalk and thus define a new standard

[X-Factor]

@Quax
Your fast sir and well said (salix n zen bit)
looking forward to testing alpha/beta release.

[Quax]

Hi all,

...well said (salix n zen bit)

those registered at other forums might help to spread the word and creating facts:

http://www.salixos.org/forum/viewtopic.php?f=32&t=1939
http://support.zenwalk.org/viewtopic.ph ... 62&start=0

Regards,

Quax

[fanthom]

Mr Junjiro just pushed out Monday Aufs update which allowed me to compile 2.6.38-rc2 kernel for Porteus!
Junjiro clearly states that 2.6.38 aufs support is not ready yet due to some upstream bugs but it didn't stop me from trying porteus with XZ compressed modules.
http://img5.imageshack.us/img5/8126/snapshot1wp.jpg

The result is INSANE:
a) default ISO size reduced by 12MB (from 300MB to 288MB)
b) booting (and overall system speed) increased as follows:
- Porteus-v09 boot to LXDE takes 27 seconds
- Porteus-v1.0-Alpha boot to LXDE takes 20 seconds

- Porteus-v09 boot to KDE takes 50 seconds
- Porteus-v1.0-Alpha boot to KDE takes 41 seconds

All test done in latest Vbox and my watch
'CONFIG_SCHED_AUTOGROUP' (200 lines patch which does wonders) really does wonders as i was compiling the kernel and listening music, browsing net, watching youtube without any significant responsiveness loss

there is only one thing which doesn't work correctly:
i wasn't able to use xz compressed initrd got this message during boot:

Code: Select all

input was encoded with settings that are not supported by this XZ decoder

i have packed initrd with default options: 'xz initrd' and got -> initrd.xz as a result
hopefully will be fixed for the final kernel release

Roadmap is officially changed now: we are shipping 1.0 release with 2.6.38 kernel!



EDIT:\\
@ brokenman
i think we can afford switching to .xzm extension. At the moment we are messing a loot anyway. better to introduce all major changes before 1.0 than later. tools rename/update should be easy. i can provide cli version of tool for lzma->xz conversion.
What do you think?

@Quax
are you planing to do the same?

@Falcony
Sorry about that but we need to move forward. I know it's a lot of work to repack all your modules, but...

Regards,
fanthom

[Hamza]

Good Works !

[Ahau]

Freaking awesome!

Might I suggest an email to the upstream developers, regarding initrd.xz?

Thanks!

[brokenman]

Excellent. I like the idea of changing to .xzm and the specs sound very good. After tidying up KDE-3.5.12 i will start work on conversion, kernel and initrd. Can you pass on the patches (and where you subscribe to hear the latest news so quickly?)

[Quax]

@Quax
are you planing to do the same?

Definitely!

When I'm finished with the repo preparations for FluxFlux-2011.1, I will test my backport of xz and squashfs to the 2.6.32.27 lts kernel.

Quax

[Falcony]

@fantom

Sorry about that but we need to move forward. I know it's a lot of work to repack all your modules, but...

I am not going to convert. Old modules I will leave AS IS - for old version of FIDOSlax and Porteus

Please kindly provide me beta version 32-bit with 38 kernel and new compression - even buggy one.

I need to start filling new repo as soon as possible


BTW, I found a little time for installation program for grub4dos at Windows.
A little bit messy - but may be I will pass some stripts to you.

[fanthom]

@brokenman
here you can find latest news about aufs:
http://blog.gmane.org/gmane.linux.file- ... .aufs.user

@Falcony
will send you download link for 32bit version of squashfs-tools with xz support + conversion script. you should be able to use it to create xzm modules for both 32/64bit editions.
great news with grub4dos - waiting impatiently

EDIT:\\
this may be useful for all repo maintainers so here are squashfs-tools with xz support + conversion script. just activate this module and then:
'lzm2xzm /path_to/lzm_folder
or
lzm2xzm /path_to/lzm_module'

Be aware that you wont be able to use xz compressed modules with Porteus-v09 as 2.6.37 kernel doesn't support xz compression. You have to wait on first beta of Porteus-1.0 with 2.6.38 kernel.
links:
http://www.mediafire.com/file/4dhi0i2yy ... z-i486.lzm
http://www.mediafire.com/file/irzb5cuos ... x86_64.lzm

you can do a sample benchmark if you want, example:

Code: Select all

cp /mnt/sda2/porteus/base/004-kde.lzm /root
lzm2xzm /root/004-kde.lzm

and now the best thing

Code: Select all

time unsquashfs 004-kde.lzm
time unsquashfs 004-kde.xzm

you should get a 10-25% faster result for uncompressing of xzm module

[Falcony]

@fantom

Thank you very much for conversion modules

Please kindly confirm that in next version of Porteus defalut extention for activation and deactivation will be xzm

And activate script will be work ONLY with *.xzm