XZ (LZMA2) compression added to 2.6.38-rc1 kernel

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#1 by fanthom » 20 Jan 2011, 23:39

Besides of XZ great news - 2.6.38 seems to be really an interesting release:
http://www.phoronix.com/scan.php?page=n ... &px=OTAyNw

Q. What does it meant for Porteus project?
A. TROUBLES

Q. Why?
A. LZMA and XZ are not compatible and conversion between them is necessary (damn - again!?)

Q. Do we need to use XZ?
A. Don't know yet but probably yes as i believe that everyone gonna switch from LZMA to XZ and LZMA support for squashfs will be simply dropped (2.6.36 and 2.6.37 kernels had Lzma support ported from 2.6.35 kernel) in favor of XZ.

Q. How wide is XZ kernel adoption?
A. Everything can be compressed with XZ algorithm: vmlinuz (kernel), initrd.xz (not lz), and modules (XZM instead of LZM???)
I'm wonder what other live distros gonna do: stick to LZM name (for historical reasons) or switch to XZM?

i was really excited with this news so i have tried 2.6.38-rc1 straight away. unfortunately Aufs doesn't support it yet (but Junjiro is working on it) so couldn't test Porteus with XZ compressed modules.
Philip Lougher prepared squashfs-tools (latest cvs pull) for xz support so simple compression/decompression tests were possible:
LZMA compression:

Code: Select all

time mksquashfs squashfs-root/ 001-core.lzm -b 256K
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on 001-core.lz, block size 262144.
[======================================================================================================================\] 14162/14162 100%
Exportable Squashfs 4.0 filesystem, lzma compressed, data block size 262144
     compressed data, compressed metadata, compressed fragments, compressed xattrs
     duplicates are removed
Filesystem size 54077.52 Kbytes (52.81 Mbytes)
XXXXXXXXXXXXXXXXXXXXXXXXXXX

real 0m51.547s
XZ compression (with x86 optimization filter)

Code: Select all

time mksquashfs squashfs-root/ 001-core.xzm -b 256K -Xbcj x86
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on 001-core.xz, block size 262144.
[======================================================================================================================/] 14162/14162 100%
Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
     compressed data, compressed metadata, compressed fragments, compressed xattrs
     duplicates are removed
Filesystem size 52548.66 Kbytes (51.32 Mbytes)
XXXXXXXXXXXXXXXXXXXXXXXXXXXX
real 1m40.081s
Result: LZMA compressed 001-core into 52.8 MB module in 51 seconds while XZ needed 100 seconds but size was 1.5 MB smaller :)
Compression time does not matter - you are doing it only once. The more important things are: size and decompression time (Porteus speed).
Lets check decompression now.
LZMA:

Code: Select all

time unsquashfs 001-core.lzm 
Parallel unsquashfs: Using 2 processors
15890 inodes (15805 blocks) to write

[======================================================================================================================/] 15805/15805 100%
created 14019 files
created 774 directories
created 1636 symlinks
created 6 devices
created 1 fifos

real 0m3.282s
XZ:

Code: Select all

time unsquashfs 001-core.xzm 
Parallel unsquashfs: Using 2 processors
15890 inodes (15805 blocks) to write

[======================================================================================================================-] 15805/15805 100%
created 14019 files
created 774 directories
created 1636 symlinks
created 6 devices
created 1 fifos

real 0m2.731s
WOW!
LZMA needed 3.282s while XZ only 2.731s which is nearly 17% faster :)
Cant wait to test Porteus with 2.6.38 kernel and XZ compressed modules.

BTW - I think we can change the roadmap a bit and ship Porteus-1.0 with 2.6.38 kernel instead of 2.6.37.x :)
Please add [Solved] to your thread title if the solution was found.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#2 by brokenman » 20 Jan 2011, 23:46

Exciting. I look forward to playing with the new compression. The faster decompression looks to be the positive thing.

Please keep us up to date as to when junjiro has cracked the egg.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
KSA_ARAB
Black ninja
Black ninja
Posts: 52
Joined: 29 Dec 2010, 00:54
Location: Saudi Arabia
Contact:

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#3 by KSA_ARAB » 21 Jan 2011, 03:11

Good news !

I think we can change "LZM" to "XZM", because it's not first time to change extension we converted "MO" to "LZM" before on old slax,
also we convert slackware "TXZ" to "LZM" currently , so..

as you said "compression will be once", and keep going with new kernel always.

thank you.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#4 by brokenman » 21 Jan 2011, 03:28

I agree. Keep up with the times. Anything with an X in it is way cooler.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Quax
Full of knowledge
Full of knowledge
Posts: 21
Joined: 29 Dec 2010, 21:28
Distribution: FluxFlux (ARM + x86), Slax7
Location: Muelheim an der Ruhr, Germany
Contact:

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#5 by Quax » 21 Jan 2011, 03:29

KSA_ARAB wrote:...I think we can change "LZM" to "XZM", because it's not first time to change extension we converted "MO" to "LZM" before on old slax,
also we convert slackware "TXZ" to "LZM" currently...
Remembering my struggle with Tomas concerning lzmo/lzm, I'm glad that now there is a chance to make it transparent to the user which
module has been compressed with legacy lzma and which with xz.

This way users can separate Slax6 modules from Porteus and others. :Yahoo!:

Regards, Quax

p.s. I think we should 'inject' the idea of .xzm into other communities like Salix or Zenwalk and thus define a new standard :Bravo:
Erfahrung bedeutet gar nichts - man kann Dinge auch 15 Jahre lang falsch machen...

Experience means nothing - one could have done things wrong for the last 15 years...

User avatar
X-Factor
Black ninja
Black ninja
Posts: 88
Joined: 29 Dec 2010, 23:19
Location: bali, indonesia

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#6 by X-Factor » 21 Jan 2011, 03:39

@Quax
Your fast sir :good: and well said (salix n zen bit) :Yahoo!: :evil:
looking forward to testing alpha/beta release.
Last edited by X-Factor on 01 Feb 2011, 14:04, edited 1 time in total.
I've been confine to my quarter
http://img143.imageshack.us/i/medusathing.jpg/

User avatar
Quax
Full of knowledge
Full of knowledge
Posts: 21
Joined: 29 Dec 2010, 21:28
Distribution: FluxFlux (ARM + x86), Slax7
Location: Muelheim an der Ruhr, Germany
Contact:

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#7 by Quax » 21 Jan 2011, 09:20

Hi all,
X-Factor wrote:...well said (salix n zen bit)
those registered at other forums might help to spread the word and creating facts: :twisted:

http://www.salixos.org/forum/viewtopic.php?f=32&t=1939
http://support.zenwalk.org/viewtopic.ph ... 62&start=0

Regards,

Quax
Erfahrung bedeutet gar nichts - man kann Dinge auch 15 Jahre lang falsch machen...

Experience means nothing - one could have done things wrong for the last 15 years...

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#8 by fanthom » 24 Jan 2011, 12:13

Mr Junjiro just pushed out Monday Aufs update which allowed me to compile 2.6.38-rc2 kernel for Porteus! :Yahoo!:
Junjiro clearly states that 2.6.38 aufs support is not ready yet due to some upstream bugs but it didn't stop me from trying porteus with XZ compressed modules.
http://img5.imageshack.us/img5/8126/snapshot1wp.jpg

The result is INSANE:
a) default ISO size reduced by 12MB (from 300MB to 288MB)
b) booting (and overall system speed) increased as follows:
- Porteus-v09 boot to LXDE takes 27 seconds
- Porteus-v1.0-Alpha boot to LXDE takes 20 seconds

- Porteus-v09 boot to KDE takes 50 seconds
- Porteus-v1.0-Alpha boot to KDE takes 41 seconds

All test done in latest Vbox and my watch :)
'CONFIG_SCHED_AUTOGROUP' (200 lines patch which does wonders) really does wonders as i was compiling the kernel and listening music, browsing net, watching youtube without any significant responsiveness loss :D

there is only one thing which doesn't work correctly:
i wasn't able to use xz compressed initrd :( got this message during boot:

Code: Select all

input was encoded with settings that are not supported by this XZ decoder
i have packed initrd with default options: 'xz initrd' and got -> initrd.xz as a result
hopefully will be fixed for the final kernel release :)

Roadmap is officially changed now: we are shipping 1.0 release with 2.6.38 kernel!

:beer:

EDIT:\\
@ brokenman
i think we can afford switching to .xzm extension. At the moment we are messing a loot anyway. better to introduce all major changes before 1.0 than later. tools rename/update should be easy. i can provide cli version of tool for lzma->xz conversion.
What do you think?

@Quax
are you planing to do the same?

@Falcony
Sorry about that but we need to move forward. I know it's a lot of work to repack all your modules, but... :pardon:

Regards,
fanthom
Please add [Solved] to your thread title if the solution was found.

User avatar
Hamza
Warlord
Warlord
Posts: 1908
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#9 by Hamza » 24 Jan 2011, 13:25

Good Works !
NjVFQzY2Rg==

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#10 by Ahau » 24 Jan 2011, 15:04

Freaking awesome!

Might I suggest an email to the upstream developers, regarding initrd.xz?

Thanks!
Please take a look at our online documentation, here. Suggestions are welcome!

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#11 by brokenman » 24 Jan 2011, 20:15

Excellent. I like the idea of changing to .xzm and the specs sound very good. After tidying up KDE-3.5.12 i will start work on conversion, kernel and initrd. Can you pass on the patches (and where you subscribe to hear the latest news so quickly?)
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Quax
Full of knowledge
Full of knowledge
Posts: 21
Joined: 29 Dec 2010, 21:28
Distribution: FluxFlux (ARM + x86), Slax7
Location: Muelheim an der Ruhr, Germany
Contact:

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#12 by Quax » 25 Jan 2011, 13:12

fanthom wrote:@Quax
are you planing to do the same?
Definitely!

When I'm finished with the repo preparations for FluxFlux-2011.1, I will test my backport of xz and squashfs to the 2.6.32.27 lts kernel.

Quax
Erfahrung bedeutet gar nichts - man kann Dinge auch 15 Jahre lang falsch machen...

Experience means nothing - one could have done things wrong for the last 15 years...

Falcony
Full of knowledge
Full of knowledge
Posts: 237
Joined: 01 Jan 2011, 12:44
Location: Russia

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#13 by Falcony » 25 Jan 2011, 17:20

@fantom
Sorry about that but we need to move forward. I know it's a lot of work to repack all your modules, but...
I am not going to convert. Old modules I will leave AS IS - for old version of FIDOSlax and Porteus

Please kindly provide me beta version 32-bit with 38 kernel and new compression - even buggy one.

I need to start filling new repo as soon as possible


BTW, I found a little time for installation program for grub4dos at Windows.
A little bit messy - but may be I will pass some stripts to you.

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#14 by fanthom » 25 Jan 2011, 17:29

@brokenman
here you can find latest news about aufs:
http://blog.gmane.org/gmane.linux.file- ... .aufs.user

@Falcony
will send you download link for 32bit version of squashfs-tools with xz support + conversion script. you should be able to use it to create xzm modules for both 32/64bit editions.
great news with grub4dos - waiting impatiently :)

EDIT:\\
this may be useful for all repo maintainers so here are squashfs-tools with xz support + conversion script. just activate this module and then:
'lzm2xzm /path_to/lzm_folder
or
lzm2xzm /path_to/lzm_module'

Be aware that you wont be able to use xz compressed modules with Porteus-v09 as 2.6.37 kernel doesn't support xz compression. You have to wait on first beta of Porteus-1.0 with 2.6.38 kernel.
links:
http://www.mediafire.com/file/4dhi0i2yy ... z-i486.lzm
http://www.mediafire.com/file/irzb5cuos ... x86_64.lzm

you can do a sample benchmark if you want, example:

Code: Select all

cp /mnt/sda2/porteus/base/004-kde.lzm /root
lzm2xzm /root/004-kde.lzm
and now the best thing :)

Code: Select all

time unsquashfs 004-kde.lzm
time unsquashfs 004-kde.xzm
you should get a 10-25% faster result for uncompressing of xzm module :)
Please add [Solved] to your thread title if the solution was found.

Falcony
Full of knowledge
Full of knowledge
Posts: 237
Joined: 01 Jan 2011, 12:44
Location: Russia

Re: XZ (LZMA2) compression added to 2.6.38-rc1 kernel

Post#15 by Falcony » 25 Jan 2011, 20:03

@fantom

Thank you very much for conversion modules

Please kindly confirm that in next version of Porteus defalut extention for activation and deactivation will be xzm

And activate script will be work ONLY with *.xzm

Post Reply