How do you make Porteus so small?

Non release banter
Post Reply
GullibleJones
White ninja
White ninja
Posts: 13
Joined: 10 Jan 2012, 00:10
Location: The Independent Republic of Massachusetts

How do you make Porteus so small?

Post#1 by GullibleJones » 08 May 2012, 14:36

Recently I experimented with making a custom live CD using Remastersys on a Debian system with Xfce. Imagine my surprise when the ISO came out to over 560 MB!

This is especially weird because:
- Debian + Xfce should really have fewer libraries floating around than Slackware + Trinity
- Debian has dependency resolution, so only what's needed gets installed
- Debian puts headers in separate packages (which I didn't install)
- My Debian installation basically had all the same system stuff as Porteus: CUPS, wicd, etc.

And yet... the ISO image is still huge. What did you guys do to get an image smaller than 250 MB? Is LZMA compression that good?

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: How do you make Porteus so small?

Post#2 by Ahau » 08 May 2012, 15:03

It's a combination of factors, and a lot of effort and thought put into keeping the size down. LZMA2+squashfs is pretty awesome for compression, so that helps. Dependency resolution in your package manager will make your ISO bigger, not smaller, because if you are manually putting your packages together you can choose to leave out some dependencies (i.e. leave them unresolved, and there are a couple in Porteus like this) or compile them out. Once you learn the ins and outs, you can put together a lean system without automatic dep resolution. There's also a bunch of files included in many packages that aren't needed for running the system, and a lot of those are stripped out of porteus.


When I started working with XFCE 4.8, I think the whole module was around 35MB, but it's down to just under 10MB in 1.2 RC2.
Please take a look at our online documentation, here. Suggestions are welcome!

GullibleJones
White ninja
White ninja
Posts: 13
Joined: 10 Jan 2012, 00:10
Location: The Independent Republic of Massachusetts

Re: How do you make Porteus so small?

Post#3 by GullibleJones » 08 May 2012, 15:31

Wow, that is quite impressive. I think Thunar alone drags in more than that on Debian!

Out of curiosity, can you provide any specifics on what sort of stuff gets stripped out? e.g. what would you leave out of, say, the Firefox module?

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: How do you make Porteus so small?

Post#4 by Ahau » 08 May 2012, 15:50

Using xfce as an example, I cut out a number of unnecessary dependencies (such as gconf, ORBit, gnome-keyring, guile, glade3 and everything gstreamer related), all locales files (/usr/share/locale/*) and localized man pages get split off into a locales module (for use with Porteus Language-Selection-Tool), and I drop everything from /usr/info and /usr/doc.

Firefox isn't as good of an example because there are fewer packages and less to cut out (but localization and documentation might be a good place to start if you don't need them).
Please take a look at our online documentation, here. Suggestions are welcome!

GullibleJones
White ninja
White ninja
Posts: 13
Joined: 10 Jan 2012, 00:10
Location: The Independent Republic of Massachusetts

Re: How do you make Porteus so small?

Post#5 by GullibleJones » 08 May 2012, 16:20

Thanks.

Doing some size comparisons...

/usr/lib is not much larger on Debian than on Porteus Xfce 1.2 RC2. And /usr/bin is a little smaller on Debian.

/usr/share/locale is huge on Debian. On Porteus it's empty. Which makes sense, though complete removal of localization is IMO a heavy price.

/usr/share/doc is also fairly large on Debian, and mostly empty on Porteus.

It looks to me like running localepurge on Debian should solve most of the space wastage problem... Though again, I'm not entirely sure if it's worth it. I assume glibc locales can be reinstalled later?

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: How do you make Porteus so small?

Post#6 by Ahau » 08 May 2012, 16:30

Yes, in an effort to minimize the size of the ISO, we strip the localization files, including glibc locales, and generate a "locales" module with those stripped files. If a user wants to have Porteus working in another language, they can use the Language-Selection-Tool, which allows the user to select their language, keyboard layout, fonts, etc,. and the tool downloads all the needed files and creates a custom module. Typically, we only have Language-Selection-Tool fully operational in a final release (not in development versions), but if you're running 1.2 RC2 you should still be able to run through the process and see how it operates.

What compression method are you using for your ISO?
Please take a look at our online documentation, here. Suggestions are welcome!

GullibleJones
White ninja
White ninja
Posts: 13
Joined: 10 Jan 2012, 00:10
Location: The Independent Republic of Massachusetts

Re: How do you make Porteus so small?

Post#7 by GullibleJones » 08 May 2012, 17:03

Compression method? Don't know. I would guess zlib or some such.

BTW, removing locales brought it down to 489 MB. Somewhat better but not ideal.

(Now let's see what clearing the doc files does...)

User avatar
Hamza
Warlord
Warlord
Posts: 1847
Joined: 28 Dec 2010, 07:41
Distribution: Porteus
Location: France

Re: How do you make Porteus so small?

Post#8 by Hamza » 08 May 2012, 18:01

lol, zlib is mostly used in PHP to compress data before to send them to browser and make "faster" the page loading and also increase the cpu load when you have a high traffic ;)
NjVFQzY2Rg==

GullibleJones
White ninja
White ninja
Posts: 13
Joined: 10 Jan 2012, 00:10
Location: The Independent Republic of Massachusetts

Re: How do you make Porteus so small?

Post#9 by GullibleJones » 08 May 2012, 22:36

Humm. Maybe gzip then?

Anyway stripping out the doc files brought it to 427 MB... And I really have to complement you guys on the job you did shrinking your distro. This is more difficult than I expected. :D

(If you're wondering what I'm up to, I'm trying to make something like Slax Popcorn Edition - only capable of a full hard drive install, with fully functional package management afterwards. I do suppose the Porteus/Slax module-based model may be better for usability in some ways, but in terms of security I would think it's better to have normal package management!)

User avatar
brokenman
Site Admin
Site Admin
Posts: 5574
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: How do you make Porteus so small?

Post#10 by brokenman » 09 May 2012, 02:56

Here is an example of some of the stuff that gets removed:

Code: Select all

rm usr/man/man2/* 2>/dev/null
rm usr/man/man3/* 2>/dev/null
rm usr/man/man4/* 2>/dev/null
rm usr/man/man6/* 2>/dev/null
rm usr/man/man9/* 2>/dev/null
rm -r usr/share/gtk-2.0/demo
rm -r usr/share/gtk-doc
rm -rf /usr/doc/*
rm -rf /usr/share/doc/*
All the man pages left are gunzipped as lzma does a better job and all ELF symbols are reduced.

Code: Select all

find ./ -type d | xargs chmod -v 755 >/dev/null 2>&1
echo "Extracting All manpages..."
find . | grep .gz | xargs gunzip >/dev/null 2>&1
echo "stripping Unneeded Symbols..."
find . | xargs file | grep ELF | cut -f 1 -d : | xargs strip --strip-unneeded >/dev/null 2>&1
Then they are removed from the distro for inclusion in the language selection tool where users can download them as needed.
# move man files for export to LST

Code: Select all

[ -d $basef/$pver/export/man ] || mkdir -p $basef/$pver/export/man
for x in `ls usr/man | egrep -v "man|cat"`; do
    cp -r usr/man/$x $basef/$pver/export/man
    rm -rf usr/man/$x
done
We leave all header files in place now, but used to remove them for inclusion in the 'development' module. These files (a.out and .h files) are really only needed for compiling stuff.

Finally all .a files are stripped and locales are exported for LST too.

Code: Select all

echo "Stripping down ..."
find . -name "*.a" | xargs -n 1 strip -g
locales=$basef/$pver/export/
cp -r usr/share/locale $locales && rm -r usr/share/locale/*
I hope that helps a little in your mission.
I would think it's better to have normal package management!
Not sure what you mean by 'normal'. A good package manager should pull packages from a trusted source, resolve dependencies and make life easy for the user. That's what our package managers do ... our main manager does a little extra too. :wink:
How do i become super user?
Wear your underpants on the outside and put on a cape.

GullibleJones
White ninja
White ninja
Posts: 13
Joined: 10 Jan 2012, 00:10
Location: The Independent Republic of Massachusetts

Re: How do you make Porteus so small?

Post#11 by GullibleJones » 09 May 2012, 03:24

Thanks! :D
brokenman wrote:Not sure what you mean by 'normal'. A good package manager should pull packages from a trusted source, resolve dependencies and make life easy for the user. That's what our package managers do ... our main manager does a little extra too. :wink:
But I would expect the combination of modules and (possibly overlapping) installable packages could potentially lead to all kinds of problems, no?

User avatar
brokenman
Site Admin
Site Admin
Posts: 5574
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: How do you make Porteus so small?

Post#12 by brokenman » 09 May 2012, 03:55

If you are using the official porteus repo this won't be a problem as modules are kept up to date to prevent library clashes. If you are pulling from slackware/salix/debian/fido and mixing old and new then only the god of linux knows what could happen. So far i have not seen or had reported any problems in this area ... but for what it's worth your theory is correct.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Ahau
King of Docs
King of Docs
Posts: 1331
Joined: 28 Dec 2010, 15:18
Distribution: LXDE & Xfce 32/64-bit
Location: USA

Re: How do you make Porteus so small?

Post#13 by Ahau » 11 May 2012, 22:02

As a follow up on squashfs+xz compression, I'm messing with a module now (002-xorg.xzm for arm), which is 240MB when extracted, and 60MB when compressed...
Please take a look at our online documentation, here. Suggestions are welcome!

Post Reply