zram for extension of usable space for aufs

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
User avatar
Quax
Full of knowledge
Full of knowledge
Posts: 21
Joined: 29 Dec 2010, 21:28
Distribution: FluxFlux (ARM + x86), Slax7
Location: Muelheim an der Ruhr, Germany
Contact:

zram for extension of usable space for aufs

Post#1 by Quax » 09 Jun 2012, 19:51

Hi all,

long time no post ;)

Preparing the next FluxFlux version I have integrated zram (successor of compcache) in linuxrc.

zram explained:

...creates a RAM based block device which acts as a swap disk, but is compressed and stored in memory instead of swap disk (which is slow), allowing very fast I/O and increasing the amount of memory available before the system starts swapping to disk.

Due to the fact that most netbooks today are memory constrained but have enough power to handle (de)compression
using zram is a perfect way ro increase the available virtual diskspace in live mode while using aufs.

On my test system (Atom Z520 with 2GB Ram) I could increase the ramsize in linuxrc from 60% to 80% while preserving
the same amount of available RAM while gaining 400MB more free diskspace. 8)

For those interested in here are the relevant parts from linuxrc/liblinuxlive.

liblinuxlive:

Code: Select all

...
# ==================================================================
# RAM compressing/optimizing function#s
# Author: Manfred Mueller, http://fluxflux.net
#
# Heavily based on zramswap-enabler from Sergey "Shnatsel" Davidoff,
# https://launchpad.net/~shnatsel
# ==================================================================

zram_start() {

   # get the number of CPUs
   num_cpus=$(cat /proc/cpuinfo | grep "cpu cores" | cut -d: -f2 | sed 's/ //g' | uniq)

   # set decremented number of CPUs
   decr_num_cpus=$(( $num_cpus - 1 ))

   # get the amount of memory in the machine
   mem_total_kb=$(grep MemTotal /proc/meminfo | rev | cut -d" " -f2 | rev)
   mem_total=$((mem_total_kb * 1024))

#   # determine modprobe parameter name, depending on kernel version
#   if (uname --release | grep -E --silent "^3\.[123]"); then
#   num_devices_parameter='zram_num_devices'
#   else
#   num_devices_parameter='num_devices'
#   fi

   # load zram modules
   modprobe zram num_devices=$num_cpus
	
   # initialize the devices, set total size to 1/2 RAM
   for i in $(seq 0 $decr_num_cpus); do
   echo $((mem_total / 2 / num_cpus)) > /sys/block/zram$i/disksize
   done
	
   # Creating swap filesystems
   for i in $(seq 0 $decr_num_cpus); do
   mkswap -L zram$i /dev/zram$i
   done

   # Switch the swaps on
   for i in $(seq 0 $decr_num_cpus); do
   swapon -p 100 /dev/zram$i
   done
}

zram_stop() {

   # Switching off swap
   for i in $(grep -E --only-matching '/dev/zram[[:digit:]]+' /proc/swaps); do
   swapoff $i
   done

   # remove module, wait instead of failing if it's busy
   rmmod --wait zram
}
...
linuxrc:

Code: Select all

...
# zram
if [ "$(cmdline_parameter zram)" != "" ]; then
if [ ! "$GERMAN_IS_ENABLED" ]; then
echolog "Using zram to increase usable RAM..."
else
echolog "Benutze zram zur Vergrößerung des nutzbaren Arbeitsspeichers..."
fi
fi

zram_start
...

RAMSIZE=$(cmdline_value ramsize)
if [ "$RAMSIZE" = "" ]; then 
   if [ ! "$(cmdline_parameter zram)" = "" ]; then
	    RAMSIZE="80%"
   else
	    RAMSIZE="60%"
   fi
fi
...
cleanup:

Code: Select all

...
# kill all processes which can be killed
killall_unneeded $(pidof cleanup)

# zram
if [ "$(cmdline_parameter zram)" != ""]; then
if [ ! "$GERMAN_IS_ENABLED" ]; then
echolog "disabling zram..."
else
echolog "Deaktiviere zram..."
fi
fi

zram_stop

...
Regards,

Quax
Last edited by Quax on 09 Jun 2012, 21:25, edited 1 time in total.
Erfahrung bedeutet gar nichts - man kann Dinge auch 15 Jahre lang falsch machen...

Experience means nothing - one could have done things wrong for the last 15 years...

User avatar
fanthom
Moderator Team
Moderator Team
Posts: 5666
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland
Contact:

Re: zram for extension of usable space for aufs

Post#2 by fanthom » 09 Jun 2012, 22:04

hi Quax,

funny thing - just today i started reading about zram and playing with it.
unfortunately my test didn't go far as seems that zram/zsmalloc is broken in 3.4.1 kernel:

Code: Select all

root@porteus:/home/guest/work# modprobe zram 
FATAL: Error inserting zram (/lib/modules/3.4.1-porteus/kernel/drivers/staging/zram/zram.ko): Unknown symbol in module, or unknown parameter (see dmesg)
root@porteus:/home/guest/work# dmesg | tail
[ 3803.031802] zsmalloc: module is from the staging directory, the quality is unknown, you have been warned.
[ 3803.031807] zsmalloc: module license 'unspecified' taints kernel.
[ 3803.031809] Disabling lock debugging due to kernel taint
[ 3803.031907] zsmalloc: Unknown symbol alloc_vm_area (err 0)
[ 3803.031925] zsmalloc: Unknown symbol free_vm_area (err 0)
[ 3803.031937] zsmalloc: Unknown symbol __supported_pte_mask (err 0)
root@porteus:/home/guest/work#
3.3.x kernel works ok but i'll probably go with 3.4.x for porteus-1.2 (also - zram is still in staging so i dont trust it fully).

anyway - my thoughts about zram (without any testing):
a) should be enabled only when user is not using 'changes=' cheat
b) should be enabled only when PC has 1GB or less of available RAM
reason
a) we need CPU power to decompress data from modules
b) porteus supports encryption which may cause extra CPU charge

i'll keep an eye on zram feature.

as always - thanks for sharing :)

Posted after 43 minutes 37 seconds:
i see now that you have made zram optional through 'zram' cheatcode.
this is a good approach but i would drop calculations and made it more flexible by letting users decide about number of devices and their size.
something like "zram=2:300M"
where "2" would be a number of devices and "300M" a size for each.

Cheers

EDIT:\\
kernel issue solved - zram depends on zcache which can be enabled only if you compile CRYPTO as 'y' and not 'm' - ehhhh....
Please add [Solved] to your thread title if the solution was found.

User avatar
brokenman
Site Admin
Site Admin
Posts: 6105
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v4 all desktops
Location: Brazil

Re: zram for extension of usable space for aufs

Post#3 by brokenman » 09 Jun 2012, 23:02

Nice bleeding edge work! I guess this is perfect for those small factor devices, and other portable devices that will no doubt become more popular with time. Personally i like my computers big and my women small.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
Quax
Full of knowledge
Full of knowledge
Posts: 21
Joined: 29 Dec 2010, 21:28
Distribution: FluxFlux (ARM + x86), Slax7
Location: Muelheim an der Ruhr, Germany
Contact:

Re: zram for extension of usable space for aufs

Post#4 by Quax » 10 Jun 2012, 10:24

@brokenman:

This is totally OT but some members depending on their member category maybe feeling offended by your signature...
How do i become super user?
Wear your underpants on the outside and put on a cape ...
Porteus admins should definitely think about changing the star beneath Image to a figure wearing a cape and underpants on the outside... :crazy:

Quax
Erfahrung bedeutet gar nichts - man kann Dinge auch 15 Jahre lang falsch machen...

Experience means nothing - one could have done things wrong for the last 15 years...

Post Reply