Creating a fully automated and versioned build environment

New features which should be implemented in Porteus; suggestions are welcome. All questions or problems with testing releases (alpha, beta, or rc) should go in their relevant thread here, rather than the Bug Reports section.
Post Reply
Sitwon
White ninja
White ninja
Posts: 16
Joined: 27 Jun 2011, 21:24
Location: Silver Spring, MD USA

Creating a fully automated and versioned build environment

Post#1 by Sitwon » 11 Jul 2011, 00:57

@fantom & brokenman
First of all, thanks a lot for publishing your development trees. It's a great help having that to look at as a resource. Unfortunately, there still seem to be a few gaps.

https://github.com/Sitwon/Porteus/tree/master/base

As you can see I have extracted lists of packages for each module as well as a list of the file remaining after those packages are all removed. Not including kernel modules there are about 150MB worth of unaccounted for files left over in the module trees. Not all of these files are accounted for in your development trees, either.

Additionally, there were a number of packages that didn't come from Slackware. Any hints where I can find the original packages or SlackBuilds for these?
https://gist.github.com/1075128

I can guess the ones labeled ponce are probably from http://ponce.cc/slackware/slackware-current/packages/ but what about the rest?

I also found in several cases some of the file described in the package weren't actually in the module. Am I correct in assuming the launch_before_go_* scripts are responsible for all of the pruning that takes place?

Advice? Hints? Suggestions?

Thanks,
Ben

User avatar
fanthom
Site Admin
Site Admin
Posts: 4546
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland, currently - Cork, IE
Contact:

Re: Creating a fully automated and versioned build environme

Post#2 by fanthom » 11 Jul 2011, 07:33

Hello Sitwon,

i see that you are referring to 32bit edition so i'll answer to some questions only as i dont know all the answers.
all non Slackware packages are compiled from sources. some are done with help of slackbuilds.org some are compiled "by hand". to track ones which comes from slackbuilds.org site go to /var/log/packages directory and issue:

Code: Select all

grep SlackBuild *
command. you should be able to recreate them easily.
for ones compiled by hand i unfortunately dont have a slackbuilds. this is my custom scrip which i'm using for compilations:
http://pastebin.com/YKveMBzY
(this script is setup for 64bits so little tweaking for i486 is necessary)
source files must be obtained from various sites individually (for example Mplayer).

packages containing the firmware (like ipw3945-ucode-1.14.2-fw-1ftm) are $arch independent and are converted from gentoo/debian repos:
http://distfiles.gentoo.org/distfiles/
http://ftp.de.debian.org/debian/pool/no ... e-nonfree/

mozilla-firefox-4.0.1-i486-1_slack13.37 comes from security backports: slackware-current -> slackware-13.37
ftp://ftp.heanet.ie/mirrors/ftp.slackwa ... /packages/

flashplayer and firefox addons are not listed in /var/log/packages as i didn't create slackware txz package for them - sorry about that. They all comes from 3rd party sites.

hope that helps a little.
Please add [Solved] to your thread title if the solution was found.

User avatar
wread
Module Guard
Module Guard
Posts: 1061
Joined: 09 Jan 2011, 18:48
Distribution: Porteus v3.2.5-kde5-64 bits
Location: Santo Domingo
Contact:

Re: Creating a fully automated and versioned build environme

Post#3 by wread » 11 Jul 2011, 12:54

Hmmm.., sounds interesting...., let's hear more about it!
Porteus is proud of the FASTEST KDE ever made.....(take akonadi, nepomuk and soprano out and you will have a decent OS).
The Porteus Community never sleeps!

Sitwon
White ninja
White ninja
Posts: 16
Joined: 27 Jun 2011, 21:24
Location: Silver Spring, MD USA

Re: Creating a fully automated and versioned build environme

Post#4 by Sitwon » 11 Jul 2011, 20:30

So what I'd like to propose is that under the base/ directory there would be a directory for each module.
000-kernel/
001-core/
002-xorg/
... and so on

Within each module directory you would have
build/ - The module's root directory. Files that aren't coming from packages should be placed in here and tracked.
pkgs/ - Initially this folder will just contain the various .t?z packages to be installed to the build directory, but eventually it should hold the SlackBuilds for each of those packages instead (SBo style).
build.sh - The build script that installs each of the .t?z packages and performs any additional post-install cleanup that is required.
Makefile - A make file that will build the packages, if necesary, from the SlackBuilds and then run the build.sh and create the module.

Ideally, on a clean checkout one should be able to call 'make' from the root and it should build a fully working porteus.iso. Eliminating the manual steps and hand-tweaking will result in a more reliable and easier to maintain build environment. As well as being easier to track in a repository (and all the benefits that implies).

User avatar
brokenman
Site Admin
Site Admin
Posts: 5436
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: Creating a fully automated and versioned build environme

Post#5 by brokenman » 12 Jul 2011, 01:05

What you describe sounds pretty much what i use for my own setup (minus the makefile). As i mentioned earlier i was playing around with an SVN for Porteus and in the end, it is just much easier to offer the ISO. The end user has to download much more than 300Mb otherwise.

Each new release changes slightly and there are always little tweaks here and there to be done. It would be ideal if one could just run the makefile and go have a beer ... but i think this method would just generate more work in the form of uploading various files and editing the makefile to accommodate the required tweaks.

My original idea involved having just the tree, custom build scripts and and a main script available through svn to download whatever packages were required and build them locally, all being done on the current version of Porteus. Other more important things came up so i never got around to realizing it. I hope you come up with something usable ... it would be good to see version control.
How do i become super user?
Wear your underpants on the outside and put on a cape.

User avatar
fanthom
Site Admin
Site Admin
Posts: 4546
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland, currently - Cork, IE
Contact:

Re: Creating a fully automated and versioned build environme

Post#6 by fanthom » 12 Jul 2011, 07:18

i have a similar experience to the brokenman - every new release (even beta,rc) bring bunch of tweaks which are this particular release specific. i'm worrying that 'automated build environment' would double the job as one need to:
a) fix the error
b) maintain the script

Second thing: new/removed/updated packages requires resolving of deps - how do you want to deal with this?

Sitwon - please dont think that i'm trying to stop you :)
i really wish to have one script which does everything automatically and save my free time....

i keep my fingers crossed for this project and will try to help as much as i can.
Please add [Solved] to your thread title if the solution was found.

Sitwon
White ninja
White ninja
Posts: 16
Joined: 27 Jun 2011, 21:24
Location: Silver Spring, MD USA

Re: Creating a fully automated and versioned build environme

Post#7 by Sitwon » 12 Jul 2011, 12:58

brokenman wrote:What you describe sounds pretty much what i use for my own setup (minus the makefile). As i mentioned earlier i was playing around with an SVN for Porteus and in the end, it is just much easier to offer the ISO. The end user has to download much more than 300Mb otherwise.
I still think it should be offered as an ISO for the end user, but for development it is useful to have tracked in a system like this.
brokenman wrote:Each new release changes slightly and there are always little tweaks here and there to be done. It would be ideal if one could just run the makefile and go have a beer ... but i think this method would just generate more work in the form of uploading various files and editing the makefile to accommodate the required tweaks.
Rather than generating more work, I expect that it will make things easier. A lot of the steps right now are rather repetitive so scripts will help relieve you of that work. Furthermore, by making your changes in scripts rather than manually you are automatically documenting and tracking every change you make.
fanthom wrote:Second thing: new/removed/updated packages requires resolving of deps - how do you want to deal with this?
Well as a long-time Slackware user I'm not quite ready to suggest any automated dependency resolution. The SlackBuild author(s) would need to make note of any compile-time and run-time dependencies in the README and the user building the package should make sure the required dependencies are installed on their development system. (This is one reason I would like to maintain binary compatibility with Slackware.)

User avatar
brokenman
Site Admin
Site Admin
Posts: 5436
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: Creating a fully automated and versioned build environme

Post#8 by brokenman » 12 Jul 2011, 14:24

I still think it should be offered as an ISO for the end user, but for development it is useful to have tracked in a system like this.
Understood
Rather than generating more work, I expect that it will make things easier. A lot of the steps right now are rather repetitive so scripts will help relieve you of that work.
Correct, alot of steps are repetitive ... but i use scripts or oneliners of code to automate these steps.
Furthermore, by making your changes in scripts rather than manually you are automatically documenting and tracking every change you make.
I like the documenting side of it, however it really does require more work ... we have to find and fix the bugs anyway, and then maintain/update the script. Updating the script means running to be sure it works ok in every situation, on every computer.

I have dep resolution under control here. I have a few scripts that will search for and download deps. I agree this would be a nice addition to Porteus ... and a few people would find it very useful, however for the amount of people that would use it (compared to downloading an ISO), i am not prepared to put in the hard yards. I am also happy to assist in any way i can.
How do i become super user?
Wear your underpants on the outside and put on a cape.

Tonio
Contributor
Contributor
Posts: 266
Joined: 28 Dec 2010, 16:37
Distribution: Slackware,porteus,FreeBSD,Slax
Location: 127.0.0.1

Re: Creating a fully automated and versioned build environme

Post#9 by Tonio » 14 Jul 2011, 02:00

Wouldn't it be much easier if one just

1) downloaded Slackware 13.37 or current
2) installed/deinstalled packages to your liking(list of packages in official build)
3) ran a ./build-porteus command aka fanthom's specialized/modified linux-live scripts
4) porteus iso built that matches the one on the site.

This way it would satisfy the build requirement/environment.

If you go package by package, module by module, there are many customizations that are difficult to account for. Just my humble opinion. What do others think?

Sitwon
White ninja
White ninja
Posts: 16
Joined: 27 Jun 2011, 21:24
Location: Silver Spring, MD USA

Re: Creating a fully automated and versioned build environme

Post#10 by Sitwon » 25 Jul 2011, 21:00

My squashfs-tools SlackBuild was accepted at SBo. I suggest future releases use this.

http://slackbuilds.org/repository/13.37 ... hfs-tools/

Tonio
Contributor
Contributor
Posts: 266
Joined: 28 Dec 2010, 16:37
Distribution: Slackware,porteus,FreeBSD,Slax
Location: 127.0.0.1

Re: Creating a fully automated and versioned build environme

Post#11 by Tonio » 26 Jul 2011, 19:43

@Sitwon,

Excellent & Congratulations :) This will make cd creation much easier I hope. Now it would probably be too much to ask, maybe find a way to convert Tomas(linux-live.org) or Fanthom's(porteus & Brokenman's specialized scripts) in here and make some packages. Now I believe I understand more of what you were heading to :) :D

@ALL

How about if the experienced users & others who don't mind playing, take a look at what Tomas use(ed) to build Slax. There was a build script that is referred in the Slax's Future thread:


Is this a goal to have, a powerful script that can build the base, or the whole? I know that more folks contribute and Fanthom & Brokenman put it together :) :D But work from many (sorry there are more folks and if I mention some names and leave some out, it would not be fair :( ) Studying this script and adapting it might help? This is just an idea. And I hope it helps in the ultimate goal :)

User avatar
fanthom
Site Admin
Site Admin
Posts: 4546
Joined: 28 Dec 2010, 02:42
Distribution: Porteus Kiosk
Location: Poland, currently - Cork, IE
Contact:

Re: Creating a fully automated and versioned build environme

Post#12 by fanthom » 26 Jul 2011, 21:05

i believe we have all features from Tomas's 'build script' included in our 'cleanup' scripts:
- removing of unnecessary files
- unzipping man pages and stripping binaries
- exporting *.a files and headers to devel module

extra features included in our version are:
- exporting of localized man pages and translations to external module (downloadable by LST)
- using .diff patches to apply Porteus specific settings to system configuration (like adding guest account, adding /opt/porteus-scripts to $PATH variable, etc...). patches are more flexible that static files as they are not overwriting changes made upstream.
(this feature is not included yet - will be in the cleanup scripts for Porteus-1.1)

i think 'automated build env' will incorporate all mentioned above.

Thanks Tonio.
Please add [Solved] to your thread title if the solution was found.

Sitwon
White ninja
White ninja
Posts: 16
Joined: 27 Jun 2011, 21:24
Location: Silver Spring, MD USA

Re: Creating a fully automated and versioned build environme

Post#13 by Sitwon » 01 Aug 2011, 18:53

Here is a relatively simplistic example of how I would ultimately like this to work.
https://github.com/Sitwon/Byzantium/com ... 02b69238a7

Currently it only handles SBo-style packages, I need to finish writing support for Pat's and Ponce's trees as well.

Post Reply