Porteus and SVN

Non release banter
Post Reply
caliban
White ninja
White ninja
Posts: 9
Joined: 26 Jan 2011, 22:46
Location: USA

Porteus and SVN

Post#1 by caliban » 22 Feb 2011, 01:13

I am ecstatic over the fact that Porteus is using revision control. A huge improvement over SLAX. But I don't understand the reasoning behind committing binary (.xzm) files to the repo. Ideally the repo would consist of shell scripts/makefiles that pull source and compile to create the distro. The problem of tracking binaries especially in subversion will make the repo unwieldy and inefficient. The ability to atomically regress commits/patches and at a glance being able understand diffs is completely lost. Please consider this friendly advice from someone who codes/runs multiple repos for win32/linux projects, you want to avoid committing binaries. Please continue the great work.

User avatar
brokenman
Site Admin
Site Admin
Posts: 5456
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: Porteus and SVN

Post#2 by brokenman » 22 Feb 2011, 02:14

This was initially my plan and i have the local svn in place doing just this, however; unfortunately after several tens of failed attempts to commit the whole 1Gb of source to google code's svn, i gave up and decided to use the current system. This means when 1 single file is updated i need to commit the entire base module it belongs to. Much much less efficient i agree 100%, but my only option at this stage.
How do i become super user?
Wear your underpants on the outside and put on a cape.

caliban
White ninja
White ninja
Posts: 9
Joined: 26 Jan 2011, 22:46
Location: USA

Re: Porteus and SVN

Post#3 by caliban » 22 Feb 2011, 02:46

brokenman wrote:This was initially my plan and i have the local svn in place doing just this, however; unfortunately after several tens of failed attempts to commit the whole 1Gb of source to google code's svn, i gave up and decided to use the current system.
Can I ask what type of errors you were seeing ? Were they network related ? If so, you might want to consider loading up the files in batches.
I am a bit puzzled on how you got 1GB of source files, are you trying to upload let's say all of /usr, /usr/bin, etc.. ? An option might be to point to the files on a well known slackware repo, or host them on a ftp/file upload site/dropbox site in a big tarball, and then have a a pseudo script like below to bootstrap the dev environment.

Code: Select all

svn co http://site/repo/trunk /data
#check to see where my source files are and fetch them
if [ -f /data/source.lst ] then;
while $URL = /data/source.lst; do
  wget $URL
done
Or failing that use svn external dependancies, which would be just as ugly. Atleast that way you can keep all source files in one big tarball in the repo.

User avatar
brokenman
Site Admin
Site Admin
Posts: 5456
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: Porteus and SVN

Post#4 by brokenman » 22 Feb 2011, 20:10

I can't remember off the top of my head, but i think they were 500 errors. There were definitely certain files it didn't like and would drop out ... from memory in /dev
are you trying to upload let's say all of /usr, /usr/bin, etc.. ?
The entire Porteus OS is tied up in the base modules. When i say source files i mean that these have been unpacked, and the total decompressed size is over 1Gb. Offering anything less than the entire file set is a broken OS.
An option might be to point to the files on a well known slackware repo
I guess this is an option, but we don't use the standard packages offered by the repo for alot of the base modules. They are stripped down and in some cases only a few files used. This would over complicate the process.
or host them on a ftp/file upload site/dropbox site in a big tarball
Isn't that exactly what is being done now except in several smaller compressed files? The base modules are the bulk of the weight and they can be considered compressed files. One big tarball leaves me back at square one where one small file update requires an upload of the entire tarball. I already have my main Porteus tree (just the porteus additions external to slack) on a Dropbox account, but this is just as a backup. We tried it as a way of sharing files between maintainers but it didn't pan out.

I am open to ideas and thanks for sharing yours.
How do i become super user?
Wear your underpants on the outside and put on a cape.

caliban
White ninja
White ninja
Posts: 9
Joined: 26 Jan 2011, 22:46
Location: USA

Re: Porteus and SVN

Post#5 by caliban » 23 Feb 2011, 02:15

brokenman wrote:There were definitely certain files it didn't like and would drop out ... from memory in /dev
That makes sense as svn does not deal with character/block devices. Better off completely removing all /dev entries and recreating them after checkout/export with makedev.
brokenman wrote:The entire Porteus OS is tied up in the base modules. When i say source files i mean that these have been unpacked, and the total decompressed size is over 1Gb. Offering anything less than the entire file set is a broken OS.
I figured as much and agree with you on offering the entirety of the OS. Where we differ is in how best to accomplish this. I am advocating that sources/binary files be kept out of the svn repo. I am expanding on this below.
brokenman wrote:I guess this is an option, but we don't use the standard packages offered by the repo for alot of the base modules. They are stripped down and in some cases only a few files used. This would over complicate the process.
The end result of a stripped package is due to developers like yourself pruning files through manual means and/or hopefully scripts at this point. This effort should be recognized for the time consuming work and potential to impact the project. Right now that work is summarized by one line svn commits that impart little information. Why settle for that level of verbosity, when you could easily see the work done. Take for example, the omission/deletion of some trivial library (libfoo.so), in your current workflow on your machine you can determine the error fairly quickly by either running svn log or svn diff. In either case, the svn output will let you find the problem quickly. But for the other developers, they will have to do two checkouts/exports of a few compressed xzm files (~300MBx2) to even find this bug. All due to the fact, that svn cannot actually calculate deltas between binary files.

So what would a ideal repo/workflow look like ? Your standard svn repo configuration of trunk, branches and tags would be valid. The benefit of tags is easy enough to understand, as you can tag a certain revision as being stable, testing, nightly, 1.0, etc.. The branches make it easy to work on the next version of porteus perhaps even changing sources/packages without mucking up the trunk branch, which could be reserved for bugfixes. The trunk branch could consist of multiple script files for each individual package or one script file for each .xzm that processes all the packages necessary for that module. The processing could involve everything from removing/stripping files, compressing manpages to creating symbolic links. This might seem like a daunting task, but all the information to reduce files in a package can be obtained by comparing the package logs to the actual filesystem.

I understand the concern for over complicating the process, but I believe that establishing a good workflow will ease your future work. Here's a couple of payoffs off the top of my head.
1. AFAIK, current Porteus x86_64 is based on a slackware-current of some date unknown to me. What happens when slackware-current goes through a huge change (KDE4.7), How do you get updated/security packages in ? Would it not be easier to let loose the old versioned scripts against the new packages and see what breaks and what doesn't, instead of manually performing the individual package upgrade drudgery. I am assuming that work right now is still manual and not automated.
2. You could potentially attract more developers. SLAX is a good example of how a good project can be overcome by actual life. Porteus will need to attract lots of helping hands to overcome potential stagnancy. Fanthom's time spent understanding the entire build process was likely tedious, sparing other developers this pain will only benefit the project.

I am a bit rushed and hope that I have expressed my ideas somewhat clearly. The summary is basically to house recipe files in the repo and keep packages outside. My comments are not meant to be condemnation of the existing process, but just my own experiences in managing my own projects. Time permitting..I am happy to help with a trial test of some these methodologies.

P.S If you haven't tried Mercurial or Trac, you should take a look. Nothing better than hosting your projects on a portable LAMP/Hg/Trac stack running out of SLAX Porteus (soon) booted from a SLAX PXE server.

User avatar
brokenman
Site Admin
Site Admin
Posts: 5456
Joined: 27 Dec 2010, 03:50
Distribution: Porteus v3.2rcX all desktops
Location: Brazil
Contact:

Re: Porteus and SVN

Post#6 by brokenman » 23 Feb 2011, 03:10

Before 2 weeks ago i had never used subversion (on this side of the fence) so i have alot to learn about it. I think i understand where you are coming from. Porteus is made up of two elements, slackware packages and unique porteus files.
The svn would contain only the porteus stuff, and some scripts to build the individual modules which would hook into the slack repo (local or remote) and download/copy the packages, strip them and then add any externally created porteus files to the module. This would be ideal. At the moment i know fanthom uses some scripts to prep the file and it takes fanthom a short time to build the next release. Each release means these scripts need to be tweaked little but in the end they do the bulk of the work for him. For 32bit the last release release was a complete upgrade of KDE so i didn't have the luxury (and i believe fanthom did same with KDE4.6).

I concur that having scripts to automate the process of upgrading to the next release, and using these for the base of the svn would be ideal. After i get the hang of updating porteus to the next release (i have only taken on 32 bit for last 2 releases) i will endeavour to create the scripts. I have a feeling there will always be manual touchups required for each release but the svn will make tracking these down much easier. I also agree that it gives other people a chance to help with development, as sometimes life does get in the way ... like a big fucking earthquake down the road that fells the main CBD ... and giving others the option to add their expertise would be ideal. I'll discuss it with Fanthom upon his return and see what we can work out. Thanks ... appreciate your input. Correct me if i have strayed off your point in my understanding of your explanation.
How do i become super user?
Wear your underpants on the outside and put on a cape.

caliban
White ninja
White ninja
Posts: 9
Joined: 26 Jan 2011, 22:46
Location: USA

Re: Porteus and SVN

Post#7 by caliban » 24 Feb 2011, 02:04

I think you have got the general idea, only track the work you are responsible for and cut down all the other clutter. If you need any assistance, be glad to help.
My sympathies to you and your countrymen on the recent disaster.

Posted after 32 minutes 26 seconds:
As I was responding to Virtual Desktop discussion started by Hamza, it reminded me of something that I use everyday and that is automated builds. Most of these are driven by subversion post-commit scripts tied to Hudson. Frees up so many wasted hours...I use this along with my Slackbuild scripts to build/update my SLAX packages nightly. So for the most part, all I edit is the download path of the source file in the script and commit it to trunk. Nightly, it kicks off the automated build and then it lets me know if it was successful or not. I usually still have to do some manual cleanup here and there.

Post Reply