16
Jul
12

Python Scripts for building in pkgsrc packages in NetBSD

I’ve been a little busy of late, so sorry this post is coming in rather late.

As you may have already read, I have been learning python for work. So I decided to use it to try to solve some problems I had been having on my SparcStation.

There aren’t any binary packages for my old sparc available so I often take it upon myself to build packages from source using the pkgsrc system. There are a few problems in doing this: Firstly the old machine doesn’t have a lot of disk space, so I do not wish to keep the build dependencies installed all the time (bison, m4, digest etc…). Secondly the machine is quite slow, so when building a package I do not wish to rebuild ones that are already built. Normally I would have to perform the installation and removal manually as the pkgsrc system doesn’t seem to detect previously built/downloaded packages. So I decided to make a python script that would automate the process, so I could ask the machine to build and install something, and not have to interact with it unless there was a problem or it finished.

In starting the script I realized I could also automate a few other pkgsrc functions such as updating all the installed packages. Because there was a lot of commonality between what I was writing I made a common module that all the scripts could use to perform some basic functions that I would need on a regular basis.

Here is the common.py code…

# this module contains common functions for all the package management tools.
from subprocess import call, Popen, PIPE

# global variables for this module.

debug = 0

# functions and procedures.

def runAndRead(param):
    """Runs the program and reads its input returning a list of the lines read"""
    lines = []
    p = Popen(param, bufsize=1, stdout=PIPE, stdin=PIPE)
    o = p.stdout
    for i in o:
        if debug>0:
            print i.strip('\n')
        lines.append(i)
    return lines

def runAndSave(param, file):
    """ run a program and save the contents of it's output to a file """
    fh = open(file,'w')
    p = Popen(param,stdout=fh,stderr=fh)
    p.wait()
    fh.close()

def run(param):
    """wrapper or the call procedure so scripts importing this don't have to import subprocess"""
    call(param)

def stripPkgName(name):
    """Strips the package version information from a package name. This is needed to help find package files, 
    installed them, and eventually remove them."""
    tokens = name.split('-')
    result = ''
    for i in tokens:
        if not i[:1].isdigit():
            result = result + "-" + i
    result = result.strip('-')
    return result

def versionString(name):
    """ extracts the version string out of the package name string. This will be helpful for determining if a 
    package should be installed to satisfy dependencies or requires updating"""
    tokens = name.split('-')
    result = ''
    s = False
    for i in tokens:
        if (i[:1].isdigit()) or s:
            s = True
            result = result + '-' + i
    result = result.strip('-')
    return result

def checkVersion(req, pkg):
    """ checks if the version of pkg satisfies the requires of the version of req"""
    if not stripPkgName(req)==stripPkgName(pkg):
        # they are not the same package! of course they won't work!
        return False;
    vreq = versionString(req)
    vpkg = versionString(pkg)
    # now we need to check each number within the version string.
    vreq = vreq.replace("nb",".").replace("-",".")
    vpkg = vpkg.replace("nb",".").replace("-",".")
    treq = vreq.split(".")
    tpkg = vpkg.split('.')
    l = min(len(treq),len(tpkg))
    for i in range(0,l):
        #do comparison
        nreq = int(treq[i],16)
        npkg = int(tpkg[i],16)
        if nreq>npkg:
            return False
        if npkg>nreq:
            return True;
    return False

Of course this isn’t very helpful without one of the scripts that makes use of it! Following is a simple script that builds the package in the current directory. It installs the binary packages that are required first for building, and then removes what it can at the end.

#!/usr/pkg/bin/python2.6 -u

# this script is meant to make building and installing pkgsrc packages from source easier
# especially if you do it on a regular basis!

from common import *

# global variables.

#path to the binary packages.
pkgpath = "/usr/pkgsrc/packages/All"

# packages required for this one to be built
required = []

# packages required that are already installed (and thus should not be removed)
installed =[]

# main code

# first thing to do is get a list of what packages are required
input = runAndRead(["make","clean-depends"])

# using the output from make clean-depends we can determine which packages need to be installed.
for i in input:
    i = i.strip(' \n=>') #strip stuff we don't need
    li = i.split(' ') #split into tokens so we can get the package name.
    p = stripPkgName(li[-1])  # get the package name.
    if not p in required:
        required.append(p)  # add to the required list.

# get a list of what is already installed.
input = runAndRead(["pkg_info"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    installed.append(p)
    try:
        required.remove(p)
        print p,"is required and installed"
    except ValueError:
        print "installed package",p,"not found in required list"

# print a nice summary!
print "packages already installed:"
display = ""
for i in installed:
    display = display + i + " "
print display

display = ""
print "packages to be installed and removed"
for i in required:
    display = display + i + " "
print display

# install the dependencies still listed in required if we can

environ["PKG_PATH"] = pkgpath

# do the installation
for i in required:
    print "installing ",i
    run(["pkg_add","-Au",i])

del environ["PKG_PATH"]

# here is where we do the actual build!
run(["make","install"])
run(["make","clean","clean-depends"])

# remove required packages (if we can!)
for i  in required:
    print "removing ",i
    run(["pkg_delete","-R",i])

#done!

I’ve made a couple of other useful scripts as well that I will post over the next couple of days (or whenever I can find the time).

Advertisements

0 Responses to “Python Scripts for building in pkgsrc packages in NetBSD”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Blogs I Follow

Enter your email address to follow this blog and receive notifications of new posts by email.


Mister G Kids

A daily comic about real stuff little kids say in school. By Matt Gajdoš

Random Battles: my life long level grind

completing every RPG, ever.

Gough's Tech Zone

Reversing the mindless enslavement of humans by technology.

Retrocosm's Vintage Computing, Tech & Scale RC Blog

Random mutterings on retro computing, old technology, some new, plus radio controlled scale modelling.

ancientelectronics

retro computing and gaming plus a little more

Retrocomputing with 90's SPARC

21st-Century computing, the hard way

lazygamereviews

MS-DOS game reviews, retro ramblings and more...

%d bloggers like this: