By sparcie

Something that was a little annoying using NetBSD is that performing what seems like routine maintenance to me can be quite difficult if you are using the pkgsrc build system. So I set about building some utility scripts to perform some of those actions for me. I did post some information about it in a few posts, but here is the more up to date version of all the files.

This first python script is really more of a utility script for the others providing some common procedures for manipulating the names of packages and executing commands. Pretty much all the following scripts use this one.

common.py

# this module contains common functions for all the package management tools.
from subprocess import call, Popen, PIPE

# global variables for this module.
# set this to 1 if you want some more details on the command line.
debug = 0

# functions and procedures.

def runAndRead(param):
    """Runs the program and reads its input returning a list of the lines read"""
    lines = []
    p = Popen(param, bufsize=1, stdout=PIPE, stdin=PIPE)
    o = p.stdout
    for i in o:
        if debug>0:
            print i.strip('\n')
        lines.append(i)
    return lines

def runAndSave(param, file):
    """ run a program and save the contents of it's output to a file """
    fh = open(file,'w')
    p = Popen(param,stdout=fh,stderr=fh)
    p.wait()
    fh.close()

def run(param):
    """wrapper or the call procedure so scripts importing this don't have to import subprocess"""
    return call(param)

def stripPkgName(name):
    """Strips the package version information from a package name. This is needed to help find package files, 
    installed them, and eventually remove them."""
    tokens = name.split('-')
    result = ''
    for i in tokens:
        if not i[:1].isdigit():
            result = result + "-" + i
    result = result.strip('-')
    return result

def versionString(name):
    """ extracts the version string out of the package name string. This will be helpful for determining if a 
    package should be installed to satisfy dependencies or requires updating"""
    tokens = name.split('-')
    result = ''
    s = False
    for i in tokens:
        if (i[:1].isdigit()) or s:
            s = True
            result = result + '-' + i
    result = result.strip('-')
    return result

def checkVersion(req, pkg):
    """ checks if the version of pkg satisfies the requires of the version of req"""
    if not stripPkgName(req)==stripPkgName(pkg):
        # they are not the same package! of course they won't work!
        return False;
    vreq = versionString(req)
    vpkg = versionString(pkg)
    # now we need to check each number within the version string.
    vreq = vreq.replace("nb",".").replace("-",".")
    vpkg = vpkg.replace("nb",".").replace("-",".")
    treq = vreq.split(".")
    tpkg = vpkg.split('.')
    l = min(len(treq),len(tpkg))
    for i in range(0,l):
        #do comparison
        nreq = int(treq[i],16)
        npkg = int(tpkg[i],16)
        if nreq>npkg:
            return False
        if npkg>nreq:
            return True;
    return False

This next script is useful for building an individual package. It uses the common.py file above, and basically runs the normal build commands for a package with some useful additions. Firstly it installs dependancies from binaries stored in the pkgsrc tree if they are available, then after the build it removes build dependencies that are only required for building the package. Fortunately pkg_add and pkg_delete are very forgiving on NetBSD, pkg_delete will only remove packages that do not have dependents unless you specify the force paramater.
build.py

#!/usr/pkg/bin/python2.6 -u

# this script is meant to make building and installing pkgsrc packages from source easier
# especially if you do it on a regular basis!

from common import *
import os

# global variables.

#path to the binary packages.
# you could write something to auto detect this, or just change it here.
pkgpath = "/usr/pkgsrc/packages/All"

# packages required for this one to be built
required = []

# packages required that are already installed (and thus should not be removed)
installed =[]

# main code

# first thing to do is get a list of what packages are required
input = runAndRead(["make","clean-depends"])

# using the output from make clean-depends we can determine which packages need to be installed.
for i in input:
    i = i.strip(' \n=>') #strip stuff we don't need
    li = i.split(' ') #split into tokens so we can get the package name.
    p = stripPkgName(li[-1])  # get the package name.
    if not p in required:
        required.append(p)  # add to the required list.

# get a list of what is already installed.
input = runAndRead(["pkg_info"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    installed.append(p)
    try:
        required.remove(p)
        print p,"is required and installed"
    except ValueError:
        print "installed package",p,"not found in required list"

# print a nice summary!
print "packages already installed:"
display = ""
for i in installed:
    display = display + i + " "
print display

display = ""
print "packages to be installed and removed"
for i in required:
    display = display + i + " "
print display

# install the dependencies still listed in required if we can

os.environ["PKG_PATH"] = pkgpath

# do the installation
for i in required:
    print "installing ",i
    run(["pkg_add","-Au",i])

del os.environ["PKG_PATH"]

# here is where we do the actual build!
run(["make","install"])
run(["make","clean","clean-depends"])

# remove required packages (if we can!)
for i  in required:
    print "removing ",i
    run(["pkg_delete","-R",i])

#done!

This next script is for cleaning up your binary package folder, it locates and removes the older duplicates of packages in the directory, then cleans up the symbolic links. The caveat with this script is that some of the package names can be the same across different pkgsrc directories. For instance apache has several directories, apache, apache2, apache22, etc which all produce packages of the form apache-x.xx.xx.tgz. This means if you have apache 1.3 and apache 2.4 compiled and built as packages, this script would remove the older of the two (based on date not version number). Most of the time this shouldn’t really be a problem, as not many people would do this, but it’s something to be aware of.
clean-packages.py

#!/usr/pkg/bin/python2.6 -u

# this script is meant to find and remove the binary packages that are old and superseeded by new ones.
# We need to also go through the category diretories and remove the symlinks for the corresponding
# packages.

from common import *
import os

#global variables
# you should point this to your the same location on your system if it is different.
pkg_path = "/usr/pkgsrc/packages/"

# functions and procedures

def processFiles(path):
    """Works out which files need to be removed and then preforms the removal"""
    print "processing files in",path
    #first we should check if this is a directory!
    if not os.path.isdir(path):
        print "  Not a directory!"
        return
    #now get the list of files and check for duplicates.
    rml = [] #the list of files to remove
    l = os.listdir(path)
    for x in l:
        sx = stripPkgName(x)
        for y in l:
            sy = stripPkgName(y)
            if sx==sy and not x==y:
                print "  Found duplicate package",x,"and",y
                # compare both the files and mark one for removal
                tx = os.stat(path+"/"+x).st_mtime
                ty = os.stat(path+"/"+y).st_mtime
                if tx>ty:
                    print "    ",y,"is older, marking for removal"
                    if not y in rml:
                        rml.append(y)
    print "Files to be removed:",rml
    for x in rml:
        print "  Removing",path+'/'+x
        os.remove(path+'/'+x)

def checkSymlinks(path):
    """ checks symlinks and see if they are valid."""
    print "Checking symlinks in",path
    #first we should check if this is a directory!
    if not os.path.isdir(path):
        print "  Not a directory!"
        return
    # Begin checking the symlinks
    l = os.listdir(path)
    for x in l:
        if os.path.islink(path+'/'+x):
            if not os.path.exists(os.readlink(path+'/'+x)):
                print "  Broken symlink ",x
                os.remove(path+'/'+x)    
# main code!

# get a list of the contents of the package store
list = os.listdir(pkg_path)
try:
    list.remove("CVS")
    list.remove(".cvsignore")
except:
    print "You don't have the CVS ignore and CVS directory present!"

processFiles(pkg_path+"All")    

for i in list:
    os.chdir(pkg_path+i)
    checkSymlinks(pkg_path+i)

The following script I created whilst developing most of the others. On my system disk space is at a premium as my disks are not large, so having the build dependencies doesn’t really make sense. This script makes a list of everything that was automatically installed with something else and attempts to remove them. Only packages which have nothing depending on them will be removed because pkg_delete will refuse to remove packages with dependants. The script will prompt you for confirmation before it begins the process.
rm-leaves.py

#!/usr/pkg/bin/python2.6 -u

# this script is meant to remove packages that were automatically installed as dependencies
# but are now orphans and no longer needed.

from common import *

#globals

#user installed packages
user_installed = []

#auto installed packages
auto_installed = []

#main code
# get a list of what is already installed.
input = runAndRead(["pkg_info"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    auto_installed.append(p)

# get a list of what is installed as a user package.
input = runAndRead(["pkg_info","-u"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    user_installed.append(p)
    try:
        auto_installed.remove(p)
    except ValueError:
        print "User package not listed in all packages list ??!!",p
        exit(1)

# display a summary of both sets of packages and allow the use to quit if they don't
# wish to remove something (or adjust other stuff)

print "user packages installed:"
display = ""
for i in user_installed:
    display = display + i + " "
print display

display = ""
print "Auto installed packages (will attempt to remove):"
for i in auto_installed:
    display = display + i + " "
print display

raw_input("Press Return to continue or Ctrl+C to stop")

# attempt the removal
for i in auto_installed:
    print "Trying to remove",i
    run(["pkg_delete","-R",i])

#done!

In the process of writing and testing these scripts, and ocasionally when building software, I ended up with a situation where I wanted to clean the entie pkgsrc tree, but using make clean takes ages! Make clean runs on only one processor so I decided to make a script that would run clean on more than one package at a time. It seems to be significantly faster, especially on my old sparc system.
fast-clean.py

#!/usr/pkg/bin/python2.6 -u

# this file is for cleaning the pkgsrc work files and dist files faster
# than the basic make clean does in the root.
# basically I am going to call make clean in each package directory 
# individually which is much faster, and I will be able to run more than 
# one thread at a time.

#in this way I hope that this script will also work on the FreeBSD ports
#tree - this is untested!

# I will use nice so that it doesn't become too much of a hog!

import os
import time
from subprocess import call, Popen

#globals

# the maximum number of cleaning threads
# adjust to best effect on your machine!
max_threads = 6

# the number of current cleaning threads
threads = 0

#processes!
ps = []

# the current working directory.
root_path = os.getcwd()

#functions

def run_thread(path):
    print "starting thread to clean in",path
    global threads
    os.chdir(path)
    threads = threads + 1    
    p = Popen(["nice","make","clean"])
    ps.append(p)

def check_and_wait():
    #initial check!
    global threads
    global max_threads
    for i in ps:
        r = i.poll()
        if not i.returncode==None:
            ps.remove(i)
            threads = threads - 1
    #return if we can
    if threads=max_threads:
        time.sleep(1)        
        for i in ps:
            r = i.poll()
            if not i.returncode==None:
                ps.remove(i)
                threads = threads - 1

def run_clean(path):
    list = os.listdir(path)
    for i in list:
        if os.path.isdir(path+"/"+i):
            check_and_wait()
            run_thread(path+"/"+i)            

#main code
#make us a nice boy!
os.nice(10)

#First get a list of all the directories to process.
root_files = os.listdir(root_path)
try:
    #remove the package directory as it should have nothing to clean.
    root_files.remove("package")
except:
    print "No package directory to worry about :-)"

for i in root_files:
    if os.path.isdir(root_path+"/"+i):
        run_clean(root_path+"/"+i)

while threads>0:
    check_and_wait()

If your pkgsrc tree is up to date you can use make show-downlevel to work out if a package needs to be upgraded or not. I wanted to use this feature to create an automated update utility, but running make show-downlevel on the root took a very long time indeed! So I decided it only really makes sense to check packages that are installed, and the following script essentially does just that. It produces the same output format as running make show-downlevel on the root of the tree. I did this so I could test it was behaving correctly. It checks auto installed packages first so that they may be updated first.
fast-downlevel.py

#!/usr/pkg/bin/python2.6 -u

# this script is meant to detect which software needs updating
# by running make show-downlevel. It is faster than running it
# in the root of pkgsrc because we are only going to check packages
# that are installed, and the auto installed dependencies first.

from common import *
import os

# global variables.

#pkgsrc root dir
# adjust this to match where you have your copy of pkgsrc.
pkgsrc_root = "/usr/pkgsrc/"

#user installed packages
user_installed = []

#auto installed packages
auto_installed = []

#functions

def checkDownlevel(pkg):
    # first things first check where this pkg is located.
    input = runAndRead(["pkg_info","-X",pkg])

    for i in input:
        if i.startswith("PKGPATH"):
            path = i.split("=")[-1].strip()

    os.chdir(pkgsrc_root+path)
    print "===> "+path
    run(["make","show-downlevel"])

#main code
# get current working directory so we can return to it.
cwd = os.getcwd()

# get a list of what is already installed.
input = runAndRead(["pkg_info"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    auto_installed.append(p)

# get a list of what is installed as a user package.
input = runAndRead(["pkg_info","-u"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    user_installed.append(p)
    try:
        auto_installed.remove(p)
    except ValueError:
        print "User package not listed in all packages list ??!!",p
        exit(1)

#check for downlevel packages.                                                
for i in auto_installed:
    checkDownlevel(i)

for i in user_installed:
    checkDownlevel(i)
#done!
os.chdir(cwd)

The final script is basically an automatic updater for packages installed using pkgsrc. It will optionally run the CVS update command as well to make sure your tree is up to date. I wanted this script to be as useful as possible and to have many options as it can take quite a while to run! It will attempt to build packages for anything that is reported as being out of date by the previous script (fast-downlevel.py). You’ll note it doesn’t try to preinstall any packages required as they may also need to be updated. It does however try to remove and build dependencies that aren’t needed to save disk space. It reports pacakages that didn’t build at the end. A package may not build if one of it’s dependants required updating, so I’ve found that in those cases it is usually a good idea to inspect the output and re-run this script if appropriate. You could probably modify it to do the job a bit more elegantly. This is because all the packages are built before any of them are installed, with a little tinkering this could be fixed.
auto-update.py

#!/usr/pkg/bin/python2.6 -u

# this script is for auto updating the entire package source distribution
# it will also optionally run a cvs update before hand if a command line
# argument is -C and -f downlevel_log.txt for reading a previously created
# output of make show-downlevel

from sys import argv
from common import *
import getopt
import os

# gloabal variables

# the root of the pkgsrc tree
# adjust this to match your installation.
pkgsrc_root = "/usr/pkgsrc"

#whether to use CVS to update the tree
run_cvs = False

# the downlevel text tile to use
downlevel_file = None

# the downlevel program to use
# adjust this for the path, and if you'd rather use the standard make show-downlevel
#downlevel_prg = ["make","show-downlevel"]
downlevel_prg = ["/home/andrew/python/package-tools/fast-downlevel.py"]

#the file which contains the list of excluded packages
exclude_file = None

# do we only want to list what should be updated.
list_only = False

#main code

try:
    optlist, args = getopt.getopt(argv[1:], "Cf:lp:X:")
except getopt.GetoptError, err:
    print str(err)
    print "Usage: autoupdate.py [-C] [-f downlevel.log] [-l] [-p /pkgsrc/root] [-X exclude-list.txt]"
    print "  -C : Runs cvs update before trying to update packages"
    print "  -f : Specifies a file containing the output of make show-downlevel"
    print "  -l : Only list packages to be updated, perform no actual changes"
    print "  -p : Specifies the root directory of pkgsrc"
    print "  -X : Specifies a file containing a list of packages to exclude from updating"
    exit(1)

for o, a in optlist:
    if o=="-C":
        run_cvs=True
    if o=="-l":
        list_only=True
    if o=="-f":
        print "processing",a
        downlevel_file = a
    if o=="-p":
        print "using",a,"as pkgsrc root"
        pkgsrc_root = a
    if o=="-X":
        print "Excluding packages listed in",a
        exclude_file = a

#load the excluded packages if required

excluded = []
if not exclude_file == None:
    fh = open(exclude_file,"r")
    display=""
    for i in fh:
        i = i.strip()
        excluded.append(i)
        display = display + " " + i
    print "Excluding:",display

# change to the pkgsrc root dir 
os.chdir(pkgsrc_root)

#run CVS if required
if run_cvs:
    print "Runnings cvs update"
    run(["cvs","update","-dP"])

#run show-downlevel if required.
if downlevel_file == None:
    print "Running make show-downlevel"
    runAndSave(downlevel_prg,"/tmp/tmp-downlevel.log")
    downlevel_file = "/tmp/tmp-downlevel.log"

#lists for storage of updates
upd_pkg = [] # the base package name being updated.
upd_path = [] # the path for packages to be updated

# now we need to load the log file and process it!
fh = open(downlevel_file,"r")
local_path = "."
for i in fh:
    tokens = i.strip().split(' ')
    if tokens[0] == "===>":
        local_path = tokens[1]
    elif i.find("pkgsrc version") > -1:
        pkg = tokens[0]
        installed = tokens[2]
        update = tokens[-1]
        if checkVersion(update,installed):
            print pkg," does not need a update: path:",local_path,"version:",versionString(installed)
        elif not pkg in excluded:
            print pkg," is at",versionString(installed),"can be updated to",versionString(update),"in package path",local_path
            upd_pkg.append(stripPkgName(installed))
            upd_path.append(local_path)
        else:
            print pkg,"was excluded from being updated: installed:",versionString(installed),"Updatable to:",versionString(update),"Path:",local_path
fh.close()    

if list_only:
    exit(0)

# get a list of installed packages (so we can weed out build dependencies later)
installed = []
input = runAndRead(["pkg_info"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    installed.append(p)

#build the new binary packages
#failed builds stored in a list for reporting
failed = []
for i in upd_path:
    print "Building in",i
    os.chdir(i)
    success = run(["make","package"])
    if success0:
        print "Build failed in",i
        failed.append(i)
    run(["make","clean","clean-depends"])    
    os.chdir(pkgsrc_root)

# now we need to install the new packages!
os.environ["PKG_PATH"] = pkgsrc_root+"/packages/All"

# do the installation
for i in upd_pkg:
    print "installing ",i
    run(["pkg_add","-U",i])

del os.environ["PKG_PATH"]

# get a list of installed packages (after the update so we can uninstall stuff that wasn't originally)
post_update = []
input = runAndRead(["pkg_info"])

for i in input:
    i = i.strip(' \n=>[]')
    li = i.split(' ')
    p = stripPkgName(li[0])
    post_update.append(p)

#if packages are installed that shouldn't be uninstall them!
for i in post_update:
    if not i in installed:
        print "Removing ",i
        run(["pkg_delete","-R",i])

#print a list of failed builds so the user can investigate.
print "Paths that failed to build on:"
for i in failed:
    print i

# at this point we should be done :-)

0 Responses to “Python: NetBSD Package tools”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Blogs I Follow

Enter your email address to follow this blog and receive notifications of new posts by email.


Mister G Kids

A daily comic about real stuff little kids say in school. By Matt Gajdoš

Random Battles: my life long level grind

completing every RPG, ever.

Gough's Tech Zone

Reversing the mindless enslavement of humans by technology.

Retrocosm's Vintage Computing, Tech & Scale RC Blog

Random mutterings on retro computing, old technology, some new, plus radio controlled scale modelling.

ancientelectronics

retro computing and gaming plus a little more

Retrocomputing with 90's SPARC

21st-Century computing, the hard way

lazygamereviews

MS-DOS game reviews, retro ramblings and more...

%d bloggers like this: