Monday, October 10, 2011

Pedigree's 'pup' package manager

Recently there's been a lot of activity in the project repositories related to the 'pup' package manager.

I've been working at getting our binaries and other ported application-related files out of each repository and move towards only having the specific binaries we need for building Pedigree from scratch, and the remainder being source code and tools.

So now a checkout of Pedigree and an Easy Build run will leave you with quite a bare build of Pedigree - you'll get bash, coreutils, and not too much else. This allows you to choose what goes into your build of Pedigree; if you just want Python and Lynx for web browsing, that's totally alright with us.

'pup' is written in Python and will be available to run in Pedigree as well as on the build machine. At the moment however our Python build is a little broken as we are working on bringing proper shared library support to our userspace ports.

In my opinion there's no reason for everybody to download over 250 MB just to get a simple checkout of Pedigree. That's what 'pup' integration is all about.

Minecraft: Fix for Error (4): null

Been having this issue on and off for a while now, but finally nailed the source of the problem today. I run Minecraft behind a Squid 2.7 proxy, which appears to have trouble handling ETags properly.

In particular, it seems to have to do with the If-Match-None header.

Upgraded to Squid 3.0 and I'm able to update Minecraft without using a browser - awesome! :)

Wednesday, June 01, 2011

And now for something COMPLETELY different!

I know this is technically a Pedigree devblog, but I also do sysadmin work and this might be handy for someone else out there.

In a Windows network environment, folder redirection comes in handy for things such as server-side My Documents and such. We use folder redirection for the Desktop and Start Menu as well, to provide a set of icons to each user that we can manage centrally.

However, it turns out Folder Redirection can't always redirect to a mapped network drive (ie, M:). The order of GPO processing means the drive mapping takes place later on, after the folder redirection takes place. So if you've mapped a bunch of network drives for the user and want to redirect to one, without using a UNC path, it won't work. This doesn't work for us as we map some drives to different locations based on the computer's organisational unit.

So the fix comes from Group Policy Preferences: all that "Folder Redirection" does is set a few registry keys. Group Policy Preferences lets us set those registry keys manually, with the added bonus of the extremely powerful granular targeting (Item-Level targeting). The targeting lets us apply different registry keys to Admin users instead of Teachers, for example.

The relevant keys for the Desktop and Start Menu are:

All Users - common across every user logged into the machine
HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders

Keys: Common Programs ("All Programs"), Common Start Menu (Icons outside of "All Programs"), Common Startup ("Startup" folder), Common Desktop (the desktop).

Individual Users
HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders

Keys: Desktop, Start Menu, Startup, Programs - all the same purpose as above.

So, by setting a combination of these keys, we were able to get the desktop and start menu completely redirected to a network share that's unique per organisational unit. This means modifying the icons for a room involves a simple copy & paste on the file share rather than rebooting a computer for the startup script to take effect.

Sunday, May 15, 2011

Live Streaming some code!

I've set up a Justin.tv channel to which I can stream live coding and debugging of issues in Pedigree and other projects. I gave it a go this afternoon with about an hour's worth of playing around with Pedigree's memset, and I hope to regularly stream new stuff - there's certainly enough to do, after all!

Thursday, February 03, 2011

Package Management & Workflow

For those who don't know, one of my "mini-projects" (of which there are many!) on the Pedigree project is the 'pup' package manager. The idea behind pup is to create a Python-oriented package manager that works great on Pedigree but can also be used for any other operating system that happens to have Python ported. Essentially all of the backend is just standard HTTP GET requests, the packages are just .tar.gz files, and building a package is dead simple (assuming the environment is okay).

The problem with 'pup' is that it current works out of a git repository which holds every package that will be in a master repository. This is great for editing the build scripts and versioning the pup source code, but when it comes to any form of management it's a nightmare. The solution for this is to write a web-based user interface for managing a package repository.

That's where it gets exciting.

Actually managing packages in their built state is very, very easy. You simply modify the SQLite database containing package information where necessary, and move the package files around. The difficulty comes from building and creating packages. The easiest way to convince users to port more software to Pedigree is to make it extremely easy to - this is the goal of pup and the build scripts in our Pedigree-Apps repository. So any web interface for management has to handle both managing existing packages (update/remove/rename/edit attributes) as well as handle bringing new packages into the system.

The workflow I came up with in my head for this is as follows:

  1. User creates a build.sh and related scripts to build the relevant port. This is hopefully tested by the user and works on that system. Creating these scripts is already documented.
  2. These scripts are uploaded to the web interface
  3. The permissions of the user are checked; depending on these the package will be automatically built on the server straight away, or added to a queue for moderation (we'll assume it was permitted to be built here).
  4. As the build continues, its status is made available to the user.
  5. Once the build completes, the user is notified of the status of the build. Most likely an IRC bot is also present to notify developers/interested parties of the build status.
  6. The package is added to the system and mirrored from the master repository to various other mirrors, ready for users to download the update/new package via pup on their Pedigree installation.
Here's the problem - as far as I can tell nothing like this exists.

So I'm looking at writing this myself, most likely in Python. It should be a fun project for when all the other projects are hitting dead ends, and has the added bonus of reducing my workload on Pedigree when done!

As an aside, notable changes since the last blog post:
  • We released a feature-test release - Foster Milestone #1.
  • SSE in the kernel, oh my! memset takes this very well, memcpy not so much due to floating point state issues. These are being resolved at the moment.
  • A variety of changes across the POSIX subsystem and within the network stack for improved stability.
Also, We're looking for anyone who's got a few year's worth of OS development under their belt to come on board and work on the project - if you feel like something new and a C++ kernel doesn't worry you, find us in #pedigree on Freenode IRC.