Sunday, May 09, 2010

New Core Features

It's been a long time since I posted anything here. Moving to a new city has slowed my development down significantly, but things are getting back to normal again.

Recently there has been a fair bit of activity, and I'll summarise what's been going on in this post. There's a couple new features, some changes to Pedigree itself, and some bug fixes. In-depth discussion of each, where necessary, follows the list.

  • ZombieQueue (commit d5314bf7)
  • Using > 4 GB of RAM now works, with the x86_64 port (commit bb6b954b)
  • MemoryPool (commit 64454975)
  • UdpLogger (versatile log callbacks) (commit ff5fddba)
ZombieQueue
ZombieQueue solves a problem that has been lying around for quite some time. Things such as signals require that a Process object is deleted as part of it being killed, so it is no longer schedulable. However, this was dealt with originally by a "delete this;" line. Any form of "delete this" can be dangerous when the next line is not a return statement. Even when the next line is a return statement, a reschedule out of the current context and into another, which allocates memory, may overwrite the existing object... making "this" invalid. That said, it did suffice for quite some time.

The idea of ZombieQueue is that you queue these deletions, allowing the thread to be rescheduled (or the call that needs deletion to return) and the object to be deleted at a later time, when it's safe to do so. Eventually ZombieQueue might also have a "delay" parameter, which ensures that it never deletes an object until after a specific time period has passed. This could easily be used for things such as TIME_WAIT handling in the TCP stack.

ZombieQueue has already been implemented into Process, Pipe, and Socket, all of which needed to delete themselves somehow. Now they call into ZombieQueue, which is able to safely delete the object.

x86_64 port using > 4 GB of RAM
A TODO all through the physical memory manager for quite some time has been to handle regions above 4 GB in the x86_64 port. Until this change was made, no ranges of memory above 4 GB were actually accessible. In fact, using the x86_64 port on a system with over 4 GB of memory actually caused anything from a panic to a triple fault and reboot.

This has been rectified now. I'm still not sure it'll handle a full 64-bit physical address space yet, but it will handle a massive amount of memory nonetheless.

MemoryPool
This one is something I'm surprised hasn't been implemented yet. The idea of MemoryPool is to take buffer allocation out of the kernel heap, where it consumes far too much memory - and sometimes even consumed the full size of the heap.

MemoryPool takes a block of memory, outside of the heap, and splits it up into buffers of a specified size. These buffers are distributed to modules and applications where necessary, and returned to the pool when they are no longer needed. Buffers in, for example, the network stack, are around 2048 bytes (rounded up to the next power of 2 from 1500). Rather than have every incoming packet cause an allocation on the kernel heap of, on average, 800 bytes, this moves the processing into a pool of memory dedicated to the job.

A key feature to MemoryPool is its blocking nature. If an allocation cannot be satisfied by the current pool of memory (ie, all buffers are allocated out), it will block until a buffer becomes available. Compared to the heap, this means a constant memory usage is enforced. Effectively, the pool provides a strict contract on its memory consumption: a pool will never resize to fit a new allocation.

This does bring up a minor problem: some code can't block - an interrupt handler for example. For this situation, a non-blocking allocation method is available, which merely returns NULL if no buffers are available.

UdpLogger
It has long been possible to install custom Log callbacks which take log entries and write them to a destination. Callbacks that already exist include the Serial logger, and a callback is used for the loading screen log dump. For testing on real machines, however, neither of these suffice. None of my test machines have a serial port, for example. They are all networked via Ethernet, however.

UdpLogger is able to be installed after the route table is installed, so it's only really useful for debugging if you can actually boot to a shell on the test machine. It dumps all log entries to a given IP address and port. A test machine can then dump all its debugging output to a development machine running netcat.

Here it is, in action:


Log::installCallback has now also been updated to dump all the old log entries to the new callback, so every logging destination should have the same data once they are all installed.

These are just a couple of exciting new features and fixes that have happened in the past couple of weeks. At this stage we have not planned a second release - we're still working through the bugs from Foster and finally getting around to fixing all those TODOs scattered in the code.

At the moment git master is being kept "stable" - that is, it builds and runs to the bash shell (apart from x86_64) - so feel free to grab the latest code and have a play!