Search Results: "xnox"

30 March 2015

Dimitri John Ledkov: Boiling frog, or when did we loose it with /etc ?

$ sudo find /etc -type f wc -l

StatelessWhen was the last time you looked at /etc and thought - "I honestly know what every single file in here is". Or for example had a thought "Each file in here is configuration changes that I made". Or for example do you have confidence that your system will continue to function correctly if any of those files and directories are removed?

Traditionally most *NIX utilities are simple enough utilities, that do not require any configuration files what's so ever. However most have command line arguments, and environment variables to manipulate their behavior. Some of the more complex utilities have configuration files under /etc, sometimes with "layer" configuration from user's home directory (~/). Most of them are generally widely accepted. However, these do not segregate upstream / distribution / site administrator / local administrator / user configuration changes. Most update mechanisms created various ways to deal with merging and maintaining the correct state of those. For example both dpkg & RPM (%config) have elaborate strategies and policies and ways to deal with them. However, even today, still, they cause problems: prompting user for whitespace changes in config files, not preserving user changes, or failing to migrate them.

I can't find exact date, but it has now been something like 12 years since XDG Base directory specification was drafted. It came from Desktop Environment requirements, but one thing it achieves is segregation between upstream / distro / admin / user induced changes. When applications started to implement Base directory specification, I started to feel empowered. Upstream ships sensible configs in /usr, distribution integrators ship their overlay tweaks packaged in /usr, my site admin applies further requirements in /etc, and as I user I am free to improve or brake everything with configs in ~/. One of the best things from this setup - no upgrade prompts, and ease of reverting each layer of those configs (or at least auditing where the settings are coming from).

However, the uptake of XDG Base directory spec is slow / non-existing among the core components of any OS today. And at the same time /etc has grown to be a dumping ground for pretty much everything under the sun:
  • Symlink farms - E.g. /etc/rc*.d/*, /etc/systemd/system/*.wants/*, /etc/ssl/certs/*
  • Cache files - E.g. /etc/
  • Empty (and mandatory) directories
  • Empty (and mandatory) "configuration" files. - E.g. whitespace & comments only
Let's be brutally honest and say that none of the above belongs in /etc. /etc must be for end-user configuration only, made by the end user alone and nobody else (or e.g. an automation tool driven by the end-user, like puppet).

Documentation of available configuration options and syntax to specify those in the config files should be shipped... in the documentation. E.g. man pages, /usr/share/doc, and so on. And not as the system-wide "example" config files. Absence of the files in /etc must not be treated as fatal, but a norm, since most users use default settings (especially for the most obscure options). Lastly compiled-in defaults should be used where possible, or e.g. layer configuration from multiple locations (e.g. /usr, /etc, ~/ where appropriate).

Above observations are not novel, and shared by most developers and users in the wider open source ecosystem. There are many projects and concepts to deal with this problem by using automation (e.g. puppet, chef), by migrating to new layouts (e.g. implementing / supporting XDG base dir spec), using "app bundles" (e.g. mobile apps, docker), or fully enumerating/abstracting everything in a generic manner (e.g. NixOS). Whilst fixing the issue at hand, these solutions do increase the dependency on files in /etc to be available. In other words we grew a de-facto user-space API we must not break, because modifications to the well known files in /etc are expected to take effect by both users and many administrator tools.

Since August last year, I have joined Open Source Technology Center at Intel, and have been working on Clear Linux* Project for Intel Architecture. One of the goals we have set out is to achieve stateless operation - that is to have empty /etc by default, reserved for user modification alone, yet continuing to support all legacy / well-known configuration paths. The premise is that all software can be patched with auto-detection, built-in defaults or support for layered configuration to achieve this. I hope that this work would interest everyone and will be widely adopted.

Whilst the effort to convert everything is still on going, I want to discuss a few examples of any core system.

ShadowThe login(1) command, whilst having built-in default for every single option exits with status 1, if it cannot stat(2) login.defs(5) file.

The passwd(1) command will write out the salted/hashed password in the passwd(5) file, rather than in shadow(5), if it cannot stat the shadow(5) file. There is similar behavior with gshadow. I found it very ironic, that upstream project "shadow" does not use shadow(5) by default.

Similarly, stock files manipulated by passwd/useradd/groupadd utilities are not created, if missing.

Some settings in login.defs(5) are not applicable, when compiled with PAM support, yet present in the default shipped login.defs(5) file.

Patches to resolve above issues are undergoing review on the upstream mailing list.
DBusIn xml based configuration, includedir' elements are mandatory to exist on disk, that is empty directory must be present, if referenced. If these directories are non-existant, the configuration fails to load and the system or session bus are not started.

Similarly, upstream have general agreement with the stateless concept and patches to move all of dbus default configurations from /etc to /usr are being reviewed for inclusion at the bug tracker. I hope this change will make into the 1.10 stable release.

GNU Lib CToday, we live in a dual-stack IPv4 and IPv6 world, where even the localhost has multiple IP addresses. As a slightly ageist time reference, the first VCS I ever used was git. Thus when I read below, I get very confused:
$ cat /etc/host.conf
# The "order" line is only used by old versions of the C library.
order hosts,bind
multi on
Why not simply do this:
--- a/resolv/res_hconf.c
+++ b/resolv/res_hconf.c
@@ -309,6 +309,8 @@ do_init (void)
if (hconf_name == NULL)
hconf_name = _PATH_HOSTCONF;

+ arg_bool (ENV_MULTI, 1, "on", HCONF_FLAG_MULTI);
fp = fopen (hconf_name, "rce");
if (fp)

There are still many other packages that needed fixes similar to above. Stay tuned for further stateless observations about Glibc, OpenSSH, systemd and other well known packages.

In the mean time, you can try out images that implement above and more already. If you want to chat about it more, comment on G+, find myself on irc - xnox @ #clearlinux and join our mailing list to kick the conversation off, if you are interested in making the world more stateless.

I am a professional Linux Distribution developer, currently employed by Intel, however the postings on this site are my own and don't necessarily represent Intel's or any other past/present/future employer positions, strategies, or opinions.

* Other names and brands may be claimed as the property of others

2 May 2014

Dimitri John Ledkov: X4D Icons released

I am releasing some icons under MIT license. They will be hosted at and available for development on GitHub.

I had to create my own icons, as I couldn't find icons of similar nature under a free license. Hopefully others will find these useful as well.

The icons below are all available in PNG, GIF, SVG and EPS. To link to a specific version directly, add .png, .gif, -v.svg or -v.eps to the generic URI (or browse the icons repository to see all versions).

Document type Light Dark
HTML 2.0 Valid HTML2 Valid HTML20
HTML 3.2 Valid HTML32 Valid HTML32
HTML 4.0 Valid HTML40 Valid HTML40
HTML 4.01 Valid HTML4.01 Valid HTML4.01
XHTML 1.0 Valid XHTML10 Valid XHTML10
XHTML 1.1 Valid XHTML11 Valid XHTML11
XHTML Basic 1.0 Valid XHTML Basic 10 Valid XHTML Basic 10
XHTML-Print 1.0 Valid XHTML Print 10 Valid XHTML Print 10
CSS Valid CSS Valid CSS
CSS 1 Valid CSS1 Valid CSS1
CSS 2 Valid CSS2 Valid CSS2
MathML 2.0 Valid MathML 20 Valid MATHML20
SVG 1.0 Valid SVG10 Valid SVG10
SVG 1.1 Valid SVG11 Valid SVG11
SVG 1.2 Valid SVG12 Valid SVG12
SVG Tiny 1.1 Valid SVG Tiny11 Valid SVG Tiny11
SVG Tiny 1.2 Valid SVG Tiny12 Valid SVG Tiny12
XML 1.0 well-formed XML10 well-formed XML10
XML 1.1 well-formed XML11 well-formed XML11

2 September 2013

Dmitrijs Ledkovs: My Debian Activities: Multiarch & upstart on kFreeBSD hacking

Looks like there is a recent trend to publicize one's debian activities, so I thought I should join in =) Not so sure if MDA acronym would work for these types of posts...
MultiarchI have been working on multi-arching additional libraries. In particular I am working on multiarching boost libraries. At the moment boost1.53 multiarched libraries are uploaded in the experimental, and I am still working out the quirks with those. At the same time I have submitted a patch to multiarch libicu-dev, which has been accepted by the maintainer and is currently waiting in new queue. This brings us closer to multiarching all of boost libraries. But as it has been pointed out at the Multiarch BOF at Debconf'13, one doesn't have to wait for dependencies (be it libs or lib-devs) to be Multiarched before multi-arching your own library. If you do it right (put all or arch-specifi headers in multiarch location, and place libraries in multiarch location, split utilities into a separate package), you can go and multiarch your libraries ahead of time. Maybe it's time to default to debhelper 9 in dh-make, and default to multiarch locations for all libraries?

Boost also has libraries which depend on mpi, and at the moment that's not multiarched. I'm not sure if there are additional considerations there given the support for alternative mpi implementations in Debian.

Boost also depends on libpython. At the moment it's shipping &, with a symlink to I do wonder if it's a sensible approach, or should the boosty_python library be simply shipped in arch specific & python implementation specific paths e.g. /usr/lib/pythonX.Y/config-X.Y$(abi_tag)-$(DEB_HOST_MULTIARCH)/ that way we'll be able to support python debug interpreter in python_boost. I guess I should seek expert opinion on other multiarch wizards.

In addition to boost, I've now started looking at qmake and how it handles multiarch. At the moment, it doesn't handle it much. All paths are statically copied around and hard-coded in each module qmake provides. I have now been working at abusing qmake's $$system to call into dpkg-architecture, and actually choose native or cross compiler & adjust settings and paths to Qt libraries. There are a couple of bugs to fix, but I should be able to propose a "debian-multiarch" qmake spec soon. Such that in calls dpkg-architecture & properly adjust compilers & locations for the DEB_HOST. qmake by the way has its quirks. For example all variable assignments seemed to default to be "by value" & not "by reference" (i.e. upon assignment all variables on the right hand side get expanded and the literal string is stored, instead of storing $(prefix)/lib/$(arch), and evaluating that later.) Hence my current qmakespec is less elegant than I hoped.
upstart on kFreeBSDFreeBSD 9.2 and up have implemented the wait6 and with it waitid is also exposed. At the moment there are some differences between glibc and FreeBSD waitid usage. I am not well versed in glibc, FreeBSD and POSIX, but the bug at hand & POSIX reference lead me to believe that POSIX is slightly ambiguous about it.

Why is waitid support is interesting? Well, kFreeBSD/libc maintainer in Debian is working on exposing it with/after (e)glibc2.18 is in Debian, which will remove one more blocker from porting Upstart to kFreeBSD. To this extend, I have in the mean time setup FreeBSD & kFreeBSD virtual machines and got libnih to compile on both (sans waitid on kFreeBSD). One of the optional upstart's features is to use inotify to spot & reload configuration files, at the moment I haven't ported that functionality to use native kevent/kqueues (EVFILT_AIO) instead I have cheated and packaged for Debian the Google Summer of Code 2011 project libinotify-kevent which provides inotify compatible API, implemented with kevent. Which is kind of the reverse strategy of libkqueue and/or libevent. It would be interesting to see kernels and/or those libraries to unify/standartise on common APIs. Unfortunately, it still seems like using native API for each kernel is best to avoid dependencies and leverage all features and semantics. Nonetheless I did package it for now. To further progress with actually porting upstart, I'd like to get my hands at (e)glibc 2.18 compiled for kFreeBSD with waitid patches. I got pointers to the patches, but I haven't yet succeeded at compiling (e)glibc with those applied. So no upstart booting kFreeBSD VM yet, but rather work in progress. It's quite fun. It's been years since I have touched a FreeBSD boxes and it's first time me using kFreeBSD. The semantics, differences and benefits of each do shine and it's a promising platform.
Multiarch on kFreeBSD?
Given the excellent multiarch implementation on Debian and sophisticated Linux Emulation Layer on FreeBSD kernel, I was half expecting the following to work:
  • on kFreeBSD enable linux i386/amd64 repository
  • apt-get install hello:i386
  • $ hello
But unfortunately this confused kFreeBSD, since it didn't find ELF tags and linker didn't seem to load the right libraries any more =( Not sure if this issue has been raised before, but I think it would be awesome if FreeBSD Linux Emulation Layer worked transparently on Debian/kFreeBSD multiarch system.

25 March 2013

Dmitrijs Ledkovs: avahi + apt-cacher-ng + sbuild ?!

Laptop enters the WiFi network and decides that it wants to build some packages using sbuild.
At the same time, on this network there is apt-cacher-ng operating with most of packages cached.
On the other hand, there is a project squid-deb-proxy which provides yet another apt proxy, but with the added bonus of avahi discovery.

Can we throw all of this stuff together and make it work? Well let's try =)

On ubuntu:

$ sudo apt-get install apt-cacher-ng squid-deb-proxy-client

On debian:

Squid-deb-proxy-client is not packaged just fetch and install it. It's really just one python script & one conffile. Or I have published the python script and the config as part of this posts gist.

Next all of the avahi magic, really is just publishing a service file & letting the python-script from the squid-deb-proxy-client package find it, and adding apt-conf.d snippet which calls the above mentioned script and generate correct proxy line.

But we are running apt-cacher-ng! So on the server where apt-cacher-ng is running drop this:

<?xml version="1.0" standalone='no'?>
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<name replace-wildcards="yes">apt-cacher-ng proxy on %h</name>
<service protocol="ipv4">

Now on any host that you want automatic apt-proxy discovery, simply install squid-deb-proxy-client & your apt-cacher-ng will be auto-discovered.

What about sbuild? Well, you don't really want to install avahi-daemon and start it inside the chroot, so we can cheat with creating an executable hook:

set -e

. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
. "$SETUP_DATA_DIR/common-config"


if [ $STAGE = "setup-start" ] [ $STAGE = "setup-recover" ]; then
if type apt-avahi-discover 1>/dev/null; then
if [ -n "$APT_PROXY" ]; then
info "Setting apt-proxy to avahi proxy at $ APT_PROXY "
printf 'Acquire::http Proxy "%s"; ;\n' $ APT_PROXY > "$ CHROOT_PATH /etc/apt/apt.conf.d/30schrootautoproxy"

Tadah. Now I'm using my apt-cacher-ng whenever I'm on my home WiFi =)

Please advertise your apt-cache proxies!

Now to integrate this properly:
  • apt-cacher-ng should auto-create avahi service file in it's init script
  • apt-get or some other package should ship apt-avahi-discover script (there is also a C implementation available)
  • apt-get or some other package should ship the apt.conf.d snippet
  • sbuild / pbuilder / cowbuilder / mk-sbuild / [p cow]builder-dist should have hooks to retrieve hosts' apt proxy settings and use them in the chroots
For full sources see lp:squid-deb-proxy and my gist with all other config snippets.

8 June 2012

Dmitrijs Ledkovs: Hello Debian, Planet and Debconf12

I am going to Debconf12
I'm going to Debconf12
Hello Debian Planet!

I have recently became a Debian Developer. One of the first things I did, was fix some RC bugs in my packages due to an ABI break from upstream in a point release. Then I fixed some bugs in offlineimap... Little did I know, that I broke it for many Debian developers including my ex boss and the Debian Project Leader all at the same time.

Speaking of which, any help will be appreciated in hunting down a memory leak in offlineimap.

I am going to Debconf12, thanks to my employer. I am interested in Python/dh_python[2-3], dh, 3.0 (quilt) with a DVCS, lvm2, mdadm, btrfs-tools, autofs, e2fsprogs, dm-crypt, debian-installer, init-systems & friends, OpenERP, offlineimap... if you are as well, find me and talk to me =)))