• Copy files, and restart services that depend upon them.
• Make minor edits to files. (Appending lines not present, replacing lines you no longer wanted, etc)
• Installing / Removing packages.
• More ..

• Download polices (i.e. rules) to apply from a central machine using nothing more complex than HTTP.
• Entirely client-driven, and scheduled via cron.

• Configuration is just data in YAML format, which makes it really easy to whack up stuff and also to generate it using other means;
• There's a lot of plugability: runners, renderers, states, modules, returners, outputters, all of which can be replaced with custom implementations;
• It builds on remote execution. In fact, in the docs, they talk first about remote execution and then about configuration management. Not only does this makes a lot of sense, it also gives the admin the additional benefit of having quick, remote access to all hosts in the infrastructure. I like that as it means I don't also have to deploy a parallel-execution solution.
• Salt supports both, a push and a pull model for configuration management. In fact, the push model is part of its remote execution base and it's implemented through persistent connections from the clients to the master. This neatly solves all problems relating to firewalling and NAT, and it also means that the master always has a pretty good understanding of who's around. It feels good to me. And there is no NIH going on with respect to pulling: there is no daemon, you have to use cron. Good.

• The default templating engine is Jinja. While that may be an excellent choice for applications in inherently insecure environments, such as the Web, they are quite painful in the context of Salt: no multi-line logic, whitespace control is a nightmare, and a completely unfamiliar set of conditionals and filters, rather than access to e.g. Python. This is only the default, and mako is an alternative, but it seems like a poor choice and that does not instill confidence.
• I don't think that the method of targeting hosts that Salt uses is scalable. I don't want to intersperse my configuration with if-then-else statements referencing hostnames (what would that mean if a hostname changed or a new host was added?). Instead, I want the configuration to be fully parametrised and assemble the parameters from a node database, ideally a hierarchical one. I have a solution for that which works with Salt, called reclass.
• I don't think much of "pillars", which are Salt way to provide random data (parameters) to nodes, again because data needs to be targeted at hosts, rather than defined for a host. This is also solved with reclass.
• Logging and error reporting in Salt are abysmal, inconsistent, and useless. This is offset a little bit by the ability to use pdb, but it's not ideal.
• Exceptions are not properly handled, the file descriptors aren't closed when daemonising, children are not reaped as they should, and the exit code will almost always be zero, even on failure.
• State requirements are not enforced where they are required, but only come into play when a higher order function is called.
• The code reinvents module loading and does that in very complex ways, while putting tight limitations on what's possible. State functions and modules cannot be namespaced, for instance, nor can they easily make use of each other. There is also no way to quickly define macros without resorting to the templating language (and thus a completely different syntax and paradigm), and the concept of having modular, self-contained states (e.g. one for sudo, one for SSH, etc.) with all their dependencies in one hierarchy is not really part of the design. Also, it's not trivial for a state definition to export an interface for other states to use.
• While we're on the topic of code and design, there's a lot of duplicate stuff in need of refactoring (e.g. saltutil._sync, and it seems that this even causes a lot of crypto work to be done over and over again. It also doesn't help that salt-call, which is used to execute commands on the clients (e.g. pull configuration), works completely independently of the daemon process running on each client (although the same code base is being used).
• And why aren't state and execution modules implemented as classes? Instead, the module is itself treated as an entity, and a home-cooked "virtual" system is used to determine which platforms support which modules. Object-oriented programming principles gives you all of that, and more.
• Fortunately, the Salt authors didn't cave in and write the entire communication layer themselves. Instead, they employed ZeroMQ, but it's still quite common for a client to become unreachable (permanently), e.g. due to an IP address change, or networking problems. What's worse is that the master does not keep track of who's listening.
• While we're on the subject of standing on the shoulders of giants, Python's msgpack apparently cannot handle Unicode. Yay.

• I'd want to keep it the way of Salt, i.e. the clients maintain persistent connections to the master, and the master regularly pings the clients, for housekeeping. And the clients know to expect such pings at regular intervals, so can reset themselves in case they don't hear from the master. Or scream.
• Of course, authentication and encryption need to be part of this. Ideally, key and roster management are already available in the tool that's being reused. I don't want to have to have a new PKI for this
• The protocol would probably be something like XMLRPC, with an extensible list of modules on the client to do the work. Data would be standardised to JSON-format.
• Asynchronous execution would be a plus. I am even tempted to say that it should be all built asynchronously, even though I don't really know a use case for this.
• The clients should be able to feed back information to the master, where data can be accumulated, allowing for cross-node configuration (cf. http://comments.gmane.org/gmane.comp.sysutils.salt.user/2098). This could be implemented using master-side polls to keep the protocol easier.

• [1] http://tinyurl.com/aens6vr
• [2] http://tinyurl.com/3s8exu4

1. Package fiefdoms waste people's time quite frequently: MIA processing, duplicated effort in DELAYED NMUs, conspiracy and infighting, trying to get an absentee maintainer to respond, the ridiculous things that happen when a maintainer is still around but not doing his job for 3 consecutive years, and so on. Having all packages maintained by "Debian Developers" is a great idea.
2. The current hierarchy might work better than some lesser evils, but that doesn't mean it's good at all. We should strive toward ZERO hierarchy, not some magical fiction where people imagine themselves wiser than everyone else because their drinking buddies said so.

1. Individual packages in Debian are not managed by consensus, but rather fall under the authority of the package maintainer, who is either a single person or a small team. (Consensus works in small teams; it's great if the number of people involved is five or less.) This drastically limits the number of decisions that are subject to a large-scale consensus process. Nearly all decisions in Debian are made directly by the person doing the work, and only in exceptional circumstances can some larger process be invoked. This, on the surface, sounds like it could be similar to Wikipedia, but note that appeals to a larger process are much rarer in Debian than they are in Wikipedia. There is a strong bias in favor of letting the person working on the package do their job however they want to do it. This is more social than structural, but it's a core value (and we should be very careful about changing it). As a result, Debian operates less by mass consensus and more as a federation of semi-autonomous fiefdoms. This sounds worse: it sounds authoritarian and non-democratic. But it's significantly more functional, and it doesn't waste people's time.
2. On appeal, and for broader questions that necessarily cut across multiple packages, there are several people in Debian who are perceived as having clear authority to make timely and relatively final decisions, who are almost never overruled, and who can therefore effectively end arguments. These include core teams like ftp-master or the release team, formal appeal bodies like the Technical Committee, and the Debian Project Leader. Not all of this ability is written in black and white in the governance process, but that's not the important part; the important part is that it works in practice to provide timely, consistent, unambiguous, and final decisions most of the time. (Some parts work better than others, and we do struggle with timely, but I've seen much, much worse.)
3. There is a well-established ultimate appeal (a General Resolution), the ultimate appeal process is very specific and concrete, there is little or no ambiguity or human interpretation involved in analyzing the results, it is time-limited, and it's almost universally respected. Everyone abides by the result of a GR if matters get to that point, and while it's theoretically possible to propose another GR immediately to overturn the first one, this doesn't happen in practice. There's also substantial pushback against invoking the GR process for anything that isn't really important.

• [1] http://etbe.coker.com.au/2013/01/28/selinux-status-lca2013/
• [2] http://danwalsh.livejournal.com/31146.html

1. Chunked responses
   App Engine doesn't have a way of doing chunked responses.
   1.5. Processing after sending the response My annoyance here is that I can't send an early response and wait for some resource. Say I want to do 5 url fetches that I cache for a short period of time. If I want to reply in 250ms, I want to use any of the results I've gotten and reply to the user. Then once the rest of the fetches come in, I want to put them in the cache, so they will come back quickly the next time.
2. Counters
   App Engine has crappy support for counters. You can shard your counters into the datastore: https://developers.google.com/appengine/articles/sharding_counters#counter_python or you can use taskqueues: http://blog.appenginefan.com/2009/10/non-sharded-counters-part-2-using-task.html . Now, how do you graph these counters over time, well, that is left as an exercise for the reader. Google has special datastores for stores and retrieving counters.
3. Backends
   Backends in appengine are versioned separately from frontends. This sounds like a good idea, but to run the devserver with backends, the backends need to be in the same source tree. Thus backends share the same url space. Backends also share the same app.yaml, so again the url space is shared. Thus, if you make a change that you want to coordinate between the frontend and the backend, you need to change the application version in app.yaml, then rename all the backends in backends.yaml. If backends are supposed to be in the same source tree as frontends, then they should be prefixed with the application version. If backends should be versioned separately from frontends, then they shouldn't share url handlers from app.yaml.
   3.5 Uploading backends is slow
   Each backend is uploaded through appcfg.py separately. So if you are crazy and have one backend per backend level, then you need to upload 4 copies of the source code each time the backends get updated.
4. Logging
   I want structured logs. Instead what you have to do is use the python logging module to write out human readable logs, then parse them out into the structure you want.
   http://code.google.com/p/google-app-engine-samples/source/browse/trunk/logparser/logparser.py
   

Facter.add("primary_interface") do
  setcode do
    ifs = Dir.new("/sys/class/net").each.to_a - \
      Dir.new("/sys/devices/virtual/net").each.to_a
      # filter out ones with no link
      ifs.reject! do  i 
        begin
          File.read("/sys/class/net/# i /carrier") != "1\n"
        rescue Errno::EINVAL
          true
        end
      end
    ifs[0]
  end
end

• Copyright assignment: Some high-profile GNU maintainers have recently criticized the FSF's copyright assignment policy and system. What are these criticisms, what does the FSF intend to do about them, and what's the point of its assignment process to begin with?
• GPL adoption: Last year here, in "Is copyleft being framed?", I put numbers supposedly showing declining GPL adoption in perspective, showing problems with the data, questioning the conclusions drawn from the data, and presenting different data leading to the opposite conclusions. We'll look at the questions that were raised since then about my data, and at some new data that's been made available, and draw new conclusions.
• App Stores: When Apple's App Store launched, the FSF concluded that its terms were incompatible with the GPL -- and with any kind of strong copyleft. Since then, we have several new App Stores; most notably from Google and Microsoft. Are the Apple terms still incompatible with the GPL? Are the other stores any better? Are these stores undermining GPL adoption, and should copyleft relax its standards in order to get free software to this audience, or should it stand its ground?

[~] # grep '^ *\(ppp\ eth\ wlan\ ath\ ra\ ipsec\ tap\ br-\)\([^:]\)\ 1,\ :' /proc/net/dev   cut -f1 -d:   sed 's/ //g
> s/\-/_/g'
eth1
eth0

[~] # ethtool eth0  grep Speed:
    Speed: 1000Mb/s
[~] # ethtool eth0  grep "Link detected:"
    Link detected: yes
[~] # ethtool eth1  grep Speed:
    Speed: Unknown! (65535)
[~] # ethtool eth1  grep "Link detected:"
    Link detected: no

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

--- /opt/sbin/munin-node.orig	2013-01-27 15:13:51.869007214 +0100
+++ /opt/sbin/munin-node	2013-01-27 16:11:20.536006950 +0100
@@ -133,7 +133,7 @@
   if [ -n "$(which ethtool)" ]; then
 	if [ -x "$(which ethtool)" ]; then
   		if ethtool $1   grep -q Speed; then
-    			MAX=$(($(ethtool $1   grep Speed   sed -e 's/[[:space:]]\ 1,\ / /g' -e 's/^ //' -e 's/M.*//'   cut -d\  -f2) * 1000000))
+    			MAX=$(($(ethtool $1   grep Speed   sed -e 's/[[:space:]]\ 1,\ / /g' -e 's/^ //' -e 's/M.*//'   sed -e 's/Unknown\!/0/'   cut -d\  -f2) * 1000000))
     			echo "up.max $MAX"
     			echo "down.max $MAX"
 		fi
@@ -535,19 +535,31 @@
     for INTER in $(grep '^ *\(ppp\ eth\ wlan\ ath\ ra\ ipsec\ tap\ br-\)\([^:]\)\ 1,\ :' /proc/net/dev   cut -f1 -d:   sed 's/ //g
 s/\-/_/g');
     do
-      INTERRES=$(echo $INTER   sed 's/\./VLAN/')
-      RES="$RES if_$INTERRES"
-      eval "fetch_if_$ INTERRES ()   fetch_if $INTER $@;  ;"
-      eval "config_if_$ INTERRES ()   config_if $INTER $@;  ;"
+      if [ -n "$(which ethtool)" ]; then
+        if [ -x "$(which ethtool)" ]; then
+          if [ -n "$(ethtool $INTER   grep 'Link detected: yes')" ]; then
+            INTERRES=$(echo $INTER   sed 's/\./VLAN/')
+            RES="$RES if_$INTERRES"
+            eval "fetch_if_$ INTERRES ()   fetch_if $INTER $@;  ;"
+            eval "config_if_$ INTERRES ()   config_if $INTER $@;  ;"
+          fi
+        fi
+      fi
     done
   elif [ "$PLUG" = "if_err_" ]; then
     for INTER in $(grep '^ *\(ppp\ eth\ wlan\ ath\ ra\ ipsec\ tap\ br-\)\([^:]\)\ 1,\ :' /proc/net/dev   cut -f1 -d:   sed 's/ //g
 s/\-/_/g');
     do
-      INTERRES=$(echo $INTER   sed 's/\./VLAN/')
-      RES="$RES if_err_$INTERRES"
-      eval "fetch_if_err_$ INTERRES ()   fetch_if_err $INTER $@;  ;"
-      eval "config_if_err_$ INTERRES ()   config_if_err $INTER $@;  ;"
+      if [ -n "$(which ethtool)" ]; then
+        if [ -x "$(which ethtool)" ]; then
+          if [ -n "$(ethtool $INTER   grep 'Link detected: yes')" ]; then
+            INTERRES=$(echo $INTER   sed 's/\./VLAN/')
+            RES="$RES if_err_$INTERRES"
+            eval "fetch_if_err_$ INTERRES ()   fetch_if_err $INTER $@;  ;"
+            eval "config_if_err_$ INTERRES ()   config_if_err $INTER $@;  ;"
+          fi
+        fi
+      fi
     done
   elif [ "$PLUG" = "netstat" ]; then
     if netstat -s >/dev/null 2>&1; then

• [1] http://www.linux.conf.au/
• [2] http://developer.servalproject.org/site/
• [3] http://noteworthyproject.com/
• [4] http://chris.neugebauer.id.au/2012/01/25/lca2012-android-is-not-vi-user-experience-for-geeks/